CI & Actions

Every workflow lives in .github/workflows/. The sections below group them by trigger: automatic on git events, or maintainer-invoked/advisory workflows via workflow_dispatch and schedules.

Automatic workflows

Quality Gate (ci.yml)

Fires on every PR targeting master and on trusted pushes to master. Composite job with multiple matrix legs:

• fmt: cargo fmt --all -- --check
• history-guard: fetches full history and checks the commit under test against origin/master; pull requests use the explicit github.event.pull_request.head.sha, while trusted pushes and merge-queue runs use github.sha. The guard and its fixture test reject an empty git merge-base, preventing a grafted second root from collapsing git blame after merge
• lint: cargo clippy --workspace --exclude zeroclaw-desktop --all-targets --features ci-all -- -D warnings, then cargo doc --no-deps --workspace --exclude zeroclaw-desktop (rustdoc warnings are fatal via .cargo/config.toml build.rustdocflags; desktop is excluded to match xtask build_api / docs-deploy and avoid GTK/glib-sys on the lint runner), and the comment hygiene gate
• build: unconditional matrix for x86_64-unknown-linux-gnu and aarch64-apple-darwin; a separate windows-build job checks x86_64-pc-windows-msvc when package ownership selects either of its commands
• check: two warnings-fatal passes over the workspace (excluding zeroclaw-desktop): no default features; and default features with --all-targets, which is the only leg that compiles test targets on the default feature surface. The all-features surface is compiled and type-checked by lint, whose clippy pass runs --features ci-all --all-targets on the same workspace; a separate all-features cargo check leg would duplicate that compilation without adding coverage
• check-32bit: i686-unknown-linux-gnu with no default features
• check-plugin-backends: selected by scripts/ci/plugin_backend_change_filter.sh. Compiles zeroclaw-plugins under each execution backend and runs the component, runtime admission, and root CLI plugin tests under plugins-wasm-cranelift. A separate plugin artifact smoke leg builds the zeroclaw binary with the same feature and runs scripts/ci/plugin_artifact_smoke.sh against it, with packages from scripts/ci/build_plugin_smoke_packages.sh. The smoke takes any zeroclaw binary, uses a throwaway configuration directory and the scripted local provider in scripts/ci/plugin_smoke_provider.py, and drives the operator’s own commands: plugin install, config set, and an agent turn that executes the tool. It checks that an installed plugin stays unused until the operator enables the plugin system, turns discovery on, and approves the tool; that values configured for a package that does not request them are not delivered; that a payload with the wrong digest, a payload that is not a component, and a component for another WIT world are refused; and that strict signature mode refuses an unsigned, untrusted, or edited package while it runs a trusted one. The leg’s last step runs the smoke with the opposite expectation and requires the exit status reserved for a failed check. The agent turn also depends on crates the path filter does not select, such as zeroclaw-providers, zeroclaw-tools, zeroclaw-memory, and zeroclaw-api, so a break there first shows on merge-queue and master runs, where the job always runs
• bench: benchmarks compile check; runs on master pushes, merge-queue runs, and manual dispatch only. PRs skip it (the gate treats the skip as non-fatal), so benchmark bitrot is still caught before a release without spending the compile on every PR
• test: the standalone firmware protocol host gate from scripts/ci/firmware_protocol_gate.sh and cargo nextest run --locked --workspace --exclude zeroclaw-desktop on Linux, including the config-write isolation and Fluent coverage (no bare user-facing strings) architecture guards. A final informational step replays the gateway golden-frame scenarios (crates/zeroclaw-gateway/tests/golden_frames.rs: WebSocket chat, webhook JSON and SSE, /api/events, and ACP) against their recorded fixtures with --run-ignored only. It reuses the job’s build, writes any drift to the step summary with a warning, and never fails the job; it stays informational until the gateway reaches the core over RPC
• test-tool-features: the tests of the opt-in SaaS, coding-CLI and external adapters, which Cargo default no longer builds: cargo nextest run over the root, config, tools, runtime, channels and gateway library tests plus the root report-template integration tests, with --features tools-compat. lint compiles the same code under ci-all but runs no test
• memory-postgres-test: cargo nextest run --locked -p zeroclaw-memory --features memory-postgres, followed by the crate’s serial ignored acceptance tests against an ephemeral PostgreSQL 17 service
• parallel-runtime-test: repeated same-process runtime/channel tests from scripts/ci/parallel_runtime_test_gate.sh, run in parallel with the main test job for relevant PR paths and unconditionally on master pushes and merge queue runs
• security: cargo deny check
• nix-eval: evaluates the NixOS module assertions (nixos-module-eval flake check)
• docs-style: markdown lint, em-dash prose check, and changed-line link gate via scripts/ci/docs_quality_gate.sh and scripts/ci/docs_links_gate.sh

Linux jobs in the required gate run on Blacksmith except for the kernel-dependent Landlock check: the compile-heavy fleet on blacksmith-8vcpu-ubuntu-2404, and the housekeeping set (fmt, the change detectors, repository guards, docs and policy gates, the Nix checks, the container smoke, security, web-permission-tests, and the gate aggregator itself) on blacksmith-4vcpu-ubuntu-2404. The one deliberate Linux exception is test-landlock, which stays on GitHub-hosted ubuntu-latest because the Landlock LSM check depends on the hosted kernel. Windows task-owner recovery uses blacksmith-8vcpu-windows-2025; macOS and the other Windows jobs keep their GitHub-hosted platform runners. This split exists because GitHub’s hosted-runner assignment stalled org-wide on 2026-09-14 while Blacksmith kept serving: with the housekeeping jobs moved, a hosted outage costs only the platform-specific checks instead of the whole required gate. All jobs name their runner labels and cache-provider inputs directly; tests/architecture/ci_runner_labels.rs checks the Linux inventories (compile fleet, housekeeping fleet, and the hosted-Linux allowlist) plus the recovery-only Windows runner and cache-writer contract. Work jobs do not wait on fmt; the deliberate trade-off is extra runner work on PRs that fail formatting. CI Required Gate still needs fmt directly and aggregates required results, so formatting errors still block merge. The separately label-gated advisory Windows workflow remains outside that gate. Its Linux selector also runs on the Blacksmith 4vcpu class. Branch protection pins the composite gate job. The master push run keeps the same quality signal while seeding trusted Rust caches for later PR runs.

Pull requests and merge-queue entries that change [workspace.package] version to a stable X.Y.Z also run crates-preflight, which calls pub-crates.yml with stage: preflight, no secret, and the compile fleet’s runner label. crates-preflight-changes decides this with scripts/ci/crates_preflight_trigger.sh; every other change skips both, and a skipped job does not fail the gate. Because the job is a reusable-workflow call with no runs-on, tests/architecture/ci_runner_labels.rs inventories it separately and requires it to pass the fleet label.

The runner inventories cover both ci.yml and windows-tests.yml, with workflow-qualified job identities so same-named jobs cannot hide one another. The Linux/macOS build matrix does not wait on any detector. The separate Windows compile, service and recovery jobs wait on the Blacksmith windows-required-changes detector. That detector reads package ownership with cargo metadata --locked --offline --no-deps, without registry resolution or downloads. The non-Rust path-changes detector remains independent of this package graph.

On master pushes only, the compile-heavy jobs (and the detectors that gate the Windows clippy and parallel-runtime jobs) first wait five minutes in master-debounce. Master pushes arrive in bursts, and the run-level cancel-in-progress otherwise cancels a superseded push only after its compile fleet has started; with the wait, a push superseded inside the window is cancelled while that cheap job is still sleeping. The job is skipped on pull requests and merge-queue runs, and a skipped job resolves without a runner, so their latency is unchanged. Because GitHub propagates a skipped ancestor through an implicit success(), every job downstream of master-debounce carries an explicit !cancelled() condition plus a result check for each dependency, and tests/architecture/ci_runner_labels.rs fails if one does not. CI Required Gate needs master-debounce, so a debounce failure cannot leave skipped compile jobs behind a green gate. The trade-off is five extra minutes before a master run starts compiling, which also delays seeding the master Rust caches by the same amount.

Fresh required CI is normally the shared evidence for the Cargo surfaces it actually runs. A local rerun of the same Cargo command on the same head, target, and feature set is duplicate confidence, not a stronger proof. Before asking for extra Cargo or Clippy, compare the changed surface with the current workflow files and the actual checks on the PR. Extra validation belongs where the required gate does not prove the thing under review:

• a platform received compile checks but not tests;
• a platform, crate, or path is outside the required lint job;
• a desktop change did not trigger the desktop workflow;
• a release target is outside the PR matrix and only covered by release/manual workflows;
• stale, cancelled, skipped, or unavailable CI is not fresh evidence.

When a definition or import is feature-gated, compare its cfg predicate with every consumer. Validate both the enabled configuration and each relevant disabled configuration: an enabled-feature pass proves the consumer still works, while the workspace-wide no-default-features check catches warning-producing mismatches such as unused private definitions or imports. That pass runs cargo check without --all-targets, so it never compiles test targets: a helper gated on plain test whose only callers sit behind a feature is caught by the default-features/all-targets leg instead. Targeted feature combinations remain necessary when neither required CI configuration exercises the changed predicate.

The PostgreSQL memory job runs all non-ignored zeroclaw-memory tests with memory-postgres enabled, then runs the crate’s ignored tests serially with ZEROCLAW_TEST_POSTGRES_URL pointed at its disposable service. To reproduce the database-backed half locally, provide an isolated PostgreSQL database and run ZEROCLAW_TEST_POSTGRES_URL=postgres://<user>:<password>@127.0.0.1:5432/<database> cargo test --locked -p zeroclaw-memory --features memory-postgres -- --ignored --test-threads=1. Each acceptance test owns a uniquely named schema and drops only that schema.

Required Windows compilation stays automatic without a label, but it checks only code that the commands can compile. On PRs, scripts/ci/windows_test_scope.py --required-jobs reuses package-root classification and reverse dependency closure. The root command runs when the affected closure contains zeroclaw; the voice-wake command runs when it contains zeroclaw-channels. Internal path edges include optional, target-specific, build and dev dependencies conservatively. A standalone ZeroCode-only edit does not run these unrelated commands. Workspace manifests, lockfiles, toolchains, Cargo configuration, workflow/action controls, unknown paths, and missing or malformed evidence select all required Windows jobs. Master pushes and merge-queue builds always run all of them. Linux/macOS compilation remains unconditional.

The service smoke builds windows_service_smoke_fixture, which owns its synthetic daemon and argument dispatch. Root CLI changes in src/main.rs do not automatically select it. Service source, runtime module wiring, the fixture and smoke script, config/spawn dependencies, and relevant manifest/build controls retain coverage. The task-owner recovery job retains direct control-plane and runtime module-wiring triggers. Package manifests and build scripts select these native jobs when their reverse dependency closure reaches runtime; unrelated gateway or standalone-app manifests do not. The recovery shell entrypoint delegates to the same package selector rather than maintaining a second ownership registry.

The recovery tests run in their own windows-task-owner-recovery job on blacksmith-8vcpu-windows-2025, in parallel with compilation because their runtime unit-test harness cannot reuse cargo check output. An unselected job skips before allocating a Windows runner. CI Required Gate directly consumes the detector and each required Windows job, rejects missing selection outputs, and requires every selected job to succeed. A selected-but-skipped, failed or cancelled job cannot count as an intentional skip.

Recovery calls the pinned Swatinem/rust-cache action directly with shared-key: windows-recovery-blacksmith, rather than assuming the GitHub build runner’s cache has a compatible Rust environment hash. Only successful push runs on refs/heads/master save this dedicated dependency cache; PR and merge-queue runs only restore it. A miss builds normally and does not skip either test. Default workspace-artifact cleanup remains enabled, so the runtime harness still compiles. This adds a cache namespace that consumes repository storage; compilation and service smoke keep their existing runners and caches.

Label-gated Advisory Windows Tests (windows-tests.yml)

Use ci:windows when executing tests on Windows can close a material gap left by compilation. Typical cases include process or service lifecycle changes, PowerShell execution, filesystem and path handling, encoding, native dependencies, and changes to Windows CI selection or execution. It is not a default label for every Rust PR, a docs-only change, or an unrelated Linux test failure.

Add the label once the candidate is stable enough for useful Windows evidence. Keep it during Windows-focused iteration when each update needs another run; remove it after a one-off validation is complete if later pushes do not need that coverage. Required Windows compilation remains automatic without the label. Nightly platform tests are a post-merge backstop, not a substitute for pre-merge execution when a change needs Windows-specific proof.

Add the ci:windows label to a PR targeting master to opt into advisory Windows tests. Adding that label starts the workflow for the current PR revision. Pushes and reopen events run it again while the label remains. Unrelated label additions neither rerun the tests nor cancel an active Windows run. Removing the label stops future automatic runs, but does not cancel a run already in progress. The label is a compute-cost switch, not a security approval; normal GitHub fork-workflow approval requirements still apply.

The workflow uses the ordinary read-only pull_request event, not pull_request_target, and tests the event’s frozen merge SHA. It has no manual dispatch trigger. Keeping it separate from ci.yml lets label additions start only the advisory workflow, without restarting required CI. The required gate does not depend on these jobs.

Read the selector summary and the Windows job’s own conclusion and logs before citing a pass. The Windows job uses continue-on-error, so a green workflow result alone does not prove its tests passed. Record the tested revision, selected scope, and any failed execution phase; distinguish test failures from selection or workflow failures using the logs.

The Linux selector checks formatting, compares the PR base SHA with the checked-out merge revision, and generates Cargo workspace metadata. It writes baseline mode, JSON packages, concise reason, and needs_plugin_host to job outputs and the step summary. The label opts into the existing selector, not an unconditional full suite. Nightly platform tests remain the automatic post-merge backstop.

skip means no changed path affects Rust compilation or tests covered by the current workspace suite. scoped maps Rust source, test, benchmark, example, and package-local manifest paths to package roots from Cargo metadata, closes that set over workspace reverse dependents, maps root src/ and tests/ to the root package, deduplicates package names, and excludes zeroclaw-desktop like the existing workspace suite. full is selected for workspace manifests or dependency resolution, .cargo, the Rust toolchain, CI or test infrastructure, the selector or workflow, unknown paths, ambiguous metadata, and other changes that cannot be mapped safely. Every Cargo.lock change selects full because workspace-wide dependency resolution can affect packages beyond the directly changed manifests. Direct changes to the root, gateway, or provider packages, plus changes to plugin, runtime, plugin config, WIT, root plugin activation, plugin backend filter, dependency, selector, selector-contract, ci.yml, or windows-tests.yml paths, set needs_plugin_host=true. The controlling-file cases make workflow revisions exercise the plugin-host path they own. Malformed or unavailable changed-path input selects baseline full and true. Missing or malformed Cargo metadata also selects baseline full with needs_plugin_host=true because the dependency closure cannot be established safely.

For scoped, the Windows job passes explicit -p arguments to cargo nextest; for full, it runs cargo nextest run --locked --no-fail-fast --workspace --exclude zeroclaw-desktop. When needs_plugin_host=true, it installs wasm32-wasip2 and appends the feature-enabled zeroclaw-plugins component targets (channel_plugin_e2e, tool_plugin_timeout_e2e, reference_plugin, reference_plugin_e2e, and tool_plugin_e2e), plugin library tests, runtime live-config and admission regressions, gateway library tests, the root CLI plugin tests (the plugin registry client, the plugins:: modules including the egress ceremony’s Windows shell proofs, the CLI egress ceremony tests, and the plugin update tests in the root binary), the plugin_update_cli and plugin_info_cli component tests, which run the real binary, and the root plugin_channel_runtime_e2e and channel_egress_e2e targets. Every appended invocation runs even when an earlier one fails, and the summary preserves each phase status plus the failure inventory. The job reports separate baseline, plugin-host, and total durations. Ordinary scoped and full selections do not pay the plugin-host compilation cost. This PR workflow restores compatible platform-test cache entries but never saves them. It checks out the frozen PR merge SHA and does not persist checkout credentials. A skipped Windows job is intentional and visible beside the selector result.

Scheduled Platform Tests (platform-tests.yml)

Runs cargo nextest run --locked --workspace --exclude zeroclaw-desktop --no-fail-fast on macos-15 and windows-latest after a cheap Linux formatting check. This nightly full-workspace run is the backstop for label-selected Windows coverage and inventories failures with --no-fail-fast. The matrix runs for:

• pull requests that change platform-tests.yml itself;
• manual dispatches; and
• the nightly 03:17 UTC schedule.

The jobs use continue-on-error and do not feed CI Required Gate. They are portability evidence, not merge requirements. Ordinary code PRs do not launch this full matrix automatically; maintainers can manually dispatch it against a branch when full platform proof is useful. The workflow does not run for ordinary push or merge_group events. Nightly and manually dispatched runs on master can write trusted caches; pull-request runs cannot. --no-fail-fast keeps every platform failure visible in a single run.

Daily Advisory Scan (daily-audit.yml)

Runs cargo deny check advisories daily at 09:00 UTC against the dependency tree. Opens an issue on findings. No action unless a vulnerability is reported.

Daily npm Audit (daily-npm-audit.yml)

Runs npm audit --audit-level=high daily at 09:23 UTC against web/package-lock.json. Opens one deduplicated security + dependencies issue when high-severity npm advisories affect the committed web lockfile.

Weekly Trivy Image Scan (trivy-scheduled.yml)

Scans the published dist and default-features GHCR images every Saturday and uploads HIGH/CRITICAL findings to the Security tab as SARIF. The scan is report-first (exit-code: 0 for findings), but a missing expected image fails the job before Trivy setup with the absent tag and the owning publisher workflow named in the error.

Weekly Scoop Bucket Canary (scoop-bucket-canary.yml)

Rehearses the Scoop publish path against the current stable release every Monday. It resolves the latest vX.Y.Z tag and calls pub-scoop.yml with both dry_run: true and credential_canary: true, so it exercises the real SCOOP_BUCKET_TOKEN against the real bucket without writing anything.

credential_canary is the fail-closed part of that contract: a missing SCOOP_BUCKET_REPO or SCOOP_BUCKET_TOKEN fails the run, and configured credentials must reach the git push --dry-run authorization probe. A generic manual pub-scoop.yml run with only dry_run: true remains permissive for manifest generation and may skip that probe when credentials are unavailable; do not use the generic mode as credential-verification evidence.

This exists because SCOOP_BUCKET_TOKEN is account-bound: it expires, and it silently loses write when the owning identity’s collaborator grant on the bucket changes. Both have happened. Before the canary, the only thing that exercised the credential was the post-publish scoop job, so a dead token was discovered after the release was already cut and announced, and the bucket had to be updated by hand.

The canary detects credential rot. It is deliberately not what keeps the bucket correct, and it is not wired into Release Stable: a dead package-manager credential must never gate or delay a release.

How the Scoop bucket stays correct

Today the release publisher is the only automated writer:

1. pub-scoop.yml pushes on release. Scoop users see the new version immediately when this succeeds. It needs the cross-repo SCOOP_BUCKET_TOKEN, which is the fragile part.
2. Maintainers recover failed pushes. Rotate or repair the token, dispatch Scoop Bucket Canary to verify it through the fail-closed credential_canary path, rerun the publisher with dry_run: false, and confirm the bucket manifest landed the release version.

A bucket-side Excavator is proposed in scoop-zeroclaw#1. Once that workflow is merged, the bucket repository grants Actions read/write workflow permission, and a maintainer smoke test proves that it commits an update, it can become a credential-independent recovery layer. Until all three conditions are satisfied, do not assume a failed publisher will self-heal.

The checkver and autoupdate blocks are already load-bearing for the planned Excavator path. The current push path also uses scripts/release/scoop_metadata.sh to derive its release URL template from autoupdate, so both paths share one manifest contract. Do not remove those blocks, and do not hand-edit them out of dist/scoop/zeroclaw.json.

PR Path Labeler (pr-path-labeler.yml)

Auto-applies path and scope labels based on changed files. It runs on PR open, reopen, and every pushed update to the PR branch. Because sync-labels: true is enabled, labels defined in .github/labeler.yml are recalculated from the current PR file set.

This workflow does not apply risk:*, size:*, type:*, contributor-tier, status, resolution, stale, or pickup labels. If a PR is missing a path/scope label, check whether the paths in .github/labeler.yml cover the changes.

Dependabot has separate label configuration in .github/dependabot.yml for its own PRs. Cargo update PRs start with dependencies; GitHub Actions and Docker update PRs start with ci and dependencies.

PR Size Labeler (pr-size-labeler.yml)

Applies exactly one canonical size:* label from PR file metadata. It runs on PR open, reopen, and every pushed update to the PR branch. The classifier counts additions plus deletions after excluding docs-like files and Cargo.lock, then applies the threshold table from Labels.

This workflow runs in pull_request_target so it can write labels on fork PRs, but it fetches the classifier script from the trusted workflow/default-branch revision. It does not check out, build, import, source, or execute pull-request code. It does not apply risk:*, type:*, contributor-tier, status, resolution, stale, pickup, or ProjectV2 fields.

PR Risk Report (pr-risk-labeler.yml)

Runs on PR updates and risk/security label changes through pull_request_target, plus manual dispatch for one PR. It fetches the classifier and JSON-compatible policy from the trusted workflow revision, reads PR metadata, changed files, labels, and API source text only when the #9530 test-only proof needs it, and writes a human-readable step summary plus JSON evidence. It never checks out, builds, imports, sources, or executes pull-request code.

Phase 2 is report-only: the workflow has contents: read and pull-requests: read permissions, does not add, remove, or replace labels, does not create commit statuses, does not enforce approval counts, and does not require a repository ruleset. risk:manual is reported as a hard freeze for future mutation, while domain:security is reported independently from the proposed risk:* result.

The classifier defaults uncertain evidence upward. Docs, fixtures, and known mechanical metadata may be risk:low; ordinary behavior changes are risk:medium; configured trust, credential, compatibility, governance, release-authority, security, or toolchain-floor paths are risk:high. The checked-in policy also carries deterministic changed-line escalation rules for workflow permission expansion, secret access, OIDC, artifact publication, release behavior, elevated pull_request_target, toolchain install or container baselines, release-floor changes, and likely WIT contracts. If one of those content-sensitive files changes but the patch is unavailable or malformed, the report stays high instead of guessing medium. The #9530 exception can recommend risk:medium only when a complete Rust patch and matching base/head source prove that every high-risk Rust change remains inside existing #[cfg(test)] code; missing, truncated, malformed, mixed, conditional-compilation, rename, add, and remove evidence stays high.

Project Dashboard Planner (project-dashboard-plan.yml)

Runs manually for a single issue number. It reads issue state and labels, then writes a report-only step summary proposing the existing Project Status value that best matches the issue.

This workflow does not run automatically on issue events, write ProjectV2 fields, edit issues, add labels, post comments, or recalculate PR risk:* or type:* labels. Live ProjectV2 mutation or automatic issue-event planning needs a separately approved field mapping, trigger policy, and project-scoped credential.

Validate PR title (pr-title.yml)

Runs on every PR open/edit/synchronize. Runs the validator unit tests (scripts/check-pr-title.test.sh) and checks the PR title against Conventional Commits (scripts/check-pr-title.sh).

Deploy mdBook docs to Pages (docs-deploy.yml)

Triggered on tag push (and workflow_dispatch); builds and publishes versioned docs to the gh-pages branch. See Release Runbook → Versioned documentation deployment for the version-floor and bootstrap rules.

Docker Image PR Check (docker-image-pr.yml)

Runs only when Docker image, Compose, or release-Docker context files change. It validates the merged default-plus-Alpine Compose configuration and, for changes beyond Compose-only edits, builds the default and Debian prebuilt smoke images plus the source Dockerfiles without pushing them. When the only Docker-path change is release-stable-manual.yml, it builds just the prebuilt smoke images: the release workflow’s Docker job packages prebuilt binaries through scripts/ci/prepare_docker_context.sh, which those images exercise, and it never consumes the source Dockerfiles. A diff that also touches any source-image input (a Dockerfile, the Containerfile, Compose files, .dockerignore, the config template, the Docker context scripts, or this workflow) still builds the full source matrix. The default and Alpine source images build for linux/amd64 and linux/arm64; the Debian source image builds for linux/amd64. Separate Alpine and Debian linux/amd64 lanes enable plugins-wasm-runtime-only so their builder contexts continuously prove that the repository WIT contract is available to plugin-enabled source builds.

The all-features Containerfile source image builds for linux/amd64 when that file or the Docker workflow changes. It uses an isolated cache scope and is neither loaded nor pushed. The Alpine amd64 lane runs both binaries, starts the built image through the merged Compose configuration, and checks the gateway health and dashboard surfaces. The Alpine arm64 lane is compile- and image-assembly coverage only. It cross-builds on blacksmith-8vcpu-ubuntu-2404, the existing 32 GB heavy Rust runner class, because its release link exhausted memory on ubuntu-latest. Other Docker jobs remain on ubuntu-latest; release optimization, build flags and matrix size are unchanged. Compose-only changes use a reduced Alpine amd64 matrix so they still exercise the runtime contract without rebuilding unrelated images. All jobs have read-only repository permissions and no registry write permission.

Docker Publish (docker-publish.yml)

Builds, signs, and scans the generated four-variant matrix from dev/ci/docker-tags.toml: minimal, default-features, dist, and all-features. A human-created v* tag starts this workflow directly. A stable release started with workflow_dispatch creates its tag with GITHUB_TOKEN, which does not emit another tag-push event, so release-stable-manual.yml calls Docker Publish synchronously at the immutable release tag after the canonical release and Docker jobs succeed.

This matrix supplements rather than replaces the stable release’s prebuilt latest, versioned, and debian images. The two paths use different build inputs and publish distinct tags.

Discord Release (discord-release.yml)

Fires after a successful stable release. Posts the release notes to the community Discord.

Tweet Release (tweet-release.yml)

Fires after a successful stable release. Posts an announcement tweet.

Weekly AUR Freshness Check (aur-freshness-check.yml)

Compares the published zeroclawlabs AUR version against the current stable GitHub release every Monday, and fails if the AUR is behind.

Publishing to the AUR is fire-and-forget: if pub-aur.yml fails, nothing re-checks, so the package silently falls behind. That is exactly what happened after v0.8.4. An aur.archlinux.org maintenance window overlapped the release, the single unretried clone failed with The AUR is down due to maintenance, and the package sat three weeks behind with no signal. The publisher now allows at most one active non-dry-run publish and retries to survive a short outage; GitHub may supersede an earlier queued real publish in the same concurrency group, while dry runs use a separate group. Every attempt reclones the authoritative package state and refuses to replace a newer epoch:pkgver-pkgrel tuple with an older one. A retry budget still cannot cover every failure, so this check is the backstop that turns a silent miss or superseded run into a visible one.

If the AUR RPC is unreachable the check warns and passes rather than failing. An AUR outage is an upstream availability problem, not package staleness, and the next scheduled run re-checks. Staleness is durable, so a delayed detection is acceptable; a weekly page about someone else’s maintenance window is not.

Docs are built and published as part of the release pipeline rather than on every master push. Translation is a local-only workflow for dedicated translation-cache PRs, new locales, and release translation passes. Routine English docs PRs may defer broad generated .po churn. See Docs & Translations for contributor guidance and the Release Runbook for the release procedure.

Manual and Advisory Workflows

Monthly Outdated Scan (monthly-outdated.yml)

Scheduled monthly scan on the 1st of every month at 09:00 UTC. Runs cargo outdated --workspace across all workspace members. Opens a dependencies-labeled issue when stale deps are found. Permissions: contents: read + issues: write. Dedup guard prevents piling up if the previous issue is still open.

The issue preview may be truncated, but its linked monthly-outdated-report artifact retains the complete scan output for 30 days.

First triage step for a new issue: check if the reported outdated crates have semver-incompatible bumps and whether the consuming crate’s API changed. If the bump is trivial (patch/minor), create a short dep-only PR. If the upgrade is blocked by semver breaks, close the issue with a note and the blocking crate name.

Cross-Platform Build (cross-platform-build-manual.yml)

Manual trigger for building release binaries across the full target matrix: Linux x86_64/aarch64 GNU and MUSL plus armv7 and arm hard-float, macOS Intel/ARM, Windows x86_64, and aarch64-linux-android (built with the NDK). Use this to verify a branch compiles cleanly on non-Linux targets before tagging.

Every dispatch also runs a small release-tool smoke matrix independently of the builds. Set release_tools_only when only this evidence is needed; the web and release-build jobs are then skipped. On trusted GitHub-hosted Linux x86_64, the smoke installs the pinned cross archive, confirms both cross and cross-util, and records cross --version. On trusted GitHub-hosted Windows x86_64, it uses the same Rust version and Bash-to-Cargo path shape as the stable release workflow, then records both cargo-tauri.exe --version and cargo tauri --version. Each leg records the exact tested commit and runner architecture in the public job summary. The smoke uses read-only repository permissions and has no publishing job, environment, secret, or artifact upload.

MUSL build legs also install cross through scripts/ci/install_release_tool.sh, which downloads the exact pinned upstream release asset and verifies its SHA-256 before installing it. The required Repository Structure job tests the supported runner-to-asset mapping and the smoke workflow contract without making network calls.

The default qualification_profile: normal retains the full matrix, the dist-broad raw-binary smoke route, the standard dist archive route, the normal cache policies, and the existing ARM64 memory lifecycle and macOS 180-minute ceiling. Its five jobs keep CARGO_INCREMENTAL=0 at job scope. The release-tool smoke validates the dispatch inputs before installing tools; normal product jobs retain their independent scheduling.

Select qualification_profile: stable-gnu-ubuntu22 with distribution: dist and release_tools_only: false for a finite, nonpublishing qualification. Empty, unknown, malformed, broad-distribution, or tools-only combinations are refused. Four GNU build rows run on Ubuntu22: x86_64, aarch64, armv7, and arm. They call the same local build-release-payloads action as Stable, with its existing linker and ARMv6 bindings, Rust1.98, locked release profile, explicit canonical features, ordinary Cargo jobs, three payloads per row, 90-minute ceiling, and inclusive 83,886,080-byte main-binary cap. Only the qualifier enables the action’s features output; its manifest consumes that single resolution. Stable leaves the evidence switch off.

Separate web and fixture producers use the existing entrypoints and Stable toolchain setup. These three build roles call the pinned Swatinem action directly with the original web, plugin-smoke-packages, and build shared keys. The fixture prefix and each Ubuntu22 target prefix match Stable, and default target restoration remains enabled. Every qualifier cache uses save-if: false, even on master. These roles omit the workflow/job CARGO_INCREMENTAL declaration, matching Stable. The pinned cache action exports CARGO_INCREMENTAL=0 before constructing its cache configuration and leaves that value in subsequent steps. Producer receipts require and record the effective string "0"; completeness checks the same value. The action-exported value is retained. The normal web cache policy is unchanged. Action logs must supply the actual primary key, restore keys, hit, and cache source before cache parity can be accepted. These inputs guarantee neither a warm cache nor a later Stable build duration.

The two native consumers run x86 GNU on Ubuntu22 and ARM64 GNU on Ubuntu24-arm, with 30-minute ceilings. Each verifies and extracts its same-run archive, runs the full verbose scripted-provider suite within five minutes, and unconditionally invokes the archive-aware plugin smoke. There is no raw fallback. The selected completeness job runs even after an upstream failure and rejects every missing, skipped, failed, or cancelled role. It checks four archives with the unchanged verifier, twelve input hashes and their saved ELF, compiler/sysroot and resource observations, and two native evidence carriers. Source commit, tree, run, attempt, profile, web and fixture hashes bind the receipts to one dispatch.

Before accepting this route, retain the actual cache receipts, interpret all twelve ABI observations, and inspect each native carrier’s named thirteen provider tests and eighteen archive controls. A successful job or carrier inventory alone does not replace those results or maintainer review. Missing or uninterpretable evidence holds acceptance; it does not authorize an automatic budget, resource, feature, or ABI-policy change. Ubuntu24 ARM64 execution is not Ubuntu22 ARM64 runtime proof. ARM32, companion execution, Darwin, MUSL, Android, Windows, Stable’s full matrix, and release readiness require their own evidence. A source change alone proves none of these runtime results.

Cross-Platform Clippy (cross-platform-clippy.yml)

Manual and weekly scheduled advisory lint coverage on macOS aarch64 and Windows x86_64 targets. It mirrors the required PR lint command with --target set for each platform, but intentionally does not run on PRs and is not part of CI Required Gate.

Required Linux Clippy, advisory cross-platform Clippy, and targeted Windows Clippy call scripts/ci/run_clippy.sh. That runner owns the supported command shapes, Cargo exit-status propagation, and the shared duration, cache, compile-count, and download-count diagnostics. The workflow files continue to own triggers, runners, toolchains, caches, timeouts, and required-gate membership.

Release Stable (release-stable-manual.yml)

Manual trigger for the full release pipeline. Builds all targets, creates the GitHub Release, pushes the prebuilt latest, versioned, and debian Docker images to GHCR, calls the generated Docker variant matrix at the release tag, triggers the website redeploy, and invokes the distribution sub-workflows (Scoop, AUR, Discord, tweet). Homebrew Core detects new releases through its own autobump service. Two environment gates require maintainer approval mid-run: github-releases (the publish job) and docker.

Downloadable assets use GitHub-hosted Build Level 2 attestations. Offline bundles and trusted-root material ship inside one verification archive, and both SBOM formats are checksummed and attested before the release is created. Cosign remains limited to GHCR image signing.

Every 64-bit archive except the experimental Android one carries the WASM plugin host, and a dispatched release proves it before the GitHub Release and its own image pushes: plugin-smoke-packages builds the fixture plugin packages once on Linux, and plugin-smoke downloads each archive on a runner of its own platform and runs scripts/ci/plugin_artifact_smoke.sh against the packaged binary (the same script the check-plugin-backends smoke leg runs on pull requests). publish and docker wait for every leg. An xtask test keeps the leg list equal to the registry’s per-target plugin policy, so a target cannot carry the host without a leg or lose its leg while it still carries the host. The build legs fail any binary over the 80 MiB cap set on the size check step. The manual Cross-Platform Build workflow rehearses the same legs against the binaries it builds, so a distribution change is measured and proven from one dispatch before a release runs. A release started by a tag push also triggers Docker Publish on its own v* tag trigger, and that run does not wait for the smoke.

See the Release Runbook for the full procedure.

Release-only build tools do not compile from source on every run. The workflow installs pinned upstream cross and Tauri CLI release binaries through scripts/ci/install_release_tool.sh; that script verifies a repository-owned SHA-256 for each runner-specific archive before placing the binary in Cargo’s bin directory. Updating either tool requires updating its version, asset name, and checksum together, then running scripts/ci/install_release_tool.test.sh.

Package Publishers

Each fires on workflow_dispatch with a version input. They are also invoked from the release workflow after a successful publish.

Homebrew Core’s official autobump service discovers stable GitHub releases and opens formula bumps independently. Do not restore a project-owned Homebrew publisher or fork token; that duplicates Homebrew’s authoritative automation.

Required secrets

Docker images push to GHCR using the automatic GITHUB_TOKEN; there is no separate registry token. Store CARGO_REGISTRY_TOKEN as a repository secret and map only that named secret into the reusable publisher. The called workflow references it only in the irreversible publish step, whose job requires approval through the crates-io environment; the tokenless preflight neither references nor exports it. The preflight packages the same immutable release commit before an approver can start the publish job.

Release Stable calls the publisher twice through its stage input. The crates-preflight job runs stage: preflight beside the binary builds, with no secret. A dispatched release has no tag yet, so it verifies the run’s commit directly. The GitHub Release publish job needs it, so a crates failure stops the release before anything is public. After the GitHub Release, the crates job runs stage: publish with the token. That stage requires the new tag to resolve to the same commit, skips re-verification, and uploads the dashboard bundle whose digest the preflight stage recorded. A manual dispatch of pub-crates.yml still runs both halves in one call.

Both jobs run the release scripts from a second, sparse checkout of scripts/release/ at .release-tooling, excluded from the packaged tree, and pass the tree through PUBLISH_SOURCE_ROOT. In a release run that checkout is the release commit. A manual dispatch from master for an existing tag uses master’s scripts against the tagged tree, so a publisher fix can reach a release without moving its tag. resolve_crates_release.sh rejects any other tooling commit.

The root crate ships the generated dashboard under web/dist. The preflight builds it once, records a digest of every file path and its content, verifies the packages with that tree, and uploads it as the crates-io-web-dist artifact. The publish job does not rebuild the dashboard. It restores that artifact, and publish-crates.sh refuses to reach crates.io unless the restored tree matches the recorded digest. The artifact is kept for 30 days so a late approval or a re-run of the failed publish job still finds it.

Most crates in the coordinated release set already exist and are eligible for crates.io trusted publishing. The v0.8.5 release additionally creates zerorelay, zeroclaw-relay-proto, and zeroclaw-tls, so its bootstrap token must include publish-new. The environment token remains the bootstrap path until every crate has a trusted-publisher entry for this workflow. After those entries are configured, migrate the job so GitHub exchanges OIDC identity for a short-lived token instead of retaining CARGO_REGISTRY_TOKEN.

The organization currently disables deploy keys on the Scoop bucket, and the automatic GITHUB_TOKEN cannot write another repository. Keep SCOOP_BUCKET_TOKEN narrowly scoped to the bucket; do not reuse a maintainer’s broad CLI token. The publisher checks write access with git push --dry-run, then uses the same Git transport for the real update.

Rotating SCOOP_BUCKET_TOKEN

Because deploy keys are unavailable, this credential is a personal access token and therefore has two independent failure modes, both of which have bitten a release:

1. The token expires. Fine-grained PATs have a maximum lifetime, so this recurs on a fixed schedule whether or not anything else changes.
2. The owning identity loses write on the bucket. The token can still be valid while the account behind it is only a read collaborator. This produces remote: Permission to zeroclaw-labs/scoop-zeroclaw.git denied to <account> and HTTP 403, not an auth error, so it reads as a code problem when it is a permissions problem.

Own the token with the ZeroClaw-Bot account, never a personal account, so the release path does not depend on one maintainer’s credentials. To rotate:

1. As ZeroClaw-Bot, create a fine-grained PAT with Resource owner zeroclaw-labs, Repository access limited to the single repository zeroclaw-labs/scoop-zeroclaw, and Repository permissions → Contents: Read and write. Nothing else.
2. Confirm the org approved the token. Fine-grained PATs against an org resource owner stay pending until approved, and a pending token authenticates but cannot push.
3. Confirm ZeroClaw-Bot still has write on the bucket: gh api repos/zeroclaw-labs/scoop-zeroclaw/collaborators/ZeroClaw-Bot/permission --jq '.role_name'. Step 1 does not grant repository access; it only scopes what the token may use. A token cannot exceed the permissions its owner already holds.
4. Set the secret: gh secret set SCOOP_BUCKET_TOKEN --repo zeroclaw-labs/zeroclaw.
5. Verify without touching the bucket by dispatching Scoop Bucket Canary. A green run proves the new token can push.

Record the expiry date somewhere durable when you rotate. The canary will catch an expired token within a week regardless, but only after it has already broken.

AUR package ownership

The project-owned package is currently zeroclawlabs, maintained by zeroclaw-bot. The canonical-name zeroclaw package is a third-party package and cannot be taken over by rotating AUR_SSH_KEY. If that maintainer remains inactive, follow the AUR orphan-request process before changing pkgname or the workflow clone target. After ownership transfers, coordinate the package rename or merge in one reviewed change.

Build cache behavior

Most Rust-heavy jobs in ci.yml cache through the local ./.github/actions/rust-cache composite, which receives use-blacksmith: 'true' from its Blacksmith compile callers, matching the runner label those jobs name directly. Both values are checked-in constants in ci.yml, not an Actions variable. vars is unreadable in fork-origin pull_request runs, so the former CI_USE_BLACKSMITH toggle silently sent every fork PR (90% of this repo’s PRs) to GitHub-hosted runners with a cold cache, because master had stopped seeding the GitHub Actions cache when it moved to Blacksmith. With the constants, those fork PR jobs use useblacksmith/rust-cache (Blacksmith NVMe sticky disk) on Blacksmith runners like the corresponding master jobs; changing the label and the provider inputs on the Linux compile jobs, which tests/architecture/ci_runner_labels.rs keeps in lockstep, moves that fleet back to ubuntu-latest + Swatinem/rust-cache in one reviewed commit. Both action references live in the composite regardless of the selection, so both must stay in the allowlist. Windows task-owner recovery is the direct-Swatinem/rust-cache exception on Blacksmith, with its own master-seeded dependency cache. The macOS and Windows build legs stay on Swatinem/rust-cache against the GitHub Actions cache, and the fmt, nix-eval, and docs-style jobs (none of which compile the workspace) use no Rust cache. codeql.yml pins the same Blacksmith label for its Rust scan, so no workflow reads the CI_USE_BLACKSMITH variable any more; it can be deleted from the repository settings. These behaviors are worth knowing when triaging cache-related flakes:

• Matrix legs need distinct cache keys. A cache key is derived from the job id (or shared-key), the runner OS and architecture, the toolchain, CARGO*/RUST* environment variables, and the lockfile. Matrix legs that differ only in feature flags or cargo arguments therefore resolve to the same key; the leg that saves first owns it and every other leg restores a mismatched dependency set. Before per-leg keys, the cranelift plugin-backend leg recompiled about 620 dependency crates (roughly four minutes) on every run, master included, because it shared a key with the pulley and runtime-only legs. check and check-plugin-backends therefore give every leg a comma-free cache-key matrix field passed as shared-key; legs whose dependency sets genuinely match may share one. A new Rust matrix job should do the same.
• Cache writes are master-only. save-if is conditioned on github.ref == 'refs/heads/master', so PR runs read the master-seeded cache but never update it. PR branches can’t pollute the shared cache with branch-specific artifacts. The push trigger on master is what gives the workflow a trusted cache-writing run after merges.
• Recovery saves only on success. Its cache-on-failure: 'false' keeps failed recovery runs from seeding its cache. Other composite-cache compile jobs retain cache-on-failure: true, so a partial master run can still seed their next attempt warm.
• Windows build cache is enabled. The Windows build leg runs the same pinned Rust cache action as Linux and macOS. If Windows cache behavior flakes or regresses, revert the workflow change and document the failing restore/save evidence in the cache issue.
• Incremental compilation is disabled. CARGO_INCREMENTAL: 0 at the workflow level. Incremental builds inflate cache size and produce non-reproducible artifacts under partial-stale conditions.
• cargo-deny and cargo-nextest are installed fresh each run. The security job runs cargo install cargo-deny --locked; the Linux test job and both scheduled platform-tests.yml legs pull the appropriate cargo-nextest binary from get.nexte.st. The advisory PR Windows job instead downloads its pinned nextest release archive and verifies a hardcoded SHA-256 before extraction because it runs on a much larger set of untrusted PRs. Neither tool is cached, so each install adds a fixed cost to its job. Switching either to taiki-e/install-action would let them be cached, but that action is not in the allowlist today.

When the gate goes red

Allowed actions

The repository runs Actions in selected mode, only the actions in this allowlist may run. The allowlist must stay tight; new third-party actions need explicit maintainer approval before being added.

All third-party refs are pinned to a full commit SHA with a trailing version comment; the version column below records that comment.

The GitHub Release itself is created with gh release create inside the publish job, not a release action.

Equivalent allowlist patterns (kept narrow on purpose):

actions/*
dtolnay/rust-toolchain@*
Swatinem/rust-cache@*
useblacksmith/rust-cache@*
docker/*
sigstore/cosign-installer@*
anchore/sbom-action@*
aquasecurity/trivy-action@*
github/codeql-action/upload-sarif@*
github/codeql-action/init@*
github/codeql-action/analyze@*

Export the current effective policy:

gh api repos/zeroclaw-labs/zeroclaw/actions/permissions
gh api repos/zeroclaw-labs/zeroclaw/actions/permissions/selected-actions

Any PR that adds or changes a uses: action source must include an allowlist impact note in its body. Avoid broad wildcard exceptions; expand the allowlist only for verified missing actions.

Maintenance rules

• Keep CI Required Gate deterministic and small. Adding jobs to the gate needs a clear quality argument.
• All third-party action refs must be pinned to a full commit SHA (per the allowlist policy above).
• Keep ci.yml, dev/ci.sh, and .githooks/pre-push aligned. Shared gates must live in scripts/ci/; each caller invokes the helper instead of copying its commands. For the standalone firmware protocol gate, the documented local entry point is ./dev/ci.sh firmware-protocol.
• Keep scripts/ci/prepare_docker_context.sh, docker-image-pr.yml, and the Docker job in release-stable-manual.yml aligned so PR validation exercises the same context shape the release workflow publishes.
• Run python3 scripts/ci/release_attestation_contract_test.py after changing the release attestation, checksum, SBOM, or verification-archive sequence.
• The docs-style gate job runs bash scripts/ci/docs_quality_gate.sh (markdown lint + em-dash prose check) and bash scripts/ci/docs_links_gate.sh (changed-line link gate). Run both scripts locally before pushing docs changes.

Emergency rollback

If the allowlist locks out a critical action mid-incident:

1. Temporarily set Actions policy back to all.
2. Restore selected allowlist after identifying the missing entry.
3. Record the incident and the final allowlist delta.

This is the only justified path to all mode, and it should never outlast the incident.