---
name: openclaw-ci-limits
description: Manage OpenClaw GitHub Actions and Blacksmith CI capacity, runner-registration budgets, fanout caps, main-push single-flight, shard sizing, hosted-runner offload, queue health, and safe ramp-down/ramp-up changes. Use when tuning `.github/workflows/*`, `docs/ci.md`, CI runner labels, matrix `max-parallel`, ClawSweeper/Blacksmith burst protection, CodeQL runner placement, or investigating slow/queued OpenClaw CI.
---

# OpenClaw CI Limits

Use this skill for CI capacity changes, not ordinary test failure triage. The
goal is to keep OpenClaw fast while distinguishing runner registration, runner
availability, Blacksmith control-plane health, and downstream queue drains.

## Core Facts

- Full main-tier CI runs hourly; ordinary main pushes run `security-fast`.
  A green push is not evidence that the full suite passed. Ordinary PRs defer
  Windows, macOS Node/Swift, iOS build/simulator, real-Gateway UI, published-driver
  update, Knip, and most architecture proof. Import-cycle and Kysely guards
  remain blocking. Use `docs/ci/scope-and-routing/selection.md` and the current
  manifest for exact event, owner, and release-fallback selection; do not restore
  deferred lanes while tuning capacity. Android screenshots remain release-only.
- Do not assume the scarce resource. Prove whether pressure is runner
  registrations, eligible runner availability, Blacksmith capacity/control
  plane, workflow dependencies, test runtime, or a downstream queue writer.
- GitHub runner registrations for `openclaw` currently report a 10,000 per
  5-minute bucket in `actions_runner_registration`. Verify the live bucket
  before each tuning pass because GitHub can change it. The `openclaw`
  organization shares one bucket.
- Core REST quota does not draw down this bucket. Check
  `actions_runner_registration` separately; core quota can be healthy while
  runner registration is throttled.
- Use about 60% of the live bucket as the operating target. With the current
  10,000-registration bucket, keep planned Blacksmith burst load under 6,000
  registrations per 5 minutes and leave the rest for other repos, retries, and
  burst overlap.
- Jobs that route, notify, summarize, choose shards, or run short CodeQL quality
  scans should stay on GitHub-hosted runners unless measured evidence says
  Blacksmith is required.

## Runner Cost Policy

- Use the smallest runner that completes the required workload reliably. Keep
  short control jobs hosted and ordinary trusted development proof local.
- Treat physical 8-vCPU and larger runners as exceptions, not a default or generic retry. Record the
  command and measured peak memory, a smaller-runner OOM, or a controlled
  comparison showing lower total billed cost. Low CPU use alone does not prove
  a memory-heavy job can move down. Preserve resource-based worker limits.
- Routine OpenClaw Testboxes use the 4-class with a 60-minute total-job
  default, including hydration. Keep the 15-minute idle ceiling. Shorter
  deadlines are welcome for known short commands; do not request four hours
  automatically or upsize only to obtain more time. Select
  `.github/workflows/ci-check-high-memory-testbox.yml` explicitly only for a
  named memory-heavy command; see `docs/reference/test/remote-proof.md`.
  That workflow has at most four concurrent leases inside the shared 32-slot
  Testbox pool. The high-memory profile uses 8 vCPUs; both profiles preserve
  their pre-November 3 physical CPU and RAM. All Testbox profiles cap idle time at 15 minutes.
- Allocate through the current OpenClaw wrapper with workflow ref `main`;
  the source capsule preserves the checkout being tested. Do not dispatch an
  old workflow ref to bypass spending limits. Queue age is checked before
  checkout; an admitted lease keeps its job and idle deadlines.
- Do not promote an entire workflow family because one command needs more RAM.
  Keep proven high-memory CI rows scoped to their owning planner and evidence;
  remeasure before changing their allocation. A 32-class label is not proof of
  32 available CPUs. Compare observed resources and total billed job cost.

## Rejected Experiments

- **UI blobless sparse checkout (2026-10-16):** Excluding native mobile/macOS
  app trees removed 48.31 MiB of logical tracked content, but did not qualify as
  a runner-cost improvement. In
  [run 38866595993](https://github.com/openclaw/openclaw/actions/runs/38866595993),
  the Blacksmith UI E2E checkout took 56s and the hosted browser checkout 58s;
  same-run full UI checkouts took 22–38s. These are small, unpaired samples,
  not a measured billing delta. Keep the full source checkout. Reconsider only
  with lower complete checkout time and billed cost, including filtered fetch,
  lazy materialization, and trusted-harness export; fewer logical bytes alone
  do not establish savings.
- **Declaration-fixture owner selection (2026-10-02):**
  `test/scripts/tsdown-declaration-fixture.ts` derives real source copies from its runtime
  entrypoint closures, so product PRs that edit one do not select the four declaration suites
  built on it. [PR #162614](https://github.com/openclaw/openclaw/pull/162614)
  broke `write-unified-entry-dts` that way; hourly scheduled CI
  ([run 36979748808](https://github.com/openclaw/openclaw/actions/runs/36979748808))
  caught it about 35 minutes after merge and `0a188c4f038f0` repaired the
  fixture within the hour. Declaring the owners as `new URL(..., import.meta.url)`
  literals reaches the suites only below aggressive selection's 20-importer
  hub cutoff: the incident owner and seven others are hubs, so the replayed
  #162614 diff still selected none. Replaying all 92 main commits from the
  prior 30 days that touched an owner through the canonical PR planner options,
  URL literals newly covered 12 (+35 rows, +1,689 class-vCPU-minutes). Full
  coverage, such as a policy watch over the same list, newly selects the suites
  for 81 and starts `build-artifacts` (median 330s on the 16-class across 12 PR
  runs) for 69, because the suites are build owners in `ci-build-manifest.mjs`:
  +242 rows and +11,628 class-vCPU-minutes, a median +161 (+8%) per affected
  PR. That buys one escaped owner break in seven weeks, so keep the plain
  paths; hourly scheduled CI, tooling-owner PRs and release validation own this
  proof. Reconsider with a materially higher escape rate or a selection that
  does not also start `build-artifacts`.
- **Boundary asynchronous input preparation (2026-09-26):** Adding the existing
  `CompilerInputSnapshot.prepare()` calls at the three declaration/boundary
  callers increased full cold validation from 464.21s to 544.68s on the same
  four-CPU/15.42-GiB Testbox; warm validation increased from 12.62s to 14.63s.
  All 125 plugin checks and the canary passed, but CPU use also increased.
  The six caller additions were removed. Do not repeat this as an assumed
  speedup; any different use needs measured end-to-end benefit.
- **Windows pnpm store (2026-09-20):** Original receipts from
  [run 35547255790](https://github.com/openclaw/openclaw/actions/runs/35547255790)
  measured median complete setup at 45.295s cold versus 52.738s restored
  (+16.4%), despite reusing all 1,453 packages with zero downloads. The
  763.8-MiB archive took 23.601–26.980s to restore; producer setup/save added
  62.527s. All seven native jobs passed, but the qualification reducer failed
  on a 24-KiB reported-RAM difference. No assertions were relaxed or jobs
  rerun. The exactly RAM-matched subset was still 11.3% slower; that is
  descriptive evidence, not a replacement passing qualification. Keep normal
  Windows setup uncached. Reconsider only with net complete-setup savings,
  including restore, extraction, frozen reconciliation, and producer work.
- **Hosted Mac exact dependencies (2026-09-01):** The same-head publisher and
  consumer in [run 33458856298](https://github.com/openclaw/openclaw/actions/runs/33458856298)
  successfully saved and restored a 1.66-GB dependency archive, but setup took
  142s versus 86s with the ordinary store cache. Extraction took 82s versus 27s;
  install improved only from 43s to 35s. Keep hosted Mac jobs on the ordinary
  store cache. Reconsider only with measured total setup savings, including
  transfer, extraction and frozen reconciliation, not a successful cache hit.
- **Actions-artifact checkout (2026-08-16):** Do not recommend replacing the
  shared Blacksmith Git fetch with a preflight-produced workspace or `.git`
  artifact. [PR #124818](https://github.com/openclaw/openclaw/pull/124818)
  measured a 16s Blacksmith checkout baseline versus 7s hosted. The best direct
  artifact variant cost 1s to pack, 3s to upload, and 11s median to restore;
  including the serial prefix left only about 1s median improvement and
  regressed Blacksmith's fast-fetch runs. The official artifact client was
  worse: [run 31971531521](https://github.com/openclaw/openclaw/actions/runs/31971531521)
  measured 22s median download plus 2s materialization. Blacksmith's fast
  Actions-cache path does not imply fast Actions-artifact downloads. Reconsider
  only with measured end-to-end proof for a different transport, including its
  producer cost and fast-fetch regressions.

## First Checks

Before changing CI, collect current pressure:

```bash
gh api rate_limit --jq '{core:.resources.core,graphql:.resources.graphql,search:.resources.search,actions_runner_registration:.resources.actions_runner_registration}'
gh run list -R openclaw/openclaw --limit 20 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
gh run list -R openclaw/clawsweeper --limit 20 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
gh api repos/openclaw/clawsweeper/actions/runs/<run-id>/jobs --paginate --jq '.jobs[] | {id,name,status,conclusion,labels,created_at,started_at,completed_at,runner_name,runner_group_name}'
blacksmith testbox list --all
curl -fsS https://clawsweeper.openclaw.ai/api/status | jq '{generated_at,fleet,diagnostics:{errors:.diagnostics.errors}}'
curl -fsS https://clawsweeper.openclaw.ai/api/exact-review-queue | jq '{generated_at,review:.lanes.review,publication:.lanes.publication,state_writer,state_append}'
node scripts/ci-run-timings.mjs --latest-main
node scripts/ci-run-timings.mjs --recent 10
```

For a suspicious queued run, inspect its jobs. A run-level `queued` status does
not reveal whether the job is waiting on dependencies or has no eligible
runner. Compare `created_at`, `started_at`, `labels`, and `runner_name`. Recheck
stale queued runs live before canceling them; cancel only runs proven obsolete.

Testbox `warmup` confirms lease creation, not runner readiness. An initial
`queued` snapshot without `providerMetadata.runURL` does not establish a capacity
shortage. Use that URL to inspect the exact workflow once associated. An assigned
`check` runner performing setup is hydration; a confirmed admission or capacity
queue follows the existing stop-owned-allocation policy. Do not infer association
from the newest workflow run or wait on a confirmed capacity queue.

`scripts/ci-run-timings.mjs` start delay can include workflow dependency wait
plus runner queue time. It is trend evidence, not runner-pressure proof alone.

Read:

- `.github/workflows/ci.yml`
- `.github/workflows/codeql-critical-quality.yml`
- `docs/ci.md`
- `test/scripts/ci-workflow-guards.test.ts`
- `test/scripts/ci-workflow-planning.test.ts`
- `test/scripts/ci-workflow-evidence.test.ts`
- touched planner files under `scripts/lib/*ci*`, `scripts/lib/*test-plan*`, or
  `scripts/ci-changed-scope.mjs`

## Diagnose The Bottleneck

Classify the issue before changing caps:

- **Runner-registration throttle:** many jobs queued before runner assignment,
  Blacksmith/GitHub reports 403/429 or spam-style 422 responses from
  `generate-jitconfig`, and API core quota is still healthy. Treat 422 as this
  signal only when the request payload is otherwise valid. Fix burstiness and
  Blacksmith job count.
- **Blacksmith capacity:** Blacksmith dashboard shows actual concurrency caps or
  unavailable capacity. Do not solve this with GitHub workflow fanout alone.
- **Blacksmith Testbox control plane:** list, warm, status, or run calls time out
  before a lease is returned. This is separate from Actions runner registration
  and Actions job capacity. Trusted source may use the documented local
  fallback; untrusted source stays blocked.
- **Unavailable runner label:** a job is queued with a custom `runs-on` label,
  `started_at` and `runner_name` remain empty, and no eligible runner exists.
  Restore an available hosted or registered label; fanout cannot fix it.
- **Workflow dependency wait:** the job is queued but required predecessors are
  not terminal. Fix or wait for the dependency; do not call the whole delay
  runner queue pressure.
- **OpenClaw test runtime:** jobs start quickly but one lane dominates wall time.
  Use `$openclaw-test-performance` instead of runner tuning.
- **Real failing CI:** one job fails after starting. Use `$github:gh-fix-ci` or
  `$openclaw-testing`, not this skill.
- **ClawSweeper review backlog:** review pending/ready grows while publication
  and state writers remain healthy. Tune review admission/workers in
  `openclaw/clawsweeper`.
- **ClawSweeper publication backlog:** publication pending/ready and oldest age
  grow, net drain is zero or negative, or dead letters rise. Inspect publication
  batches, state-writer coordination, and GitHub mutation latency first.
- **State materializer/append backlog:** `state_append.pending_rows`,
  `pending_bytes`, or oldest age grows while the materializer is queued or
  absent. Recover that sole drain first; more review workers make it worse.

## Registration Budget Math

Estimate worst-case registrations for a change before editing:

```text
new Blacksmith registrations ~= number of Blacksmith jobs that can become queued
inside one 5 minute window
```

For matrix jobs, count every row that can start in the 5-minute window.
`strategy.max-parallel` only caps simultaneous rows; short rows can turn over
and register more runners before the window resets. Use job duration, retries,
and queue turnover to justify any lower estimate. Add non-matrix Blacksmith jobs
such as `preflight`, `security-fast`, `build-artifacts`, and platform lanes.

For repeated pull-request pushes, multiply by the number of runs expected to
reach Blacksmith admission in the same 5-minute window, including runs canceled
after admission. Canonical `main` uses two run-number-parity slots. Each slot
keeps one active non-canceling run and one coalesced pending tip. Budget for up
to two active main matrices plus their two pending tips entering the next
admission wave, not every intermediate merge.

Reject a change unless the org-level worst case stays below about 60% of the
live bucket. With the current 10,000-registration bucket, keep planned
Blacksmith burst load under 6,000 registrations per 5 minutes with headroom for
ClawSweeper, ClawHub, Clownfish, OpenClaw RTT, and Clawbench.

Native compact plans admit 90 rows and GitHub-hosted plans 96; final Node caps remain 70 push and 130 PR rows.
The current automatic main/PR source has a conservative union of 71 potentially
self-hosted non-Node rows, including five core-lint stripes for fork PRs,
five type stripes, five Windows rows, and thirteen UI E2E rows. Retain an 84-row
allowance with thirteen rows reserved: `4 × 154 + 21 × 214 = 5,110` for the
four-main/21-PR arrival envelope, leaving 890 below the 6,000 reference target.
The six extension-lint rows remain hosted. Manual/frozen releases and other
workflows are outside this envelope. This is conditional planning, not live
organization-wide capacity proof; older 5,010/5,085/5,160 examples below describe
historical inventories and do not replace this source-audited reserve.

## Safe Levers

Prefer these in order:

1. Preserve cancel-in-progress for superseded pull-request heads.
2. Preserve canonical `main` as two non-canceling parity slots; each slot's
   default pending run coalesces to the newest tip.
3. Move high-frequency, short, non-build jobs to `ubuntu-24.04`.
4. Reduce matrix rows by bundling related tests inside one runner job when the
   combined job stays under timeout and keeps useful failure names.
5. Lower `strategy.max-parallel` for bursty Blacksmith matrices.
6. Right-size runners from timing evidence. Use fewer/larger jobs only when
   elapsed time improves enough to justify registration count.
7. Split truly slow tests with `$openclaw-test-performance`; do not hide a slow
   test problem by registering more runners.

Do not:

- add another Blacksmith installation expecting a higher registration bucket;
- move CodeQL Critical Quality back to Blacksmith;
- raise all `max-parallel` values at once;
- make manual `workflow_dispatch` runs cancel normal push/PR validation;
- delete coverage just to reduce runner count;
- treat cancelled superseded pull-request runs as failures without checking the
  newest run for the same ref.
- cancel old queued runs from a stale snapshot; re-query the exact run first and
  preserve any current run that still owns live work.

## Current OpenClaw Knobs

These are intentionally guarded by the `ci-workflow-guards`,
`ci-workflow-planning`, and `ci-workflow-evidence` tests under `test/scripts/`:

- Current-target Control UI locale checks share an already-selected public
  artifact producer on real PR/push/schedule events. Keep verification before
  generators, settle the existing strict/advisory outcomes after build diagnostics,
  and route the logical locale gate to that explicit result. Dispatches, frozen
  and compatibility targets, and standalone locale selections retain their job.
  Hosted row accounting omits only the shared locale row; no build is newly selected.

- `CI` concurrency key version, PR cancellation, and canonical `main`'s two
  non-canceling parity slots, each with one coalesced pending tip.
- `preflight` starts immediately without a debounce or standalone admission job.
  `security-fast` waits for its hosted budget decision and still runs after
  preflight failure unless the workflow was canceled. The protected `vitest-cache-warm` workflow
  publishes the immutable semantic dependency archive after setup succeeds,
  in an independent short job. Dependency and code publishers serialize per
  backend/platform/ref without a workflow-wide lock or cross-platform dependency.
  Hybrid adds a bounded hosted tooling/contract/UI seed and native SDK archive,
  without repeating the full Linux build. Docs-only pushes skip warming. The split
  adds one Blacksmith registration per eligible warmer admission, not per CI job;
  include short-publisher turnover in burst estimates. Preflight and downstream Node jobs are
  restore-only consumers on eligible self-hosted runners. Exact misses and
  hosted paths, including Mac Node jobs, use the ordinary pnpm-store cache.
- Ordinary canonical same-repository hybrid first attempts keep `preflight`,
  `pr-failure-report`, and `ci-gate` on hosted Ubuntu. October 15's 26-PR sample
  found 148 minutes of Blacksmith preflight queueing, with a 12.9-minute p75.
  Fork first attempts, RunsOn, and admitted qualification/release dispatches
  retain their existing routes. Count these mandatory hosted controls before
  optional offload; keep the 45-row ceiling and all job/worker limits.
  Core lint uses the 16-class on canonical automatic first attempts with an
  unset, `blacksmith`, or `hybrid` backend, including fork PRs. Admitted
  qualifications retain their existing routes. The first packed lint row took
  621s on the 8-class; retain four
  actual CPUs for that row. The second packed row later exceeded its existing
  15-minute limit on the 8-class, so it uses the same 16-class. This adds no
  jobs or registrations and keeps the deadline and complete stripe