Stalled or stuck feature
Diagnose and clear a feature that is dispatched but not progressing: no-task-progress stalls, provider-preparation exhaustion, build-progress ceilings, rate-limit waits, auth parks, and kickback loops. For operators reading .pipeline/ state and the daemon log.
Not sure this is the right runbook? Run
/daemon-triage— it gathers the evidence read-only, classifies the failure, and routes to the one runbook that owns it.
Symptom
Any of these:
- The daemon log shows
▶ start <slug>and then nothing for a long time. ■ done <slug>: haltedor■ done <slug>: errorappears and the slug stops being dispatched.- The startup dashboard lists the slug under HALTED instead of IN-PROGRESS or ELIGIBLE.
- The dashboard lists a reason and
remedy:line for a PARKED, HALTED, or retained slug. - The same build step retries repeatedly with the same reason.
- A feature is marked complete but the PR never appeared.
Diagnosis
All per-feature evidence lives in the feature’s own worktree: .worktrees/<slug>/.pipeline/. All daemon-level evidence lives at the main repo root: .daemon/.
Blocked merged spec
Symptom: A merged plan never dispatches, and ai-conductor daemon status lists it in BLOCKED with a reason and remedy. This is a discovery refusal, not a feature HALT: no feature worktree or .pipeline/HALT exists yet. The startup dashboard does not render this state (#1332); use daemon status.
Diagnosis: Read the BLOCKED line. Common reasons are unresolvable-stories-ref, stories-missing, stories-not-approved, adr-not-approved, no-dependency-tree, and missing-coherence. adr-not-approved means the default branch’s .docs/decisions/ corpus contains an ADR whose first declared status is not APPROVED or SUPERSEDED (or that declares no status); it blocks every merged spec, not just the one it names. The snapshot is .daemon/blocked.json; a missing or malformed snapshot means only that no completed discovery pass is available, not that the spec is clear.
Recovery: Apply the line’s remedy on the repository’s default branch. Do not repair only a feature branch or a local working tree: discovery reads the committed default-branch tree. After an upgrade that adds this visibility, the first discovery pass may dispatch a previously invisible, otherwise-buildable merged spec when the repository has no processed marker for it. That is expected; review the plan before starting the daemon if those older specs are not ready to build.
Verification: Run ai-conductor daemon status after the next pass. A repaired spec disappears from BLOCKED and is eligible for dispatch; a remaining entry includes its current remedy.
First, read the committed halt record
For an operator-actionable halt, read .docs/halted/<slug>.md from the feature branch before consulting worktree-local state. The committed record survives a recreated worktree and can be read from any clone that has the feature branch:
git show <feature-branch>:.docs/halted/<slug>.md
It gives the halt class, step, phase, branch, HEAD SHA, timestamp, and complete HALT body. A Status: resolved record preserves those original details and also records how and when it was cleared. mechanical halts deliberately create no record, so begin with the live marker when this file is absent for that class.
The push is best-effort. If the daemon reports halt_record_push_failed, the record was still committed locally on the daemon host but may not be visible from the remote branch. Read it from that feature worktree instead:
git -C .worktrees/<slug> show HEAD:.docs/halted/<slug>.md
Repair the push failure separately; do not treat a missing remote copy as a missing halt record.
1. Read the halt marker
head -1 .worktrees/<slug>/.pipeline/HALT
cat .worktrees/<slug>/.pipeline/HALT.class
.pipeline/HALT is the durable park state for that feature: the daemon never advances, opens a PR, or merges past it. The first non-empty line of the body is the reason the dashboard surfaces. Read it by name during recovery; persisted halt events in .pipeline/events.jsonl supplement this marker and do not replace it.
.pipeline/HALT.class classifies it. Every HALT the daemon writes now carries one — the daemon stamps any HALT still missing a class at startup as legacy, once per feature, so the sidecar is never absent for long. Content the reader doesn’t recognize still reads as unclassified.
| Class | Meaning | Cleared by the re-kick sweep? |
|---|---|---|
needs-human | Only an operator can resolve it. A detail naming a remediation-planner mismatch is a plan gap wearing this class; see the remediation planner disagreed with the parsed findings. | No — except a matching, one-use kickback-budget authorization for a budget-cap halt. |
kickback-cap | A bounded prd_audit or as-built-review remediation allowance is exhausted. | No — except a matching, one-use kickback-budget authorization. |
plan-gap | prd_audit or the as-built review found an outcome no active plan task owns; see the plan-gap recovery. | No — skipped on every sweep. |
mechanical | The daemon may safely retry it. | Yes, on a base-branch advance. |
protected-artifact | BUILD or SHIP found a genuine protected DECIDE-artifact violation. | No — skipped on every sweep until an operator resolves it. |
legacy | Predates total classification; stamped by the daemon’s startup migration. | Yes, on a base-branch advance, same as mechanical. |
| (absent / unrecognized) | Treated as unclassified. | No — skipped on every sweep, same as needs-human. |
A feature that errors gets a diagnostic HALT written at the deterministic .worktrees/<slug>/.pipeline/HALT path even when worktree creation itself fails. Read and clear that marker after fixing the cause. If the daemon log reports unrecoverable-state, it could not write that marker; repair the target directory or permissions first, then re-run the dispatch.
NEVER-STARTED is not a halt: the dashboard found no readable conduct-state.json, and the feature remains dispatchable. Do not reclaim or unpark it merely to make it run. For a retained row, follow its remedy: line: an open PR needs no action until it lands; a closed, unknown, or legacy PR state can be handled with ai-conductor daemon reclaim-worktree <slug> when reclaim is appropriate.
2. Classify the stall
grep -E 'build_stall|build_no_progress|build_active_stall|zero_work_product|rate_limit|credentials_park' \
.worktrees/<slug>/.pipeline/events.jsonl | tail -20
A later step is blocked by an unsatisfied prerequisite
Symptom: .pipeline/HALT says Step '<step>' is blocked by unsatisfied prerequisite and lists one or more prerequisite names with their recorded statuses. The HALT class is needs-human.
Diagnosis: The listed prerequisite cannot run again automatically: it is no longer pending. Read that prerequisite’s halt record or its earlier events before changing anything. A refused status means the prerequisite’s own work was not judged failed; its original HALT explains the operator action required.
Recovery: Resolve the prerequisite’s stated cause, then clear HALT and HALT.class using the resume procedure. Do not hand-edit conduct-state.json to mark the prerequisite done or failed. The resumed conductor admits the unsatisfied prerequisite again and preserves already completed steps.
Verification: Run ai-conductor daemon status after the next dispatch. The prerequisite runs again; the blocked later step does not remain the terminal step unless the underlying condition is still unresolved.
no_task_progress
The build step’s circuit breaker. From attempt 2 onward, the step is declared stalled only when both signals stay pinned across the attempt: the count of resolved plan tasks did not increase, and HEAD did not move. The resolved count is the union of two sources:
- rows in
.pipeline/task-status.jsonwhosestatusiscompletedorskipped, and - plan task ids carried by
Task: <id>trailers on commits inmerge-base(origin/<default-branch>, HEAD)..HEAD.
So a task committed with a valid Task: trailer counts as resolved even if its row was never flipped. Work committed with no Task: trailer and no row update is invisible to the count — but not to the breaker, because it still moves HEAD.
Commit movement is the liveness authority (adr-2026-07-23-commit-movement-liveness-floor). The attributed-task count is advisory routing and telemetry; it never independently proves a build is stuck. An attempt that lands real, committed-but-un-trailered work moves HEAD while the count stays pinned, and is classified unattributed_progress — telemetry, not a stall — instead of no_task_progress. A SHA read that fails on either side of the attempt is treated fail-closed, degrading to the old count-only behavior. A genuine wedge (zero commits, count pinned) still stalls and halts exactly as before: count alone can never kill a build.
If the build step’s retry budget then exhausts with at least one HEAD-moving attempt, the run does not take the generic “retries exhausted” HALT. It routes into build_review — the sole completion authority — through the same seam a normally-completed build uses, recording the unresolved plan task ids in conduct-state.json (build_routed_reason). This route requires a clean worktree. That is not an always-pass: build_review re-grades the diff independently and can still FAIL the routed build, kicking it back to build under the same MAX_KICKBACKS_PER_GATE bound as any other build_review kickback. A build with zero commit movement across every attempt never routes.
active_stall
Symptom: the daemon log shows <step> active without movement for <N>m (<resolved>/<total>) · action <warn|end_attempt>, and events.jsonl has a build_active_stall event. With end_attempt, a build_stall event with reason active_stall follows.
Diagnosis: the provider is still writing heartbeats, but no task resolved and HEAD did not move for build_progress.active_stall_minutes. warn only reports it. end_attempt ended the attempt; that spends one ordinary retry, and a pinned retry from attempt 2 onward becomes no_task_progress.
Recovery: inspect the attempt’s transcript for a loop or an unreachable task. To end such attempts automatically, set build_progress.active_stall_action: end_attempt; to tolerate long non-committing work, raise active_stall_minutes. See build_progress.
Verification: later build_progress events show activity: active-committing, and the log stops repeating the active-stall warning.
Build halted with uncommitted paths
Symptom: .pipeline/HALT names one or more uncommitted paths, for example 2 uncommitted paths: src/a.ts, test/a.test.ts. The build cannot complete or use the exhausted- but-working route while those paths remain dirty.
Recovery: inspect the named paths, then either commit the intended work or discard only the named paths when it is not intended. Do not clear the halt first or re-dispatch around the dirty tree. Once the worktree is clean, use the resume procedure.
Verification: git status --porcelain in the feature worktree prints no output before the resumed build advances.
In daemon mode the engine synthesizes a remediation prompt into .pipeline/build-stall-question.md and dispatches /remediate, bounded to 2 remediation rounds per gate. A zero-work stall never terminal-HALTs from that path; it falls through to the ordinary retry and park route.
Unresolved step command
Symptom: .pipeline/HALT reads Cannot dispatch '<step>': /<command> is not available in the provider skill catalog. and its class is mechanical.
Diagnosis: the provider ran but reported the exact dispatched slash command as unknown — the run’s home is missing the skill that step renders to. This is a deterministic environment failure: retrying, escalating model or effort, or walking providers cannot make a command appear in a catalog that does not have it, so the conductor halts immediately instead of spending further attempts. Under Pi, the engine raises the same HALT before spawning pi when <name>/SKILL.md exists in none of ~/.agents/skills, $PI_CODING_AGENT_DIR/skills (default ~/.pi/agent/skills), or the project’s .agents/skills.
Recovery: re-provision the provider home (or self-host sandbox) with the missing skill, then clear the HALT using the resume procedure. Because this is a mechanical-class HALT, the daemon’s ordinary re-kick sweep also clears it on a base-branch advance — but re-kicking without fixing the catalog just reproduces the halt on the next dispatch.
Verification: after the next dispatch, the daemon log shows the step’s command resolving and the run advancing past it; it must not return to the same commandUnresolved HALT.
Unresolved build-review rubric skill
Symptom: the event log contains build_review_rubric_infrastructure_failure with reason invalid-provider-result. Its excerpt says that a build-review rubric skill could not be dispatched and names the unresolved command when the provider supplied it.
Diagnosis: the rubric produced no judgement. The provider reported that its dispatched skill command is unavailable, so the conductor stops that auxiliary member immediately instead of retrying it, requesting a judged-result repair, or trying another configured provider.
Recovery: relink or re-provision the provider skill catalog. If the feature branch predates the rubric skill, rebase it onto a base that contains the skill. Then use the resume procedure when the feature has a HALT; otherwise rerun the blocked build-review gate.
Verification: the next build-review event is a rubric result or another actionable infrastructure failure, not the same unresolved-command excerpt.
Provider preparation exhausted
Symptom: .pipeline/HALT begins Provider preparation exhausted. and its class is needs-human. The daemon log or dashboard may also show the provider lifecycle as recovering followed by halted with preparation-timeout-exhausted and recovery 1.
Diagnosis: read the marker body:
cat .worktrees/<slug>/.pipeline/HALT
cat .worktrees/<slug>/.pipeline/HALT.class
It names the logical step, phase: preparing, attempt id, elapsed milliseconds, and recovery count. Self-host dispatches waiting for root-mutation admission are queued outside this deadline; the daemon log and event ledger distinguish queued, admitted, and cancelled (operator park). Queue time consumes no preparation recovery attempt.
After admission, the timeout applies before a provider process starts. It means candidate resolution, session setup, or self-host preparation did not complete within provider_preparation_timeout_minutes. The first expiration already used the one automatic replacement; because this is a needs-human HALT, the re-kick sweep will not clear it.
Do not diagnose this from .pipeline/step-heartbeat. Heartbeats are activity telemetry only, and a quiet running provider is not automatically killed, retried, or replaced.
Recovery: repair the preparation failure. If preparation is expected to take longer, increase provider_preparation_timeout_minutes in the active configuration; 0 or a negative value disables that pre-spawn deadline. Do not use step_heartbeat_stall_minutes: it is a deprecated compatibility no-op and cannot change provider lifecycle behavior. Then clear the HALT using the resume procedure.
Verification: after the next dispatch, the dashboard/log shows a new preparing attempt and, once spawned, running; it must not return to halted with preparation-timeout-exhausted. Confirm the resumed feature with the general verification steps.
Live-boundary violation (self-host only)
If .pipeline/HALT reads <live checkout|provider state> changed during self-host execution — N added, N removed, N changed: …, the self-host live boundary detected a change to the harness checkout or to the operator’s real ~/.claude/~/.codex while a step was in flight. The reason names the paths — each tagged added, removed, or changed, capped at eight entries followed by and N more, with exact counts. Read them before investigating anything else.
First determine whether the completed dispatch was proven contained. A contained dispatch records self_host_containment_verdict and, when the live checkout changed, a contained_live_checkout_drift event in .pipeline/events.jsonl. That event names the containment evidence, labels the drift concurrent-operator, and does not write a HALT: the dispatch could not have made the live checkout writable. A clean contained dispatch still records its containment verdict. If the verdict says containment was unavailable, disabled, or its probe failed, the ordinary fail-closed behavior below applies.
- A path under the live checkout is usually an untracked file an operator session wrote — a permission or approval grant (Claude Code’s
.claude/settings.local.jsonis untracked and fingerprinted), a scratch or generated artifact, or a new file staged withgit add. Git can say a path is tracked but not who wrote it, so the guard fails closed on everything it cannot attribute. An edit or deletion of an already-tracked file reportsM/Dand does not halt; a git operation that rewrites tracked content without leaving it modified (git pull,git checkout,git stash) does halt, becausegit statusthen comes back clean. Batch root-checkout work between dispatches, or do it inside a worktree — see the live-checkout rule inAGENT_INSTRUCTIONS.md’s Daemon Operations Safety section. Issue #1301 tracks the attribution machinery that will remove this false-halt class. - A provider-state path that is config-like (
settings.json,config.toml,hooks.json) means an unrelated interactive session changed operator config mid-build. That trip is deliberate and fail-closed — those files stay fingerprinted precisely because a real leak would look identical. - A provider-state path that is pure telemetry or cache means the exclusion list needs a new entry; see self-hosting: provider-state exclusions.
EROFS during a contained dispatch: a Read-only file system failure usually means the dispatch attempted a legitimate live-checkout write that is absent from the shared LIVE_CHECKOUT_VOLATILE policy. Confirm the attempted path from the provider output and verify it is an intended, safe volatile surface. As a short-lived recovery, set harness_self_host.live_containment: false for the affected run; this restores the fail-closed boundary guard, so avoid concurrent root-checkout changes. Then correct the shared LIVE_CHECKOUT_VOLATILE policy and its containment bind set with focused tests; do not add an ad-hoc local carve-out. Resume the feature using the resume procedure.
Verification: after resume, inspect .pipeline/events.jsonl. A repaired contained dispatch has one self_host_containment_verdict with contained: true; concurrent live-checkout drift, if any, has one contained_live_checkout_drift record with attribution: "concurrent-operator" and its changed-path summary. A temporary opt-out instead must either remain clean or halt with the uncontained reason, never silently ignore drift.
This is a mechanical-class HALT, so the daemon’s ordinary re-kick sweep clears it on the next base-branch advance. The step that was running keeps its own real verdict, so the re-kick resumes after it rather than repeating it.
Retained draft PR identity is unavailable (self-host only)
Symptom: .pipeline/HALT begins Self-host release gate HALT: retained draft PR identity is unavailable — no OPEN pull request exists for <branch> into <base>.
Diagnosis: the release gate reads its release metadata from the draft PR opened at SHIP-phase entry. When the in-memory URL is absent — the ship-start git push failed, for instance — the engine looks the PR up by branch instead, and only an OPEN PR whose head is that branch and whose base is the build’s base branch satisfies it. Reproduce the same lookup:
gh pr list --head <branch> --base <base> --state open --json url,state
- Rows come back. The gate should have resolved it; capture the daemon log line
[ship-draft-pr] branch lookup for <branch> …and treat it as a bug. - Empty, but a PR exists in
--state all. It is closed or merged. That is not a draft the finish step may flip or rewrite — open a fresh PR for the branch, or land the work as already-shipped viaai-conductor shipped-record. - Empty entirely. No PR was ever opened; check whether the branch reached origin (
git pushfailures are logged as[ship-draft-pr] push of <branch> failed) and push it.
Recovery: ensure an OPEN PR exists for the branch into the base with valid release metadata in its body, then clear the HALT using the resume procedure.
FINISH publication halts
A FINISH publication failure or non-converging progress halts one of four ways, and the marker says which:
FINISH publication retry exhausted: <reason>— the reason was transient (transport, GitHub, filesystem, provider judgment, or a re-observation), so every attempt in the budget was spent before giving up.FINISH publication cannot proceed: <reason> is not retryable — …— the reason can never be satisfied by re-running the identical transition, so the run halted on the first observation and the retry budget was deliberately left unspent. Onlyauthor_pr_proseandjudge_pr_prosecross the provider boundary between attempts, so no retry authors a commit, wires a missing effect, or reconciles a remote. These aredraft_pr_no-commits,draft_pr_skipped,draft_pr_lease-rejected, and the five*_effect_unavailablereasons. Both halts areneeds-human; recovery is the same — resolve the cited condition, then clear the HALT.FINISH publication progress allowance exhausted after <N> transition(s); last transition: <transition>. Human review required.— FINISH made verified publication progress, so none of the step retry budget was spent. The separate progress allowance reached its 14-transition bound (two passes over each of the seven publication transitions) before the publication state converged, and this is aneeds-humanhalt. When the last prose transition follows a judged-deficient revision, the halt also ends withDetail: <judge objection>.- A plain-prose sentence with no
FINISH publication …prefix, e.g.The PR prose judgment was refused and requires an operator decision. Next action: Review the refusal and decide how to continue publication.and, when the provider supplied one, a trailingDetail: <provider text>. This is ahuman_requireddisposition — a condition only an operator can resolve, distinct from the three retryable/non-retryable shapes above. Its reasons cover judgment refusal or halt prose (judgment_refused,judgment_halt_prose), a PR that still carries aneeds-remediationtitle prefix, banner, label, or body marker (halt_state_pr— resolved before judgment is ever dispatched, so no provider pass is spent), a publication transition that ran but left its owned state unchanged (publication_transition_unmoved— the detail names the transition and the state that did not move, whether the transition reported success or asked for a retry it could not perform), an unresolvable PR match or shipped record (ambiguous_pr_identity,invalid_shipped_record), and destructive or unrecognized publication intent (interactive_intent_*,unattended_intent_*). It is always aneeds-humanhalt.
Diagnosis: inspect the named last transition and the preceding FINISH publication events in the daemon log. Fourteen verified transitions without convergence means the publication state machine is cycling or an external publication state is not settling; do not clear the HALT merely to repeat the same cycle. An author_pr_prose / judge_pr_prose alternation means the authoring pass keeps producing prose the judgment pass rejects. The Detail: text on a prose-related halt is the judgment for the exact rejected revision; use it with the current PR body to correct the prose before resuming.
Recovery: reconcile the cited transition and the external PR/remote state until the next FINISH entry can converge. Only then clear the HALT using the resume procedure.
An unrecognised reason always keeps its retries: the classifier fails closed toward retrying.
Clearing any FINISH publication halt is enough to get the feature back: see a cleared FINISH halt resumes to a recorded ship for what the daemon does next and how to confirm it. Never repair a halted FINISH by hand — an operator-opened PR is not a harness finish, and the daemon has no way to learn about it.
The post-FINISH shipment audit refused the ship
Symptom: the daemon logs ✋ <slug> false-ship halted — worktree kept (durable shipment evidence refused ship: <code> (expected <sha>, observed <sha>)), and .pipeline/HALT plus .docs/halted/<slug>.md carry the same sentence. The audit runs after FINISH converges: it re-proves the durable .docs/shipped/<slug>.md record against the commit the implementation PR will merge, and refuses every terminal ship side effect when that proof fails.
Diagnosis: the parenthesised pair is the whole diagnosis. expected is the head the PR reported (gh pr view <url> --json headRefOid); observed is the commit the audit validated. The same pair is on the event spine as shipment_evidence_refused and in .pipeline/audit-trail/events.jsonl, so it stays readable after the branch moves on. For shipment-candidate-not-on-implementation-head the two commits are unreachable from each other, or the head’s own tree carries no valid record — most often a record that was committed but never pushed. Confirm with:
git -C .worktrees/<slug> show <expected-sha>:.docs/shipped/<slug>.md
A local branch that is merely ahead of the reported head is not a refusal on its own: the conductor’s post-finish cost refresh commits and pushes after publication, and GitHub updates a pull request’s head asynchronously, so the audit accepts that shape whenever the head it names carries the record itself.
Recovery: push the record-bearing commit to the PR branch, then clear the HALT using the resume procedure. Never open or repair the PR by hand — see the manual-PR warning.
draft_pr_lease-rejected
Symptom: .pipeline/HALT reads FINISH publication cannot proceed: draft_pr_lease-rejected is not retryable — the remote branch carries commits this checkout has never observed ….
Diagnosis: establish_pr publishes the feature branch with git push -u origin <branch> --force-with-lease, because the finish-time rebase step has just rewritten that branch’s history — the branch diverges from its own remote by construction, and a plain push could never succeed. A rejected lease means something else: the remote branch carries commits this worktree has never observed, so the push was refused rather than overwriting them. (An ordinary transport or permission failure reports draft_pr_push-failed instead.)
git -C .worktrees/<slug> fetch origin <branch>
git -C .worktrees/<slug> log --oneline HEAD..origin/<branch>
- Rows come back. Real unseen work is on the remote. Integrate it (rebase the feature branch onto
origin/<branch>, or reconcile by hand) before letting FINISH retry. - Empty. The remote-tracking ref was merely stale; the fetch above refreshes the lease.
Recovery: once HEAD..origin/<branch> is empty, clear the HALT using the resume procedure. Never resolve this with a bare git push --force — that discards the very commits the lease protected.
halt_marker
The pipeline skill wrote .pipeline/halt-user-input-required — a genuine question that no retry will answer. Its content becomes the stall question:
cat .worktrees/<slug>/.pipeline/halt-user-input-required
cat .worktrees/<slug>/.pipeline/build-stall-question.md
The build completion gate returns “not done” while that marker exists, so a surviving marker also blocks the build gate directly.
ENVIRONMENT_CLAIM_REFUTED
A step failed with a reason beginning ENVIRONMENT_CLAIM_REFUTED. That is not an environment problem: the dispatch made an operation-specific claim that the environment blocked git push or gh, and the engine disproved it from the dispatch it actually performed (unsandboxed claude, and a write fence whose generated script carries no such rule). The failure message quotes the claim and states the facts.
Nothing needs fixing in the sandbox — do not go looking for one. The attempt is retried with the disproof as its retry hint so the step performs the operation for real. If the same refutation repeats until the budget is exhausted, the model is inventing the blocker rather than running the command: escalate the step’s model tier or take the finish over manually (a manual PR still needs its shipped-record, see shipped-record reconciliation).
Claims from codex are never refuted — its unattended runs really are sandboxed (sandbox_mode="workspace-write"), so a blocked operation there may be genuine.
A claim that the environment blocks all, every, or any Bash, shell, tool, or terminal command (or blocks everything) also does not produce this marker. The audit passes that broader claim through unchanged rather than selectively refuting a named operation in the same output.
Build-progress ceilings
A build that is resolving new tasks re-dispatches without consuming the fixed retry budget, bounded by the build_progress_halt block. Defaults: enabled, attempt_ceiling: 30, dispatch_ceiling: 20. Hitting the attempt ceiling parks with a distinct reason so you can tell “genuinely stuck” apart from “still progressing but out of runway”. Key details are in configuration.
A refunded build retry log line includes a separate allowance fragment, for example progress allowance: attempt 2 of 30. The fixed retry counter continues to describe the reused fixed-budget slot and stays within its own maximum; the allowance fragment tells you how much of the independent progress-attempt ceiling the running build has consumed.
A halt awaiting operator action is neither progress-re-kicked nor cleared by rate-limit episode recovery; the daemon log names the blocking halt disposition.
The ▶ build <resolved>/<total> line counts a task as resolved when its .pipeline/task-status.json row reads completed/skipped or a commit on the branch carries its Task: <id> trailer — the same union the build completion gate routes on. Nothing writes those rows back mid-build except the pipeline skill’s explicit ai-conductor task done, so on a run where the agent only stamps trailers the rows stay pending and the trailers are what moves the number. A resolved count that does not advance while commitCount keeps ticking therefore means work is landing without task attribution — check the commit trailers before treating it as a stall.
Rate limits
A rate-limited dispatch emits rate_limit and waits — to the deadline parsed from the provider message when one is available, otherwise 300 seconds. The wait does not burn the retry budget. HALTs written while a rate-limit episode is active are stamped so they can be recovered when the episode ends. A halt awaiting operator action is neither progress-re-kicked nor cleared by episode recovery; the daemon log names the blocking halt disposition.
Known limitation. The episode stamp is in-memory, scoped to the running daemon process. Restart the daemon during an episode and its episode-caused halts are no longer recognized as recoverable — they must be cleared by hand or by a base-branch advance. Tracked in #1023.
Auth parks
credentials_park means the run is waiting on a credential source, not stuck on your code. credentials_park_progress events report readiness probes as they happen. The park times out after harness_self_host.auth_park_timeout_minutes (default 60), then HALTs with an actionable reason naming the credential to refresh.
ai-conductor build-auth-status
Exit 0 means clean (state=api-key when there is no daemon-owned token, or state=valid). Exit 1 covers missing, unreadable, invalid, and unverifiable, and prints the remediation message with the token path.
Kickback loops
A gate that blocks routes work back to an earlier step. Each gate allows at most 2 kickbacks; past that the run halts instead of looping. Separately, a kickback into build that ends with zero net progress and an unchanged gate verdict escalates to a halt immediately rather than spending the remaining budget. Read the verdict that keeps blocking:
cat .worktrees/<slug>/.pipeline/gates/<step>.json
The record is { satisfied, reason, checkedAt, kickback }. reason is the exact string the gate computed. The concepts behind gate verdicts are in gates; the per-step evidence files are listed in artifacts.
For a budget-cap halt, do not remove HALT or HALT.class. From the main checkout, inspect the budget and authorize the intended recovery:
ai-conductor kickback-budget inspect --feature <slug>
ai-conductor kickback-budget raise --feature <slug> --gate <gate> --by <positive-integer> --rationale "<why more allowance is justified>"
# or
ai-conductor kickback-budget reset --feature <slug> --gate <gate> --rationale "<why this count may restart>"
When the budget view identifies exhausted plan growth, recover it with raise: it extends the growth allowance. reset is for lap evidence only and cannot recover a plan-growth halt.
A prd_audit or as-built kickback-cap halt fires at the build transition, after the repair’s tasks are already appended and pending. Once the authorization is consumed, the feature resumes into build on those tasks without re-running the audit. If it halts kickback-cap again with a new generation, the raise was too small or named the other gate: re-inspect and raise the exhausted allowance.
The daemon clears only a live halt whose gate and generation match that one-use authorization. It does so on its next loop iteration, without waiting for a base-branch advance. If the feature was already operator-parked, unpark it after the command; otherwise it remains intentionally halted.
If the halt persists and inspect reports Resume authorization: stale, the authorization was bound to an earlier halt generation; the daemon log records it as stale and not consumed. Run raise or reset again against the live halt — the new authorization replaces the stale one. The full command contract is in the CLI reference.
BUILD verification after a repair
Symptom: a repair returned to BUILD and you need to determine whether test_suite reused evidence or derived it again.
Diagnosis: read the feature narrative, not merely the old gate files:
ai-conductor daemon logs | grep 'BUILD member .* settled:'
The daemon writes BUILD member test_suite settled: reuse (<basis>) or ... recompute (<basis>). reuse (fingerprint-match) reports still-valid full-suite evidence. A recompute line reports a closed reason such as fingerprint-mismatch or fresh-evidence-required. A prior passing suite result on disk is not a reason to skip the verifier; its current result decides satisfaction.
Recovery: do not create an operator park for the historical terminal-less stale-verdict path; it is retired. The common stale-proof case re-dispatches test_suite automatically, so let the re-verification round settle. If build_review instead HALTs needs-human because its inputs cannot change, rewind from the feature worktree rather than editing pipeline state:
cd .worktrees/<slug>
ai-conductor rewind --to test_suite
rewind clears the affected gate verdicts and both halt markers only after it has atomically demoted the feature to test_suite. It refuses a target that is not earlier than the recorded step. Do not hand-edit conduct-state.json, gate files, HALT, or HALT.class.
Verification: the log contains one test_suite settle line, followed by build_review or an explicit HALT. After a rewind, confirm it also prints Rewound to test_suite. and that the next dispatch starts at test_suite.
Build review cannot resolve the feature plan
Symptom: .pipeline/HALT begins build_review cannot resolve a plan for feature and lists candidate plan stems. Its class is needs-human; no provider was dispatched and no .pipeline/build-review.json verdict was written.
Diagnosis: More than one plan exists, but neither a recorded active plan nor a plan whose stem equals the feature slug identifies this feature’s plan. Read the named candidates; do not select one because of its alphabetical position.
Recovery: Restore or correct the feature’s authoritative plan through the normal DECIDE process. When several plan files remain, its filename stem must equal the feature slug. Then rerun the preceding verifier from the feature worktree:
cd .worktrees/<slug>
ai-conductor rewind --to test_suite
rewind clears the halt and stale gate state atomically, then runs test_suite before a fresh build_review. Do not delete the halt markers by hand or rename an unrelated plan to force a match.
Verification: The log shows Rewound to test_suite., then a test_suite settle line followed by build_review; the new review either has the resolved feature plan as input or reports an independent gate result.
Full-suite verification lock remains occupied
Symptom: test_suite reports Unable to acquire full-suite verification lock within 30000ms.
Diagnosis: the verifier serializes suite execution with .worktrees/<slug>/.pipeline/test-suite.lock. A dead lock owner is recovered automatically. If that recovery was interrupted, its recovery.json claim is also reclaimed when its recorded process is dead, or when unparseable claim bytes are at least five minutes old. A live claim and a fresh unparseable claim remain exclusive.
Recovery: rerun the verification or let the daemon’s next BUILD attempt run it. Do not delete the lock directory or its claim: a live verifier may own it. If verification instead reports an error naming a recovery claim, preserve the directory and diagnose the filesystem or process-probe failure before retrying.
Verification: the next test_suite run either settles normally or reports its actual suite failure; it does not remain blocked by a dead owner or reclaimable stale claim.
Setup failures
If the project’s bin/setup failed inside the worktree, the feature may be quarantined: .pipeline/QUARANTINE exists and a wip/setup-quarantine-<slug> branch holds the evidence.
Automatic setup-triage park
Symptom: a setup failure’s triage ends with a retained worktree and a main-root marker at .daemon/parked/<slug> whose first line begins auto-parked:. This is a durable automatic park, not a missing dispatch or a transient in-memory daemon state.
Diagnosis: read the first line of .worktrees/<slug>/.pipeline/HALT together with the marker. The first line has one of three meanings:
feature parked — will not re-dispatch on the next scan— the automatic marker was written (or was already present), so the daemon will keep the feature excluded.feature errored — automatic park failed: …; run ai-conductor daemon park <slug>— the marker could not be written. Repair the reported filesystem error, then park it explicitly before changing its worktree state.feature errored — will re-dispatch on the next scan— this was a non-park termination, so no automatic marker suppresses the next scan.
Recovery: fix the underlying setup problem first. Clear the feature’s HALT and HALT.class using the resume procedure, then remove the automatic marker and restore dispatch eligibility:
ai-conductor daemon unpark <slug>
Verification: test ! -e .daemon/parked/<slug> succeeds. The next daemon scan logs ↻ resume <slug> or starts the feature, rather than another parked skip.
3. Re-verify SHIP evidence with --diagnose
--diagnose re-runs the SHIP-gating completion predicates — test_suite, manual_test, finish — against the on-disk evidence and reports which ones cannot reproduce a pass. It does not modify feature state.
ai-conductor inline --diagnose "<feature description>"
Run from the main checkout with the feature description, or from inside the worktree with no description at all (ai-conductor inline --diagnose).
| Outcome | Output | Exit |
|---|---|---|
| Evidence consistent | State OK: … has consistent SHIP-phase evidence. | 0 |
| No state for that description | No conductor state found for "<desc>" — nothing to diagnose. | 0 |
| Evidence gaps | Per-step gap report on stderr, plus remediation guidance | 1 |
State says past worktree but no worktree exists | Orphaned conductor state in <path>. | 1 |
A non-zero exit here is the precise signal that the state is marked complete while the evidence that would justify it is missing. That combination is what silently produced “complete” features with no PR.
4. Read the run timeline with --report
ai-conductor inline --report
Read-only. Renders four tables from .pipeline/events.jsonl — Step Durations, Retry Hotspots, Token Spend, and Kickbacks by Source Gate — then exits 0. An unreadable events log exits 1. Run it from inside the worktree; it reads .pipeline/ relative to the current directory.
Known limitation.
--reportdoes not render halt tables, althoughloop_halt,rebase_conflict_halt, andhalt_marker_write_failedpersist inevents.jsonl. For halt occurrences, usecost-rollup.halts, the shipped record’s## Costblock,ai-conductor kpi, or the engineer-loop signal assembler; use.pipeline/HALTas the durable park state and.pipeline/gates/<step>.jsonfor the gate verdict. Tracked in #1023 and #1008.
5. Read the daemon’s own narrative
ai-conductor daemon logs --lines 200
.daemon/daemon.log carries every dispatch line, per-step result, engine warning, and the startup dashboard snapshot.
Known limitation. The log rotates to
.daemon/daemon.log.1only when it is reopened and already exceeds 1 MB, and no CLI reads the rotated file. A long-running daemon never rotates mid-run; once it does, the previous history is only reachable by opening.daemon/daemon.log.1directly. Tracked in #1008.
Recovery
Pick the branch that matches the diagnosis. Every step says what it changes.
The stall was a false negative — real work exists
The commits are on the branch but carry no Task: trailer, so the resolved-task count cannot see them and the build completion gate keeps reporting those ids as pending. (The stall breaker itself is not fooled — those commits move HEAD — but the count still drives the gate.) Do not re-run the tasks. Make the work visible instead:
- Confirm the commits exist and check their trailers:
git -C .worktrees/<slug> log origin/<base-branch>..HEAD \ --format='%h %s%n Task: %(trailers:key=Task,valueonly)'Substitute your repository’s default branch for
<base-branch>. A blankTask:line means that commit is invisible to the stall breaker. - Flip the corresponding rows in
.worktrees/<slug>/.pipeline/task-status.jsonto"status": "completed". What it changes: the routing input for the build gate. A row already markedcompletedorskippedis preserved verbatim across every re-seed, so this edit survives. - Confirm the gate now sees them:
ai-conductor inline --diagnoseand re-read
.pipeline/gates/build.jsonafter the next dispatch — itsreasonshould no longer list those task ids as pending.
ai-conductor task done <id> will not help here: it clears the .pipeline/current-task stamp and never modifies task-status.json.
The stall was real — the build genuinely cannot proceed
- Read
.pipeline/build-stall-question.md(or.pipeline/halt-user-input-required) and act on the question: fix the spec, the plan, or the code. - Remove the user-input marker if it is still present. What it changes: unblocks the build completion predicate, which returns “not done” while the file exists.
rm -f .worktrees/<slug>/.pipeline/halt-user-input-required - Clear the halt (next section).
The halt refused a DECIDE entry
Symptom: .pipeline/HALT begins DECIDE entry refused and .pipeline/HALT.class is needs-human. The body names the source gate, requested target, evidence, and why autonomous entry was refused. The daemon will not re-kick this class of halt.
Recovery: Read that body first. Correct an unknown target, missing artifact, or wrong routing before authorizing anything. When the named DECIDE authoring pass is the intended operator decision, run these commands from any directory inside the repository:
ai-conductor decide-grant --slug <slug> --step <target-from-HALT> \
--reason "<why this one DECIDE entry is approved>"
rm -f .worktrees/<slug>/.pipeline/HALT .worktrees/<slug>/.pipeline/HALT.class
The first command resolves the main checkout and writes a durable grant for only that target into its daemon-owned .daemon/grants/<slug>.json. Outside a repository, it refuses without recording a grant. The conductor consumes it immediately before provider dispatch, so it cannot authorize another DECIDE step or a later retry.
Never hand-write a grant file. The grant deliberately lives outside the feature worktree: a decide-grant.json written inside .worktrees/<slug>/.pipeline/ authorizes nothing, because that directory is the build agent’s own scratch space and an agent must not be able to authorize itself. Use the command from any directory inside the repository; it resolves the main checkout itself.
plan is never grantable. ai-conductor decide-grant --step plan exits non-zero, and the entry policy refuses plan before consulting any grant. If the HALT names plan as the requested target, the correct recovery is to drive the plan revision yourself — interactively, with /conduct or by editing the plan — and then clear the halt so the feature resumes into BUILD. See the plan-revision recovery below.
Clearing the HALT alone does not authorize entry. Without a valid matching grant, the next poll writes the same needs-human HALT and launches no provider. A grant for stories, for example, does not authorize architecture_review.
Verification: After the daemon resumes, confirm the one-use artifact was consumed and then inspect the result of the named step:
test ! -e .daemon/grants/<slug>.json
ai-conductor daemon logs | grep '\[<slug>\]'
A halt requests a plan revision
Symptom: .pipeline/HALT is needs-human and names plan as the requested target — typically from a remediate or build_review disposition asking for a DECIDE revision.
There is no grant for this. The daemon may not re-plan under any authorization. Recover by hand:
- Read the halt body and the remediation that produced it (
.worktrees/<slug>/.pipeline/remediation.json) to see which plan change is actually being asked for. Verify the request against the gate evidence — a remediation whose fix would re-trigger the gate that caused it is wrong, and amending the plan to match it makes things worse. - Make the plan edit yourself in the feature worktree. When BUILD discovered that an approved DECIDE assertion must change, add the correction beside the original rather than rewriting it — see amendment requests above. Story artifacts under
.docs/stories/are the exception: replace the superseded assertion in place instead, with no amendment note. - Rewind so the feature resumes into BUILD and actually builds the revised plan, from inside the feature worktree:
cd .worktrees/<slug> && ai-conductor rewind --to buildai-conductor rewindmarksbuildand every later non-skipped step stale, clears their gate verdicts, and clearsHALTandHALT.classatomically. Do not delete the halt markers by hand: that leaves the already-recordedbuild,test_suite, andbuild_reviewverdicts standing, so the revised plan task is never built. If the plan you edited is sealed — any plan amended after first BUILD entry is — reseal it first, per the protected-artifact recovery.
If the feature is parked while you do this, unpark it last — the daemon re-dispatches as soon as it is eligible.
If the grant remains, the feature did not enter the authorized step; re-read the HALT rather than clearing it again.
The halt is a plan gap
Symptom: .pipeline/HALT.class is plan-gap. prd_audit or the as-built architecture review found an outcome the shipped work does not deliver and no active plan task owns the repair. The approved design is the ceiling here: the daemon retains this halt on every sweep, and clearing the marker without amending an artifact just re-halts on the same criterion. A committed record of the halt is on the feature branch at .docs/halted/<slug>.md; clearing the halt flips its status to resolved.
Blast radius: the amendment rewrites committed DECIDE artifacts on the feature branch and re-runs BUILD and every later step. Nothing outside the feature moves.
- Read the gap.
.pipeline/prd-audit.md(its per-criterion detail section) or.pipeline/architecture-review-as-built.md(its## Recorded Findings) names each criterion and why no active plan task owns the repair. - Amend the artifact the gap actually indicts, in the feature worktree — usually the plan (add a task naming the omitted work), sometimes the story (when the criterion overpromises what the approved design can deliver). Update the feature’s coherence table task and criterion rows in the same commit, and commit inside the worktree.
- Reseal the amended protected paths from the main repository checkout, not the worktree — a plan or story committed after first BUILD entry leaves the seal baseline stale:
ai-conductor reseal --slug <slug> --path <path> [--path <path> ...] \ --reason "<why this amendment is approved>"ai-conductor resealis TTY-only by design; run it interactively. See the protected-artifact recovery for what it checks. - Rewind, from inside the feature worktree:
cd .worktrees/<slug> && ai-conductor rewind --to buildai-conductor rewindmarksbuildand every later non-skipped step stale, clears their gate verdicts, and clearsHALTandHALT.classatomically. Clearing the markers by hand instead leavesbuild,test_suite, andbuild_reviewrecorded as done, so the amended plan task is never built.
Two plan gaps need no artifact amendment:
- The gap is reader-facing documentation the plan deliberately delegated to the
maintain-documentationgating step. That step runsafter: rebase, downstream of the audit that blocks on the criterion, so it can never clear its own gap. Make the documentation edits by hand — documentation pages are not protected artifacts, so no reseal is needed — then clear the halt by renaming:mv .worktrees/<slug>/.pipeline/HALT .worktrees/<slug>/.pipeline/HALT.cleared rm -f .worktrees/<slug>/.pipeline/HALT.class - The gap needs machinery another, unmerged feature owns. Leave this feature halted and sequence the two. Amending this feature’s plan to build the other feature’s deliverable is wrong.
How to confirm: the next dispatch re-enters BUILD rather than re-writing the same halt, and the audit that raised the gap passes its criterion on the following lap. `git show