Troubleshooting

Start with the local doctor from the repository checkout:

gitmoot doctor --repo .
gitmoot status --repo owner/repo
gitmoot daemon status

Most Gitmoot failures come from one of four places: the installed binary, GitHub CLI auth, runtime/plugin discovery, or a local daemon/job/lock state.

Local Review Lead Is Refused

Symptom: gitmoot agent review or a review-resolved gitmoot agent run refuses before creating a job because the lead is missing, cannot access the repo, lacks implement, or has a non-write autonomy policy.

Likely cause: a review-only agent was dispatched without --lead, or the named implementer is not dispatchable from its agents-database row.

Fix: keep the reviewer read-only and name a separate registered implementer:

gitmoot agent review reviewer --repo owner/repo --pr 12 --lead implementer "Review this PR."
gitmoot agent show implementer
gitmoot agent repos implementer

The implementer must be allowed on the repo, hold implement, and use workspace-write or danger-full-access. Do not grant implementation access to the reviewer merely to bypass the preflight.

Install Script Failed

Symptom: curl -fsSL https://gitmoot.io/install.sh | sh exits before installing gitmoot.

Likely cause: network failure, unsupported platform, missing shell tools, or a release artifact that is not available for the current OS/architecture.

Check:

uname -s
uname -m
curl -fsSL https://gitmoot.io/install.sh -o /tmp/gitmoot-install.sh
sh -n /tmp/gitmoot-install.sh

Fix: retry the installer or use the direct binary fallback from the install page. Verify the artifact checksum before running it.

Binary Not On PATH

Symptom: gitmoot: command not found after install.

Likely cause: the install directory is not on PATH, or the shell has not been restarted after pipx ensurepath or installer profile changes.

Check:

command -v gitmoot
echo "$PATH"
ls -l ~/.local/bin/gitmoot

Fix: add the install directory to PATH, restart the shell, or move the binary to a directory already on PATH.

Checksum Mismatch

Symptom: the local SHA256 does not match the release checksum.

Likely cause: partial download, wrong artifact, stale checksum, or a tampered download.

Check:

sha256sum <artifact>
shasum -a 256 <artifact>

Fix: delete the file, download the artifact again from GitHub Releases, and compare against the checksum for that exact release and platform.

GitHub CLI Auth Fails

Symptom: PR comments, issue comments, review publication, status checks, or merge actions fail.

Likely cause: gh is not installed, is authenticated as the wrong account, or does not have access to the repo.

Check:

gh auth status
gh repo view owner/repo --json nameWithOwner
gh pr list --repo owner/repo --state open

Fix: authenticate gh for the account that can read and write the repository, then retry the Gitmoot operation.

Send A Gitmoot Bug Report

Symptom: a Gitmoot job failed, blocked, or was cancelled, and you want to send a useful report without exposing raw runtime output.

Likely cause: the failing job has local context that matters for debugging: repo, agent, runtime, action, task, selected error, result summary, and recent events.

Check:

gitmoot job show <job-id>
gitmoot report bug --job <job-id> --preview

Fix: preview first. The report is redacted, omits raw runtime output by default, adds the gitmoot-dashboard-report and bug labels, and includes a fingerprint marker so open duplicates can be reused.

Create the GitHub issue only when you intend to file it:

gitmoot report bug --job <job-id> --create --yes

The command prints either created issue: ... or existing issue: ...; use that URL when sharing status.

Plugin Doctor Fails

Symptom: Codex or Claude Code does not discover Gitmoot, or gitmoot plugin doctor reports missing package or runtime state.

Likely cause: plugin package was not generated, runtime CLI is missing, runtime uses a different home directory, or the package cache is stale.

Check:

gitmoot plugin doctor
gitmoot plugin doctor codex
gitmoot plugin doctor claude
gitmoot plugin path codex
gitmoot plugin path claude

Fix:

gitmoot plugin install codex --force
gitmoot plugin install claude --force
gitmoot plugin doctor

The plugin is discovery and guidance. It does not replace gitmoot, GitHub CLI auth, or runtime/model credentials.

Runtime Session Not Found

Symptom: gitmoot agent doctor <name> cannot validate a Codex, Claude, or Kimi session, or a job resumes the wrong session.

Likely cause: a last reference changed, the runtime home changed, or the session id is stale. For a Kimi agent, the runtime reference must be a Kimi session id (session_<uuid>) or empty.

Check:

gitmoot agent list
gitmoot agent show <agent>
gitmoot agent doctor <agent>
codex exec resume --help
claude --help
kimi --help

Fix: prefer explicit session UUIDs or thread names over last, then re-subscribe the agent with the correct session reference — or, for a registered agent whose session is genuinely dead or stranded, rebind it in place without re-registering:

gitmoot agent restart <agent>

agent restart abandons the old runtime session and starts a fresh one for the same agent; it refuses while the session is live or the agent has in-flight jobs (finish or cancel those first).

Note that some session failures self-heal without your intervention: a dead Claude --resume target is retried on a fresh session and re-pinned (#443), and a transient 401 ("socket connection closed unexpectedly") under sustained concurrency is retried with backoff (#487/#509). A job whose events show one of these errors followed by a success worked as designed.

Daemon Not Running

Symptom: queued jobs do not move, PR comments are not consumed, or dashboard shows the daemon as down.

Likely cause: daemon was never started, exited, or is running with the wrong repo/home.

Check:

gitmoot daemon status
gitmoot daemon logs
gitmoot status --repo owner/repo

If daemon status warns that the advertised log is missing or was last written before the running daemon started, the path is not receiving current output. gitmoot doctor reports the same confirmed condition as a non-fatal daemon log check. If the daemon runs under systemd --user, follow the likely live stream instead:

journalctl --user -u gitmoot-daemon -f

The warning does not assert how the daemon was launched. It is omitted when the daemon is stopped, the log is current, or the comparison cannot be made.

Fix:

gitmoot daemon start --poll 30s --workers 1

Use gitmoot daemon run only when you intentionally want a foreground process.

--repo owner/repo scopes the daemon to a single repo: it polls only that repo's PRs and claims only that repo's queued jobs. Omit --repo to supervise every enabled registered repo from one daemon (#581). If queued jobs for one repo never move while another repo's jobs do, check whether the daemon was started with --repo scoping it to a different repo, or with a --session <root-job-id> (alias --root) filter: a daemon started with --session runs only jobs whose root_job_id matches that orchestration run plus the root coordinator job itself. Restart it without --repo/--session to drain unrelated jobs. Also check the repo is enabled in gitmoot repo list.

Claude auth is independent of daemon restarts. Rotate the owner-only runtime-auth.env with gitmoot auth set claude; the next delivery observes it. Inspect masked sources with gitmoot auth status and validate them with gitmoot auth probe claude or gitmoot doctor.

Daemon Already Running

Symptom: gitmoot daemon start/run refuses with daemon already running with pid ….

Likely cause: a daemon is already supervising this Gitmoot home. One daemon per home is enforced with a pidfile plus a flock backstop (#550/#556); starting a second one is refused by design (a stale pidfile whose owner is dead is liveness-checked and recovered automatically, so restarts after a crash work).

Fix: use the running daemon — it supervises all subscribed repos. To change its settings, send kill -HUP <pid> for a live [daemon] config reload (#577), or use gitmoot daemon restart. Scripts that start daemons should treat this refusal as success, not an error.

Job Stuck Or Failed

Symptom: a job is queued, blocked, failed, or no longer changing state.

Likely cause: runtime delivery failed, worker is read-only, GitHub auth failed, or another lock is active.

Check — read the stuck reason first:

gitmoot job list --repo owner/repo   # WHY: column on queued/blocked jobs
gitmoot job show <job-id>            # why_stuck: / next_retry_at: lines
gitmoot job events <job-id>
gitmoot agent show <agent>
gitmoot lock list --repo owner/repo

A job that is simply WAITING now says so in its own record (#1553). Every dispatch pass that examines a queued job and declines to claim it writes a dispatch_held_back event naming the reason: an admission-budget refusal, a checkout held by an in-flight job, a runtime session lock and its holder, or - when the pass cannot attribute the wait - the bare fact that the job was not selected this pass. Before this, that reason went only to daemon stdout, so gitmoot job events could not tell a job waiting two hours from one about to start. The event is throttled to one row per job per distinct reason every five minutes, so a long wait produces a readable handful of rows rather than one per tick.

gitmoot job list appends a WHY: column and gitmoot job show prints a why_stuck: line for queued/blocked jobs (#552) — e.g. a runtime-session lock wait (naming the holder), blocked: awaiting human, auth failing: …, throttled: …, retrying: …, or a blocked-operational: <class> deferral with the attempt schedule. A deferral that needs a human (dirty/wrong-head checkout) also prints a suggested_action naming the fix.

Deferred jobs recover on their own (#532): a delivery failure classified as a retryable operational blocker — runtime_auth, runtime_quota, network_outage, or checkout_contention — is re-queued with a bounded retry budget instead of failing terminally. job show --json carries the blocker_class, attempt count, and suggested_action. A runtime_auth deferral only re-dispatches once a live doctor-style credential probe passes (a failing probe extends the hold without spending a retry), and over [events] the deferral is a first-class job.deferred emitted instead of job.failed. A job that "failed then reappeared as queued" is the deferral working; only act when the retry budget is spent and the job stays failed.

A Claude read-only seat (every review/ask job under the read-only autonomy policy) does not rely only on its staged credential snapshot. It stages the runtime config dir (payload.runtime_config_dir, else the daemon's CLAUDE_CONFIG_DIR, else ~/.claude) and also carries the resolved runtime-auth.env overlay. When that snapshot is already expired the job records one readonly_seat_credential_expired event naming the expiry, the refresh-token state and whether an overlay was available; read it with gitmoot job events <job-id> before treating the runtime's own "OAuth session expired and could not be refreshed" wording as an account problem. It never refuses the job. gitmoot doctor and gitmoot auth probe claude both report that seat credential beside the ambient one, and doctor FAILS (non-zero exit) when it is expired with no refresh token, since every read-only seat job on that runtime will fail until the account is re-logged in.

An OMP read-only seat uses no operator profile or ambient provider key. Set OMP_AUTH_BROKER_URL to an HTTPS or loopback HTTP origin and point OMP_AUTH_BROKER_TOKEN_FILE at an owner-only regular file; the daemon refuses an upstream OMP_AUTH_BROKER_TOKEN because every seat must read /proc for runtime startup. The job's effective model must be provider-qualified, such as kimi-code/k3. Gitmoot writes a random job-local token and loopback URL only into the seat's minimal config.yml; no broker secret enters the child environment. The proxy exposes only that provider, forwards refreshes only, and keeps broker writes local to the disposable client. The daemon-staged OMP binary receives job-private PI_CONFIG_DIR and PI_CODING_AGENT_DIR roots. Missing or unsafe broker configuration refuses before staging; an unqualified effective model refuses before OMP launches. The state and proxy are removed at the job boundary.

When broker and clients share an OS account, run auth-broker serve with a dedicated relative PI_CONFIG_DIR and point OMP_AUTH_BROKER_TOKEN_FILE at that directory's auth-broker.token. Otherwise both roles use ~/.omp/auth-broker.token, so a client token can invalidate the service.

A kimi credential that goes blank mid-session is REPORTED, not repaired (#1856). Kimi's credential (~/.kimi-code/credentials/kimi-code.json) has been observed blanked IN PLACE after a failed refresh: still-valid JSON, 136 bytes, with access_token and refresh_token empty strings and expires_at 0, while scope and token_type stay populated. Gitmoot writes that file NOWHERE — the only writer is the vendor CLI — so it cannot make the write atomic, and a safety net whose failure mode is "gitmoot corrupted the owner's credential" would be worse than the defect. Instead the two paths that run a kimi child against the LIVE profile — a non-seat delivery, and gitmoot agent doctor's health check — read the credential immediately before and after the child and report a degradation: the delivery records one kimi_credential_degraded job event (on success and failure alike, because the measured case ran to success), and doctor prints the same observation on stderr since a command has no job row. It fires only on a credential that demonstrably carried a token before and demonstrably carries none after, so a host that is simply logged out stays silent, and a reading gitmoot cannot interpret (unreadable file, unmeasured schema) is never treated as evidence in either direction. Read it with gitmoot job events <job-id>. Attribution is inferred, not confirmed: the event records that a kimi child ran while the file changed, never that this child wrote it. A read-only seat is deliberately NOT observed — it reads a staged clone inside its own writable cache root, which the child may legitimately rewrite, and gitmoot never grants a seat read or write access to the live profile.

A Claude produce stage does NOT run against the operator profile. The daemon copies .credentials.json and settings.json from the configured account (CLAUDE_CONFIG_DIR, else ~/.claude) into a job-private profile, points the runtime's CLAUDE_CONFIG_DIR and XDG_CACHE_HOME at that job-private state root under <gitmoot home>/cache/produce-runtime/<hash>/run-*/ — one directory per dispatch, so two runs of one job id cannot wipe each other — and removes it when the job finishes. The operator profile is never granted writable and is never named in any read grant, so a token the runtime refreshes mid-job lands in the discarded copy: re-login on the host, not in the job. It may still be READABLE: an agent that declares no readable paths falls back to a read-only grant over /, which is a read, never a write.

Only those two files cross into a produce job. Everything else in the operator profile — agents/, commands/, plugins/, CLAUDE.md, settings.local.json, ~/.claude.json — deliberately does not, so a produce job sees runtime defaults rather than operator customisation. A profile that does not exist yet is valid and starts the job with an empty one. A settings.json that is symlinked is followed; one that is empty, holds a non-object, or is not a file at all is SKIPPED rather than failing the job. An unusable .credentials.json does fail the job, because it decides which account the work runs as.

sandbox-exec: sandbox read path "…": no such file or directory means a path granted to the sandbox does not exist on disk. Explicit read grants are required, not skipped — check readable_paths in gitmoot job show <job-id> --json and create (or stop granting) the named path.

A job stuck in running is recovered automatically once it shows no lease progress past the staleness window (default 30m; tune with the GITMOOT_STALE_RUNNING_AFTER environment variable; the smallest honored value is 1m — below-1m, malformed, or non-positive values are rejected in favor of the 30m default rather than clamped, #560). The window is a same-boot crash backstop, not a timeout: a job holding a runtime session lock whose lease has not elapsed is left running regardless of the window. After a reboot there is no wait at all — the kernel boot id changes, so on its next startup and every tick the daemon immediately requeues every job claimed on the previous boot and reclaims its stranded runtime session lock, regardless of any unexpired lease (#651). Boot-aware recovery is Linux only; elsewhere recovery falls back to the lease/age window above.

Fix: resolve the underlying runtime/auth/lock issue, then retry when safe:

gitmoot job retry <job-id>

Cancel (abandon) only when the job should not continue. Cancel now dismisses a blocked job — one paused awaiting a human — as well as a queued/running one (#631); a dismissed job is not lost, since gitmoot job retry accepts a cancelled job and resurrects it:

gitmoot job cancel <job-id>

A backlog of blocked jobs never clears on its own. Clear a stale batch with the bulk form, which is a dry-run by default (it prints id/agent/repo/age and cancels nothing) until you pass --yes:

gitmoot job cancel --state blocked --older-than 7d          # preview the selection
gitmoot job cancel --state blocked --older-than 7d --yes    # cancel it

Only blocked is accepted for --state; narrow the selection with --older-than (a Go duration like 168h, or a <N>d days suffix), --repo owner/repo, and --agent name. gitmoot doctor warns when blocked jobs older than 30d have piled up and prints the exact command to dismiss them.

To sweep the backlog automatically, set [orchestrate].blocked_ttl to a positive Go duration (e.g. blocked_ttl = "168h"): the daemon then dismisses any blocked job idle longer than the TTL through the same cancel path, recording a blocked_ttl_expired job event. It is off by default (empty or 0s disables it; a negative value is rejected), because a blocked job is a human-awaiting decision that is never auto-discarded unless you opt in. This is the single-job counterpart of [orchestrate].escalation_ttl, which auto-finalizes a whole paused delegation tree and is on by default (24h).

Stale Lock

Symptom: implementation or merge work is blocked by a lock whose owner is gone.

Likely cause: a worker died or the daemon stopped before cleanup.

Check:

gitmoot lock list --repo owner/repo
gitmoot lock show owner/repo <branch>

Fix: let a running daemon reclaim stale resource locks automatically. Task lane locks are released only after they have been unchanged for at least 24 hours and no non-terminal task or job references the same repository and branch. Human- resumable blocked, awaiting_human_merge, and awaiting_human tasks retain their locks. Release a branch lock manually only after confirming the owner is no longer working:

gitmoot lock release owner/repo <branch> --owner <agent>

Merge Gate Deferred By An Active Branch Job

Symptom: a ready PR remains unmerged and the gate reason says an active job is in flight on its branch.

Likely cause: an ask, review, or implement job targeting that PR branch is still queued or running. Gitmoot treats this as a transient deferral so it cannot squash-merge, delete the source branch, and strand an in-progress fix. The task stays ready_to_merge rather than entering a blocked state or emitting a merge-gate error.

Check:

gitmoot job list --repo owner/repo
gitmoot job show <job-id>
gitmoot job events <job-id>

Fix: let the job settle or cancel it deliberately. The daemon re-evaluates the unchanged policy merge path on its next tick; do not release its branch lock while the job is active.

Gate Status Says A Head Was Never Cleared

Symptom: a pull request carries gitmoot/merge-gate = pending with the description Gitmoot merge gate has not cleared this head.

Likely cause: this is the intended signal, not a failure. GitHub's mergeStateStatus reports the absence of blockers, so a head Gitmoot has never evaluated used to read exactly like one it approved. Gitmoot now marks an active managed head as unjudged until its own gate produces a verdict.

Check:

gh pr view <number> --repo owner/repo --json statusCheckRollup,mergeStateStatus
gitmoot task list --repo owner/repo

Fix: nothing to do. A later gate evaluation replaces the marker with the specific pending, failure, or success verdict for the same head. Gitmoot marks heads only while it owns the merge decision: with [merge_gate] auto_merge = false, with GITMOOT_DISABLE_NATIVE_MERGE_GATE=1, or once a task reaches awaiting_human_merge, dismissed, superseded, stranded or merged, it replaces only its own generic marker with Gitmoot merge gate is not applied to this head and leaves any real gate verdict untouched. A blocked or awaiting_human task keeps the marker, because that head genuinely has not been cleared and Gitmoot can still resolve it when the task resumes. A draft pull request keeps the marker until it is undrafted.

Reviewed PR Is Behind Main

Symptom: a reviewed pull request is behind its base. Gitmoot either merges the reviewed head without a branch update or blocks the merge while preserving that head.

Likely cause: both outcomes protect the exact-head review. Gitmoot merges the reviewed head as it stands only when GitHub reports it mergeable and the base explicitly permits behind heads. A branch update would create a new head and invalidate the approval.

Check:

gh pr view <number> --repo owner/repo --json headRefOid,mergeable
gh api repos/owner/repo/branches/<base>/protection --jq '.required_status_checks.strict'

Fix: if the base requires an up-to-date head, mergeability is conflicting or unknown, or protection cannot be determined, update the branch explicitly and obtain a new exact-head review. /gitmoot merge bypasses the automatic-merge kill switch, but it never updates a reviewed head.

Live Docs Or LLM Context Stale

Symptom: gitmoot.io/docs or /llms.txt does not show current source docs.

Likely cause: docs were changed but not rebuilt/deployed, or stale deployed files were not deleted.

Check:

cd website
npm run build
curl -fsS https://gitmoot.io/docs/reference/cli | rg 'gitmoot dashboard'
curl -fsS https://gitmoot.io/llms.txt | rg 'Dashboard|Release Notes'

Fix: deploy the current static build with delete semantics:

cd /root/gitmoot/website
npm run build
rsync -a --delete build/ /var/www/gitmoot-docs/

Worktrees consume too much disk

Every five minutes, the daemon checks task-owned worktrees whose task is merged, dismissed, superseded, or stranded. Age alone never qualifies a task worktree. Gitmoot retains it unless the recorded path is deterministic, there is no non-final job or branch lock, the /proc/<pid>/cwd scan is conclusive with no live process inside it, the worktree HEAD is reachable from the recorded task branch, and git status --porcelain --ignored reports no tracked, untracked, or ignored content. Removal is non-force and preserves the branch.

A malformed payload on any non-final job also pins task worktrees because Gitmoot cannot prove that the job owns some other path. The bounded retention log names one malformed job responsible for this global safety pin.

A worktree registered to an older checkout root is removed through the owner in its .git pointer. If that owner cannot inspect or remove it, Gitmoot records terminal_worktree_unremovable once instead of retrying every daemon tick. Retention decisions log three times per path and classification before identical messages are suppressed.

gitmoot doctor reports delegation worktrees only; ordinary task worktrees appear in the bounded terminal task worktree retained daemon lines instead. Doctor separates reclaimable final delegation owners, pinned non-final owners, and unproven directories. /api/health exposes the same delegation metric and quarantined cleanup count in its top-level worktrees field.

[workflow].delegation_worktree_ttl = "72h" is default-on. After that grace period the daemon force-removes dirty terminal-owned read-only and delegation worktrees. Set the TTL to "0" to disable this pass. Blocked, queued, and running owners remain pinned.

No NEW fix clone is created — #2203 removed the fix-worktree lifecycle and the auto-fix leg that allocated one — and no gitmoot path deletes one that already exists. It is a standalone object database, and Linux has no inode-conditional unlink that can make deletion match a preceding proof. Commit and nested-repository checks can diagnose obvious retention reasons, but they do not close over every blob, tree, annotated tag, pack, or concurrent write. Passing them therefore records delegation_worktree_retained_unproved, never a proved-disposable handoff.

Terminal cleanup leaves the managed clone in place. Allocation recovery and enqueue failures rename it to a .ttl-reclaiming-orphaned-* sibling so retries can allocate without destroying bytes. Cleanup obligations remain open. Managed-path absence is not treated as removal, even when no sibling is found, and dangling symlinks remain visible through lstat.

gitmoot doctor and /api/health discover the canonical fixes/ directory structurally, including set-asides created before a job row exists. Directory discovery and logical-size accounting each stop after 4096 entries; the API sets truncated=true, the summary marks counts and bytes as lower bounds, and doctor warns. Removal remains a manual operator decision after inspecting the working tree and object database.

The daemon queries at most 256 due pending or aged delegation owners host-wide. Attempted candidates and selected rows skipped by repository, session, lifecycle, or checkout-liveness filters persist a later next-attempt time, so fairness survives restarts rather than depending on an in-memory cursor. The aged pass also attempts at most eight candidates per repository per tick. Candidate-local failures skip only that worktree, and later candidates continue. The five-minute pass cadence advances after every attempt so a failed cleanup cannot hot-loop. Cleanup obligations retry once per minute and stop in quarantined after the third failure. Inspect them with gitmoot job cleanup list --state quarantined and reopen a repaired target with gitmoot job cleanup reopen <resource-id>.

Repeated terminal-task failures log three times per path before identical messages are suppressed. The terminal-task pass proves at most eight candidates per tick, because each proof takes the checkout mutation lock and walks the ignored tree twice; the window rotates through the candidate list so a permanently retained worktree cannot starve the ones behind it.

For immediate relief, list candidate directories and prove ownership before removing anything:

find "$HOME/.gitmoot/worktrees" -type d -path '*/delegations/*/*' -prune -print
sqlite3 "$HOME/.gitmoot/gitmoot.db" -header -column '
SELECT id, state, updated_at, json_extract(payload, "$.worktree_path") AS path
FROM jobs
WHERE state IN ("succeeded", "failed", "cancelled")
  AND datetime(updated_at) <= datetime("now", "-72 hours")
  AND json_extract(payload, "$.worktree_path") <> ""
  AND (json_extract(payload, "$.delegation_id") <> ""
       OR json_extract(payload, "$.read_only_worktree") = 1);
'
gitmoot job show <job-id> --json
git -C <registered-checkout> worktree remove --force <verified-worktree-path>
git -C <registered-checkout> worktree prune

The sqlite3 command is an optional operator aid, not a Gitmoot dependency. Verify that job show still reports succeeded, failed, or cancelled and the same payload.worktree_path. Never remove a worktree for a blocked, queued, or running job; settle it first.

Read-Only Review Seat Cannot Run Go Or Read Prior Verdicts

A read-only review seat is Landlock-confined. Two symptoms have the same root cause - it is reaching for something outside its grants - and neither needs a wider grant to fix.

Permission denied, exit 126, running Go

A read-only seat is now given a Go toolchain automatically, and the old bootstrap-and-download workaround is retired. The daemon enumerates the Go installations on its PATH, resolves the effective go.work plus checkout-root go.mod requirement, and selects the lowest installed release that satisfies both. It stages an immutable copy under the gitmoot home and sets GOROOT, PATH, GOTOOLCHAIN=local, and a deterministic GOWORK. A seat should simply run go.

The copy exists because a seat cannot be granted the operator's own installation safely: three review rounds on that shape each produced an escape, most recently a member symlink that walked a grant outside containment with no privilege required. A copy the daemon owns has no outside root to contain.

What to expect:

• go version works and reports the selected satisfying release. GOTOOLCHAIN=local is correct here because staging already selected a release new enough for the active workspace and module; there is nothing to fetch.
• The copy is per selected installation, materialised on first use, and reused. It is keyed on content, so replacing one installed version in place produces a new copy rather than a stale compiler.
• GOWORK matches selection. off stays off; unset or auto searches from the checkout; an explicit absolute path is honored. A selected workfile outside the checkout receives a file-only read grant, never a parent-directory grant.
• The staged copy is read-only to the seat, always. If it were writable a seat could rewrite its own go binary, which is the defect the copy exists to remove.
• CGO_ENABLED=0 is still required. cgo reads /usr/include, which is outside the seat's grants, so -race remains unavailable in a seat.
• TMPDIR is a private, short directory, /tmp/gmr-<8 hex>. Each seat gets its own, mode 0700, granted to that seat only and deleted when the job ends. TMP and TEMP point at it too. GOCACHE and HOME stay under the seat's own cache root. Writes anywhere else under /tmp still return EACCES.

If a seat gets exit 126 running go, no usable toolchain was staged. The command is an engine-owned failure stub, not a fallthrough to an operator installation; it prints gitmoot: runtime unavailable: no daemon-staged artifact exists. Host-specific causes also print gitmoot: read-only seat toolchain: on the daemon's stderr. A declared real review also records seat_toolchain_unavailable with that diagnostic and blocks.

• No Go installation is on the daemon's PATH. This normal case has no daemon diagnostic, so the seat_toolchain_unavailable event is the only recorded reason and invoking go still fails explicitly through the stub.
• A candidate is not a recognizable Go installation. It must have an executable bin/go and a VERSION file naming a Go release, both real files rather than symlinks.
• Every installation prefix is copied. /opt, /usr/local, /nix/store, /snap, and profile paths follow the same staging path; none receives a recursive host-root grant.
• No installed release satisfies the active workspace and module. The diagnostic names the required release and every rejected candidate.
• The active go.work is unsafe or invalid. Relative explicit GOWORK values, symlinks, non-regular files, oversized files, and files without a valid go directive fail closed before a mismatched compiler can run.
• Free space is below the floor. Staging refuses up front rather than filling the filesystem, and the message names both the free bytes and the 4 GiB floor.
• The source contains a symlink. A symlinked name is refused anywhere in the copied set, because following one copies something the daemon never validated. The check also proves the file it opened is the file it inspected, and staging walks a directory through a held handle rather than re-resolving its path, so a source whose name is redirected mid-copy is refused rather than followed - including when the directory containing it is the thing replaced.

Two limits of that, stated rather than left implied. First, it proves the file did not change between being inspected and being opened. A hardlink does not change the file - it is the same underlying object - so replacing a source file with a hardlink to another file is not detected. Two things bound this rather than excuse it: with the kernel default fs.protected_hardlinks=1, creating such a link already requires read and write access to the target, so it gains nothing an attacker did not already have; and the selected source installation is expected to be operator-owned and not writable by a seat, so a seat cannot create one. Second, staging reads the source twice, once to fingerprint it and once to copy it, and only executables are reconciled between those two reads: a non-executable that differs between them is not detected. Each read is internally consistent, so this is a gap between the passes rather than inside one. If you point staging at a directory that untrusted processes can write, these are the gaps to know about - the intended source is an installation only the operator can modify.

To recover, fix the cause and restart the daemon, or run the seat's job again: staging is attempted per launch and a previously failed stage is retried rather than remembered. A staged copy that has become corrupt is detected on reuse and re-staged automatically, so deleting it by hand is not required. Staged copies are retained by current pin: a version still pinned is never collected, and a version nothing references is removed.

One limit, stated rather than implied: the identity covers every executable in the copy, so a substituted compiler is always detected. It does not cover non-executable members such as src or doc, because re-hashing the whole 221.6 MiB tree on every seat launch would cost more than the copy itself. The consequence of a torn non-executable member is a visibly broken toolchain and a build error, not a silently wrong compiler.

Seat temp dir, Go module cache, and other toolchains

Temp dir. A seat's TMPDIR is /tmp/gmr-<8 hex>, 17 bytes, so a test that binds a Unix socket under t.TempDir() stays inside the 108-byte socket path limit. It used to be <cache root>/tmp, which was 57+ bytes on a real host, so socket tests failed for reasons unrelated to the code under review. When the daemon's own TMPDIR is a clean absolute path of at most 24 bytes, it is used as the parent instead of /tmp; a longer one is ignored. That is also how gitmoot run inside a seat (a reviewer running gitmoot's own tests) nests its seats: in the outer seat's /tmp/gmr-<8 hex>. If no short parent is writable, as inside a seat of a daemon from before this change, the seat falls back to <cache root>/tmp and logs no short writable temp parent; using cache-root temp; it works, but its socket paths are long again. The directory is mode 0700, owned by the daemon user, and only that seat is granted it: the parent directory, other seats' gmr-* directories, the checkout and the gitmoot home stay read-only. The daemon records each directory in <gitmoot-home>/cache/seat-tmp/ with its owner's pid and removes it when the job ends. If the daemon crashes first, the next daemon start (and the next seat launch) removes every directory whose owner process is gone. The owner is the process that made the directory (the daemon or a foreground job run), so a seat process that outlives it loses its temp dir. A record that names a gmr-* directory under a parent no seat could have used (anything but an absolute path of at most 24 bytes), or the sweeping process's own TMPDIR or a directory holding it, is kept and reported, never removed. The cache-root fallback has no record; it goes with the cache root.

Go module cache. Offline go build and go test work in a seat. Before the seat starts, the daemon runs go mod download for the checkout's modules into <gitmoot-home>/cache/seat-go-mod/<repo hash>/, one cache per repository, and grants the seat that cache read-only with GOPROXY=off. Repeat reviews of the same repository reuse it. The download runs under its own Landlock sandbox, with the checkout read-only and only the cache writable, the daemon's GOPROXY minus direct and off, GOVCS=*:off and GOTOOLCHAIN=local, so no version-control tool runs against a URL the checkout chose. A module the download could not fetch fails in the seat as module lookup disabled by GOPROXY=off; the cause is on the daemon's stderr as gitmoot: read-only seat Go modules:. A seat never writes this cache, so one review cannot plant a module for the next.

Before #2314 each seat's GOMODCACHE was a new, empty directory keyed on the checkout path, so every offline build failed. Pointing a seat at a cache populated for another checkout path does not work either: the Landlock grant does not cover it.

Python and other toolchains are not available in a local seat. pytest, ruff and uv fail with Permission denied because they live in operator-owned trees (/usr/local, ~/.local/bin, Python's dist-packages) that a seat is not granted. Granting those trees is the operator-tree grant #1878 rejected, and copying them is unbounded: the interpreter's packages are hundreds of megabytes, include editable installs that point into operator checkouts, and a project's own dependencies are often not installed on the host at all. Rust, Node, Flutter/Dart and similar toolchains need their own staged toolchain and writable caches. These checks move to disposable VMs (sandboxd/E2B) under #2315. A local seat that cannot run them reports evidence: static_only with a checks_blocked reason such as toolchain. Terminal devices (/dev/ptmx) stay closed: seats run as root, and a terminal device would let a reviewer type into other agents' terminals.

Read-only seat runtime executable is unavailable

Claude, Kimi, and Codex read-only seats run a copied runtime artifact from <gitmoot-home>/runtimes, never a recursive read grant over the operator's profile or package root. That directory is a SIBLING of toolchains/, not a child: the toolchain collector removes anything under toolchains/ that is not a pinned toolchain, which would delete staged runtimes mid-launch.

Every source read uses openat2 with RESOLVE_NO_SYMLINKS and RESOLVE_BENEATH, so no component of a source path may be a symlink and nothing resolves outside the boundary. os.Root is not sufficient here: it refuses escapes out of a root but follows symlinks that stay inside one, so a member replaced by a link to a sibling credential was previously copied.

An AMBIGUOUS SOURCE TREE IS REFUSED WHOLE rather than filtered, because file mode bits cannot distinguish a secret from a payload - a 0640 file under a 0750 or setgid directory reads as group-readable, and a POSIX ACL shows up in the group bits. Staging therefore refuses, and publishes the runtime unavailable, if the boundary contains a symlink, a non-regular member, a setuid or setgid member, any POSIX ACL, a directory the world cannot traverse, or a member that is not world-readable. The resolved entrypoint itself is exempt from the world-readable rule only, so a runtime installed mode 0700 still runs.

A SCRIPT RUNTIME GETS AN EXPLICIT LAUNCHER. Codex resolves to codex.js with #!/usr/bin/env node, so the engine stages that interpreter too and writes a launcher that execs the STAGED interpreter with the STAGED entrypoint. The kernel never resolves the entrypoint's original shebang, which would reach the operator's copy. A binary runtime gets a relative symlink instead, needing no shell.

THE ENGINE'S OWN LAUNCHER DIRECTORY IS EXCLUDED FROM REVALIDATION, and the exclusion is a traversal rule rather than a filter applied afterwards (#1974). A published tree is re-proved on every reuse by recomputing the digest of its members with the same symlink-refusing traversal, and .bin/ never contributed to that digest - the recorded value is the digest of the SOURCE members, which cannot contain a launcher the publish created after reading them. Collecting it and dropping it afterwards still had to OPEN it, so a binary runtime's symlink launcher failed the reuse with too many levels of symbolic links and the runtime was republished as the exit-126 shim on every subsequent staging. Only the TOP-LEVEL .bin is excluded; a .bin directory that came from a source package, which is the ordinary shape of a node-packaged runtime, is payload and is still hashed.

Every runtime class the engine can dispatch is staged, not only the seat's own. An absent runtime, an unresolvable interpreter, an ambiguous tree, or a kernel without openat2 all resolve to engine-owned commands that print gitmoot: runtime unavailable: no daemon-staged artifact exists and exit 126. This preserves ordinary system commands through the inherited PATH without permitting a runtime name to fall through to its host copy. Failures also appear as gitmoot: read-only seat runtime: on daemon stderr.

One limit, stated rather than implied: published runtime copies are not reclaimed. Each distinct content fingerprint publishes a new directory and nothing removes it, because safe reclamation needs to know which trees live seats are executing. toolchain.Collect has no production caller either, so the same is true of staged Go toolchains.

Codex reviews report zero executed checks (bwrap: setting up uid map)

A codex read-only seat used to fail EVERY command before running it, so its reviews came back static-only with an executed-check count of zero. Codex's own sandbox is bwrap, which needs a user namespace that gitmoot's Landlock domain denies, so two sandboxes were fighting.

Measured on this box, one command with only the domain differing:

bwrap --dev-bind / / --unshare-user -- /bin/true
  outside the Landlock domain        -> rc=0
  inside the domain                  -> rc=1  bwrap: setting up uid map: Permission denied
  inside + /proc writable            -> rc=1  bwrap: Failed to make / slave
  inside + /proc + /dev writable     -> rc=1  bwrap: Failed to make / slave
plain /bin/true inside the domain    -> rc=0

So the domain executes fine and NESTING was the cause; granting more only moved the error from the uid map to mount propagation, which is why widening grants could not settle it.

A codex seat therefore now runs with its own sandbox OFF and relies on the Landlock domain, which is the boundary gitmoot builds and tests. End to end in that seat, same grants either way:

• before: bwrap: setting up uid map: Permission denied, nothing executed
• after: /bin/bash -lc 'echo PROBE_RAN' … succeeded in 0ms: PROBE_RAN
• writing outside the granted roots is still refused - sh: cannot create /root/…: Permission denied, and no file appears on the host

If a codex seat still reports zero executed checks, the seat's own state directory is the usual cause: CODEX_HOME must be WRITABLE (it is staged inside the seat's cache root). A read-only CODEX_HOME fails earlier and differently, with failed to initialize in-process app-server client: Permission denied.

A review reporting it could not read prior verdicts

The seat is given a RENDERED, REPO-SCOPED list of prior verdicts inside its own cache root, and GITMOOT_PRIOR_VERDICTS names the file. It states its own as_of time and scope, so a frozen list cannot be mistaken for a live query.

Rendered rather than a copy of the database, for two measured reasons. Granting the live store as read-only files does not work at all: SQLite opens the -wal and -shm sidecars O_RDWR|O_CREAT, which a read-only grant refuses, so the open fails before any verdict is read - and journal_mode=wal persists in the header, so a quiescent database behaves the same. And a whole-database snapshot is far too wide: it would hand every repo's jobs, prompts and results, plus lock and fencing tokens, to a runtime that forwards what it reads to a third-party model API. The rendered list carries only verdict fields, so everything else is structurally absent rather than merely unmentioned.

If a seat has no single repo scope, no list is staged and the reason is recorded in the job's dropped diagnostics rather than passing silently.

A review on the remote execution backend gets the same list at /home/user/.gitmoot/runtime/evidence/prior-verdicts.json, also named by GITMOOT_PRIOR_VERDICTS; when it cannot be rendered the job records a remote_review_evidence_unavailable event with the reason.

Read-Only Reviewer Seat Refuses To Start

A read-only seat runs the runtime against an ISOLATED home rather than yours, so anything the runtime reads at startup must be staged into that home first. When a startup input is missing, Gitmoot now fails BY NAME instead of letting the runtime report something unrelated.

Symptoms:

• read-only seat requires runtime input "config.toml", and <path> does not exist - the host profile has no such file. This is a HARD PREREQUISITE for kimi: ~/.kimi-code/config.toml must exist on the host, or the seat cannot start. Previously this surfaced as "No model configured" behind an auth message that pointed at kimi login, which cannot fix it.
• runtime input "<path>" must be a regular file - the path is a directory, socket, device or fifo. A SYMLINK is fine and is followed, so a stow/chezmoi-managed profile works; a symlink whose target is missing counts as missing, and a symlink to a directory is refused.
• read-only seat credential <path> is unusable: claudeAiOauth expired at <time> and carries no refreshToken - the profile stages and parses but cannot authenticate. Re-authenticate the host claude profile. An expired token WITH a refresh token is accepted, because refreshing it is the runtime's job.
• narrow runtime input "<path>": codex config.toml has a section header that is not readable - the file has a section header that never closes, so Gitmoot cannot locate the credentials it must strip. It refuses rather than stage a file it cannot classify. Fix the TOML or remove it from the host state dir.

What the seat stages, per runtime:

runtime	staged from	inputs
claude	~/.claude	.credentials.json, narrowed to claudeAiOauth and checked for usability. Nothing is staged in gateway mode, where the gateway supplies the credential.
codex	~/.codex	auth.json, plus config.toml when present - NARROWED, see below
kimi	~/.kimi-code	config.toml (REQUIRED), credentials/kimi-code.json
omp	none	No operator profile input is staged. The daemon reads its upstream token from the owner-only file named by OMP_AUTH_BROKER_TOKEN_FILE. A minimal job-private config.yml holds the random token and loopback URL for a proxy filtered to the provider in the required provider/model selector. PI_CONFIG_DIR and PI_CODING_AGENT_DIR both resolve inside the disposable state root.

Why the staged codex config.toml is not a copy

The seat's staged state lives inside the one writable path granted to the sandbox, so a reviewer can read anything placed there. A codex config.toml routinely carries credentials that have nothing to do with running the model, so Gitmoot narrows it before staging:

• [mcp_servers.*] is dropped ENTIRELY. Its env table holds tokens, args can hold them just as easily, and a read-only reviewer seat has no business spawning third-party servers. MCP servers are optional to codex, so dropping them cannot stop the seat starting.
• [model_providers.*] KEEPS its structure and loses only api_key and http_headers. Dropping the section would look stricter and would break a custom model, because removing base_url and wire_api makes the provider unresolvable. env_key and env_http_headers name environment variables rather than hold values, so they stay.
• Everything else is kept: those are the model and sandbox settings the file is staged for.

Dotted and quoted forms are classified the same way, so mcp_servers.github.env.TOKEN = "..." at the top level and [mcp_servers."my server".env] are both dropped.

The staged kimi config.toml is narrowed too

kimi's config.toml is REQUIRED, and it carries more than codex's does: measured on a live host, api_key under [services.*] and a key under [services.*.oauth] alongside [providers.*]. For kimi the staged copy lands directly inside the seat's writable root, so it is narrowed on the same rule:

• [services.*] is dropped ENTIRELY. It configures optional tooling (moonshot search and fetch) and carries those tools' credentials.
• [providers.*] keeps its structure and keeps api_key and its env sub-table ONLY for the provider default_model resolves to - the model's segment before the first / matched against the provider name's segment after the last :, so default_model = "kimi-code/k3" selects [providers."managed:kimi-code"]. Dropping every provider credential is not an option: kimi refuses to start when both api_key and the env sub-table are absent, so the seat legitimately needs exactly one.
• If NO provider can be resolved, or two match ambiguously, every provider credential is withheld. A credential nobody can prove is needed is not staged.

The same rule now applies to codex: [model_providers.*] keeps api_key and http_headers for the provider model_provider selects and strips every other. Stripping the selected provider's key too left the seat unable to authenticate at all, because env_key names a variable the sandbox's environment allowlist does not pass through.

Finding out what was withheld

Narrowing is not silent. When anything is withheld the job records a read_only_seat_config_narrowed event naming it, so a reviewer whose MCP tool is missing - or a seat that cannot reach a provider - can find out why:

gitmoot job events <job-id> | grep read_only_seat_config_narrowed

Two more refusals, and what gateway mode withholds

Narrowing reads the file with ONE scanner shared by narrowing and provider selection, so escapes, comments, multi-line strings and unbalanced brackets are interpreted identically everywhere. Two shapes are refused rather than staged, because in both cases part of the file could not be classified, and staging an unclassified tail is the leak the narrowing exists to prevent:

• config has an array or inline table that never closes - an unbalanced [ or { reached the end of the file.
• config has a multi-line string that never closes - as before.

A triple delimiter inside a COMMENT or inside a single-line string is not a multi-line string, and a backslash-escaped quote is valid TOML that stages unchanged.

One provider shape is refused for a different reason:

• codex model_provider "<name>" authenticates only through env_key, and a read-only seat's environment allowlist does not pass that variable through - set an inline api_key for that provider, or point model_provider at one the seat can reach. Previously this staged cleanly and the seat failed later with the runtime's own message.

In GATEWAY mode the gateway supplies the credential, so a gateway seat stages NO credential file for any runtime - claude's .credentials.json, codex's auth.json and kimi's credentials/kimi-code.json are all withheld. Model settings are still staged, narrowed as above.

Comments, and what a seat does when a config cannot be narrowed

Narrowing classifies the COMMENT-FREE part of each line and stages the line unchanged, so an ordinary TOML comment cannot change what is withheld. Both directions were defects: [services] # see [docs] used to have the comment's ] swallowed into the section name, staging that section's secrets verbatim, and default_model = "kimi-code/k3" # pinned used to make the comment part of the provider name, withholding the one credential the seat needs. Your comments survive into the staged file.

A key/value pair is split on the first = OUTSIDE quotes, so a quoted key containing = is kept rather than dropped.

A config that cannot be narrowed is treated by whether the runtime needs it:

• codex's config.toml is OPTIONAL. If it cannot be narrowed - an unbalanced [ or { at end of file, an unreadable section header, or a selected provider that can only authenticate through env_key - the seat starts WITHOUT it and falls back to the runtime default. The file is not staged and the reason is named in the read_only_seat_config_narrowed job event.
• kimi's config.toml is REQUIRED, so the same refusal fails the seat by name. kimi cannot start without it, and a silent fallback would surface later as "No model configured".

In GATEWAY mode no runtime stages a credential file, and the policy the seat computes says so rather than naming files it then withholds.

Isolated worktrees duplicate gigabytes of tool cache

An isolated-worktree job re-materializing its own uv/go/npm/pip cache inside its worktree (rather than reusing a shared one) is the largest driver of gitmoot doctor's worktree disk figure once a fleet has run for a while — one worktree can carry several gigabytes of immutable, content-addressed packages that duplicate what every other job already downloaded.

Gitmoot points UV_CACHE_DIR, PIP_CACHE_DIR, npm_config_cache, GOCACHE, and GOMODCACHE at one shared, host-level directory (default <home>/cache/tools) for isolated-worktree jobs, so package caches are reused across jobs instead of duplicated per worktree. This is on by default; set [cache] enabled = false in config.toml to opt out, or [cache] dir = "/absolute/path" to relocate the shared directory (must be absolute).

A read-only/auto codex job does not get the redirect — codex never grants it writable-path access, so pointing its tools at the shared directory would break rather than help. Those jobs keep their pre-existing (in-worktree) cache behavior.

If an existing worktree already carries a large in-worktree cache from before this took effect, it is cleaned up the same way as any other worktree content: by the delegation-worktree reclaim pass above, or manually once the owning job is terminal.