Skip to main content

Run Jobs In Parallel On A Repo

By default the daemon runs queued jobs one at a time per repo: --workers defaults to 1 and the scheduler defaults to barrier (a per-tick batch that serializes same-repo jobs on one checkout lock). That is the safe, conservative default — but it means a fan-out of independent jobs on one repo runs serially unless you opt in to parallelism.

This guide shows the one-step on-ramp.

The short version​

# Intent-level: workers + the pool scheduler together, named for the goal.
gitmoot daemon start --parallel 5

# Equivalent, explicit form:
gitmoot daemon start --workers 5 --scheduler pool

--parallel N sets --workers N and --scheduler pool in one flag. It is sugar for the explicit pair and cannot be combined with --workers or --scheduler (that would be ambiguous).

If you raise --workers above 1 and do not pass --scheduler, the daemon now auto-selects pool for you — requesting multiple workers under the serializing barrier is almost never what anyone wants:

gitmoot daemon start --workers 5     # auto-selects --scheduler pool

To keep the old per-tick semantics with multiple workers, ask for them explicitly — an explicit --scheduler barrier is always honored:

gitmoot daemon start --workers 5 --scheduler barrier

Confirm the daemon is configured for it​

gitmoot daemon status reports the scheduler mode and worker count, so you can answer "is the daemon actually set up for the parallelism I asked for?" without re-deriving it from launch flags:

daemon running pid 12345
log: ~/.gitmoot/logs/daemon.log
scheduler: pool, workers: 5
claude auth: ok (...)

If you started under barrier with multiple workers, the line flags it and points at the fix.

Preflight warning​

If a parallelizable workload is queued under a serializing config, the daemon logs an actionable warning with the exact relaunch command instead of silently serializing the work:

warning: 3 parallelizable jobs queued for owner/repo will run serially under the
current scheduler config; relaunch with: gitmoot daemon restart --parallel 3

"Parallelizable" is counted conservatively: same repo, dependency-unblocked (already true of queued jobs), and distinct runtime sessions (see the next section). Two jobs on the same agent session are not counted as parallelizable, because they serialize regardless of the scheduler.

The warning is rate-limited: it is re-logged only when the parallelizable set changes, not on every poll, so a steady backlog does not spam the daemon log.

What actually runs in parallel (two serialization layers)​

Parallelism under pool is bounded by two independent locks, not one:

  1. Checkout lock (per worktree). Jobs run concurrently only when they have distinct checkout keys. Every local review gets a detached per-job worktree at its requested head; background taskless ask jobs can be auto-isolated at checkout HEAD, and read-only delegation legs get their own detached worktree from the workflow engine, so they parallelize. An engine review that requests changes reports the verdict back to the requester's org role and to the --lead seat, and allocates no implement job — there is no per-PR opt-in that turns a verdict into a fix dispatch and no writable fix clone (#2203). The lead implements in its own checkout and records it with gitmoot job record --type implement, which is outside the scheduler entirely and so never contends for a checkout key.

    Read-only refs depend on the action. Review uses its exact requested head, preserves that head in the payload, and fails dispatch closed if allocation fails. Ask/delegation isolation uses the committed tip and remains fail-open where it is only a parallelism optimization. Detached worktrees do not contain gitignored paths (e.g. vendored clones under repos/**) or any uncommitted working-tree changes. Isolation only applies to asks when same-repo readers are contended; a task-bearing or foreground ask keeps its existing checkout. Committed-tip isolated asks carry a note with the canonical base-checkout absolute path, so a worker whose sandbox can read it (e.g. codex) reaches the real tree instead of reporting a working-tree feature as missing. Exact-head review does not add that note. For whole-working-tree ask analysis, use a foreground or task-bearing ask, or pass an absolute path to the file/dir under analysis.

  2. Runtime session lock (runtime:<runtime>:<ref>). Two jobs that use the same agent/runtime session serialize on the session lock even if their checkouts differ. So same-repo parallelism is bounded by distinct runtime sessions — give the work distinct sessions (or distinct agents) to spread it across the pool. This applies to resumable runtimes (Codex thread_id, Claude, Kimi); the checkout lock is runtime-agnostic, the session lock is not.

The practical recipe for N-wide same-repo work today is therefore: N delegation/orchestra legs (each engine-isolated into its own worktree) running under distinct runtime sessions, with --parallel N.

Reconfigure without restarting (SIGHUP)​

Changing workers/scheduler/poll/idle cadence no longer needs a daemon restart (#577): kill -HUP <daemon-pid> re-reads the [daemon] config section (poll, workers, scheduler, parallelism, idle_grace_ticks, and idle_max_multiplier) live — no teardown, no dropped jobs, no environment re-inheritance (so the daemon's runtime auth is untouched). Values pinned by explicit launch flags win over the re-read config. When a full restart is genuinely needed, prefer gitmoot daemon restart, which recovers the per-delivery Claude auth file.

Cap one repo's parallelism from config​

To cap a single repo on a shared daemon — without touching the global worker count or relaunching anything — add a [repos."owner/repo"] section (#576):

[repos."owner/repo"]
max_parallel = 1          # cap this repo's in-flight jobs; 0/unset = global default
# scheduler = "barrier"   # optional per-repo scheduler override

max_parallel = 0 (or an absent section) means "use the global default"; a positive value caps that repo's concurrent jobs. The keys are re-read every tick, so edits apply live.

Host-wide admission budget​

On a memory-constrained host, the opt-in [admission] section adds a second, host-global gate the daemon applies before starting each agent session, on top of --workers/pool and the per-repo locks (#365):

[admission]
max_concurrent_sessions = 0   # cap total in-flight sessions; 0 = off
max_memory_gb = 0             # cap summed per-runtime RAM estimate; 0 = off
# codex_memory_gb = 0.2       # operator-tunable per-runtime RAM priors
# claude_memory_gb = 0.85
# kimi_memory_gb = 0.5
# default_memory_gb = 0.5

With both caps 0 (the default) the budget is disabled and scheduling is byte-identical to a config without the section. A job that does not fit BOTH caps is left queued and retried next tick — never failed — so "jobs stay queued for no visible reason" on a small host can mean the admission budget is holding them. The budget is enforced per daemon process (host-global for the normal single-daemon deployment).

GitHub rate-limit-aware scheduling​

The daemon's polling and every agent's gh/API calls share one GitHub account. GitHub's secondary (abuse-detection) rate limit fires on burstiness and concurrency, not total volume, so concurrent bursts can trip it (HTTP 403 "secondary rate limit") and freeze all GitHub ops even while the primary quota is fine — the only manual workaround being to stop the daemon and wait out the cooldown (#683).

The primary limit is a separate failure with a separate remedy. When a call fails on primary exhaustion, Gitmoot re-issues that same request once with headers and reports the window its own response named: the resource, how empty it is, the reset timestamp and the remaining wait. A primary window carries no Retry-After and cannot be shortened, so waiting is the only remedy and a retry before the reset cannot succeed.

It reads the failed request's headers, never GET /rate_limit. That endpoint is not a usable precondition: measured on one credential and one host seconds apart, it reported core 4999/5000 used=1 while a real core request returned 403 with remaining=0 used=5000, their resets thirteen minutes apart, and minutes later it reported a fresh 5000/5000 used=0 while real calls were still refused. The 403 header's reset was accurate throughout.

The opt-in [github] section installs a GitHub call budget + adaptive backoff that is in-process to the daemon — it covers the gh/API calls gitmoot itself issues from the daemon process (polling, comments, merges, status). It is enforced per daemon process (host-global for the normal single-daemon deployment), the same scope as the admission budget above. It does not reach into separate foreground processes (a foreground gitmoot orchestrate/pool/review/pr comment) or the gh calls a codex/claude runtime subprocess makes on its own — those run outside the daemon process and never touch the shared limiter.

[github]
max_concurrent = 0        # cap in-flight gh calls; 0 = unlimited (default)
min_interval = "0s"       # min spacing between call starts; 0 = off (Go duration or bare seconds)
secondary_backoff = true  # pause all GitHub calls on a secondary/abuse limit (default true)
backoff_base = "60s"      # exponential fallback base when no Retry-After (default 60s)
backoff_max = "5m"        # exponential fallback cap (default 5m)
conditional_requests = true # ETag conditional polling (default true)
calls_per_hour_warn = 0      # daemon-local sliding-hour warning; 0 = off

Safe defaults: the proactive caps (max_concurrent, min_interval) default off, so single-call latency and steady-state throughput are unchanged; only the reactive secondary_backoff is on, and it is invisible on the happy path — it engages only after a gh call actually fails with a secondary/abuse limit. On a hit the limiter pauses all GitHub calls process-wide (respecting the response's Retry-After, else the exponential fallback) rather than retry-storming the abuse detector, which only prolongs the block. Calls are never dropped — they queue/delay until the window passes. On a busy host, set max_concurrent (e.g. 6) and/or a small min_interval (e.g. 250ms) to also smooth bursts proactively. gitmoot daemon status shows the configured budget.

The four per-tick repository list reads use an in-memory ETag cache. A 304 Not Modified replays the prior raw JSON and does not consume GitHub's REST quota. After three consecutive successful all-304 ticks, a quiet repo moves to 2x base cadence and then to 4x; configure the thresholds under [daemon]:

[daemon]
idle_grace_ticks = 3
idle_max_multiplier = 4  # 1 disables idle decay

Any response-body miss, poll error, queued repo job, or in-flight repo job resets the streak and promotes the repo immediately. A repo with an open PR remains at base cadence because its per-PR comment reads are deliberately non-conditional. The decayed NextPoll gates GitHub calls only: heartbeat and pipeline still wake at the resolved base interval. The local call count is approximate and covers only this daemon process; foreground commands and agent-owned gh processes are outside it.

Not yet automatic (follow-ups)​

  • Top-level implement auto-isolation. pool does not auto-isolate plain same-repo implement jobs into worktrees. The engine's review-fix path is the narrow writable exception; read-only ask/review jobs have their separate isolation paths. Parallelizing independent top-level implement jobs via the daemon needs implement-eligible auto-isolation (a real branch worktree + branch-lock handling
    • a worktree cap and disposal sweep); that is intentionally deferred.

See also​