Agents that get better the more you use them

scenario

A weak agent you use every day.

You have an agent template (a reviewer, a summarizer, a codegen helper) that is fine, not great. You keep hand-correcting it in the same ways. That steady stream of corrections is exactly the signal a better version should learn from, but there is nowhere for it to go.

why gitmoot

A promotion you can trust, not a vibe.

Optimizing a prompt is easy; knowing the new one is actually better is hard. Greedily accepting whatever scores higher on a handful of examples p-hacks itself: you ship drift and call it progress. Gitmoot pairs the SkillOpt train/review loop, which generates and ranks candidates from your usage, with the PACE anytime-valid commit gate, which only promotes a candidate once it has beaten the champion by a real statistical margin.

how you drive it

Prompt an agent, or run it yourself.

Same loop, two ways: ask your coding agent to run it in plain language, or drive the Gitmoot SkillOpt CLI directly.

You describe the outcome you want; the agent runs the whole train → rank → optimize → gate loop and reports back with the one thing that matters: whether the new version earned its promotion.

you

My code-reviewer agent flags too many false positives. Train a better version from our past reviews, but only promote it if it’s genuinely better.

@coordinator

Started. I scaffolded a SkillOpt training session for code-reviewer against your review history and enabled auto-promotion behind the PACE commit gate. SkillOpt generated candidate reviews, you ranked them, and the optimizer produced code-reviewer@v2.

usedgitmoot skillopt train initgitmoot skillopt train startPACE gate enabled

you

How do we know v2 is really better, and not just lucky on a handful of examples?

@coordinator

That’s exactly what PACE guards against. I’m serving live reviews through both the current champion and v2 and recording which one you prefer. PACE runs an anytime-valid test over that candidate-vs-champion record and only promotes v2 once the evidence crosses the significance threshold: no p-hacking, no drift. It’s ahead right now, but not yet decisive, so it’s holding.

usedgitmoot skillopt abPACE commit gate

you

Ping me when it promotes.

@coordinator

Will do. The moment the evidence is decisive, v2 becomes the champion automatically. If it never clears the bar, it stays a candidate and nothing about your live reviewer changes.

usedPACE auto-promote

setup

Scaffold a training session, then gate it with PACE.

Point SkillOpt at the template you want to improve and the repo where reviews live. This writes a config scaffold you can start from.

shell

gitmoot skillopt train init \
  --name reviewer-v2 \
  --template code-reviewer \
  --review-repo acme/store \
  --artifact-kind review \
  --preview none \
  --request "Fewer false positives; cite the exact line."

Then turn on auto-promotion and the PACE commit gate in the same [skillopt] config. Both are off by default and purely additive: auto-promote lets Gitmoot ship a winning candidate for you behind its usual confidence floors, and PACE is the final say: it refuses to let anything promote until the candidate-vs-champion evidence is statistically decisive.

config.toml

[skillopt]
# Let gitmoot ship a winning candidate for you, behind its
# usual confidence floors...
auto_promote = true
auto_promote_min_samples = 5
# ...but PACE is the final say: an off-by-default, additive
# anytime-valid e-process over the candidate-vs-champion
# pairwise record must cross 1/pace_alpha before it promotes.
pace_enabled = true
pace_alpha = 0.05

the run

Generate, rank, optimize, then earn the promotion.

01
Run the train/review loop
SkillOpt generates candidate outputs and collects the human ranking that says which one was better, and why.
gitmoot skillopt train start --config .gitmoot/skillopt/reviewer-v2/config.toml
02
The optimizer produces a candidate
Ranked signal in, an improved template version out. Check where the session stands at any point.
gitmoot skillopt train status --session <session-id>
03
Inspect the candidate
Every candidate version is listed and diffable before anything ships.
gitmoot skillopt candidate show <version-id>
04
Put the candidate in front of real work
Serve a live prompt through both the champion and the candidate, record which answer was better, and Gitmoot tallies the candidate-vs-champion pairwise record (and prints P(challenger>champion)). Every run adds one data point.
gitmoot skillopt ab code-reviewer "Review PR #42" --challenger <version-id>
05
PACE promotes it, but only once it’s earned
There is no gate command to run: PACE lives in the auto-promote path. It runs an anytime-valid e-process over that pairwise record and, the moment the wealth crosses 1/pace_alpha, promotes the candidate for you. Until the evidence is decisive it holds. To ship one by hand instead, promote it yourself.
gitmoot skillopt candidate promote <version-id>

payoff

Agents compound; every promotion is earned.

The reviewer you used yesterday trains the reviewer you use tomorrow, and the only versions that ship are the ones that beat the incumbent on the evidence. Improvement stops being a hopeful prompt edit and becomes a ratchet: each promotion strictly better, none of them a regression you find out about later.

variations

Tune the loop to your risk tolerance.

Lower pace_alpha for a stricter bar (a higher commit threshold), or leave it at the default for a balanced gate.
Watch a candidate’s standing any time with gitmoot skillopt candidate show <version-id>; PACE’s promote-or-hold decisions surface in the candidate’s notify events.
Reject a candidate that never clears the bar with gitmoot skillopt candidate reject <version-id> --reason "…".
Discover the available task kinds and templates with gitmoot skillopt train init templates --json.

Gitmoot SkillOpt builds on Microsoft SkillOpt. For the full command reference, see the docs.

A weak agent you use every day.

A promotion you can trust, not a vibe.

Prompt an agent, or run it yourself.

Scaffold a training session, then gate it with PACE.

Generate, rank, optimize, then earn the promotion.

Run the train/review loop

The optimizer produces a candidate

Inspect the candidate

Put the candidate in front of real work

PACE promotes it, but only once it’s earned

Agents compound; every promotion is earned.

Tune the loop to your risk tolerance.