Retrofit Automation Loop: Our Honest First Try

TL;DR

  • We wrote six parts on loops, then audited our own setup and found we had none. Just rituals we ran by hand.
  • We converted the design gap-log triage first, because it had the smallest machine-checkable “done”, and wrapped it in a state file, a stop condition, and a heartbeat.
  • The four-step recipe at the end is the one you can reuse. The week-one report is here too, including the bug that faked success and the day the loop quietly fell out of the schedule.

I spent six parts of this series teaching loops. Then I ran an audit on our own setup and found something I did not enjoy writing down: we did not have any. What we kept calling loops were rituals, things a human kicked off every morning and babysat to the finish. This is the honest log of building our first real retrofit automation loop, wrapping a heartbeat, a state file, and a stop condition around one ritual we already did by hand, including the parts that broke.

The frame the whole series runs on, byte for byte:

A loop is a recursive goal: you define a purpose and a verifiable stop condition, and the system iterates agents against it until the condition holds — without you prompting each turn.

Read that definition back against your own “automation” and the gap shows up fast. Ours did. The promise of this part is a four-step recipe, proven against a real week of runs rather than a diagram. The honesty contract is the other half: this is a journey log, so it includes the false starts, the cost, and the place where my own plan was wrong. The week-one breakage is recorded below, drawn from the loop’s own run history, including the part I did not see coming.

The audit: rituals that looked like loops

The audit was one question asked of three rituals: does this run without me prompting each turn? All three failed it, and they failed the same way. They had a maker step and a vague sense of “done”, but no heartbeat to wake them, no state file to remember progress, and no machine-checked stop condition. I was the scheduler, the memory, and the stop condition, every time. That is not a loop. That is a person with a checklist and good intentions.

This is the constraint paradox showing up again: the rituals felt productive precisely because I was in the loop, so the missing automation never announced itself. Writing the test down is what exposed it.

RitualThe loop test it failed
Design gap-log triageNo heartbeat. I ran it “next session”, whenever that was.
Content review passNo machine-checked stop. I eyeballed “good enough”.
Agent-map drift checkA human prompted every step, start to finish.

None of these was lazy work. They were just honest manual rituals wearing the word “loop” because we had written about loops so much we assumed we ran them.

Picking the first ritual to convert, and why

The first ritual to convert is the one with the smallest machine-checkable “done”, not the one that hurts the most. The content review pass hurt more and would have felt heroic to automate. But its stop condition is a judgment call, hard for a machine to check honestly. The design gap-log triage had a cleaner shape: a backlog of gap entries, and a “done” that a script can verify, which is every gap carries a verdict. Small, real, and checkable.

Key insight: The right first loop is the ritual with the smallest machine-checkable “done”, not the most painful one.

That criterion did the selecting for me. The triage also had a real backlog waiting, 28 gap units across four GAPS files, so the loop would have actual work on its first wake rather than an empty run. Starting on a ritual with a fuzzy stop condition would have meant debugging the loop and the definition of done at the same time. I wanted exactly one new thing to break at a time.

The retrofit, step by step: state file, stop condition, schedule

The retrofit was three pieces in order: a state file to remember verdicts, a stop condition a script can check, then the schedule that wakes the whole thing. Built in that sequence, the retrofit automation loop came together without much drama. Each piece reused a primitive an earlier part already owns, so this beat is assembly, not invention.

The state file. Each gap gets one entry, keyed by a stable id, recording its bucket, route, and which brain decided. This is the state file from Part 4, nothing exotic:

{
  "schema": "gaplog-triage/v1",
  "updated": "2026-06-16T09:15:27Z",
  "triaged": {
    "75e172c0cfdd": {
      "title": "<gap row text>",
      "file": "V0.25-SPRINT-2A-GAPS.md",
      "line": 5,
      "bucket": "deferred",
      "route": "park to backlog",
      "brain": "deterministic"
    }
  }
}

The stop condition. The loop is done when every discovered gap has a verdict in that file. A script checks it and exits non-zero while work remains, which is exactly the verifiable “done” from Part 3. On day one it read like this:

$ python3 triage.py stop
STOP-CHECK: 28 units, 8 triaged, 20 pending
RESULT: NOT DONE (loop iterates)

Eight triaged on the first pass, 20 pending, stop says NOT DONE. The loop has a reason to wake again tomorrow.

The schedule. The heartbeat runs on master, because the Actions schedule only fires from the default branch. The gap log itself lives on develop, so the loop materializes it read-only at run time rather than committing it to master. That is the heartbeat from Part 6, pointed at a board on another branch.

There are two brains behind the verdict. One is a deterministic keyword ranker, cheap and offline. The other calls claude -p to let the agent reason over the gap, and is designed to fail soft to the deterministic ranker if the model’s answer does not parse. The honest cost lives here: the agent brain draws metered Agent-SDK credit on every call, so the daily heartbeat defaults to the free deterministic brain and the agent brain is opt-in via a manual dispatch. Empty runs have to be cheap, or the schedule becomes a slow money leak. Whether the agent brain actually behaved the way I designed it to is week-one material, and that is the next beat.

What broke in week one

The first thing to break was my own bookkeeping. The original state file keyed each gap by a hash of its row text, which seemed stable until three different gaps turned out to carry byte-identical text: three “DEFERRED” rows all parked for my own eyeball. They hashed to a single id, so the state file recorded one verdict where there were three gaps, and the stop counter quietly undercounted. The fix was to key each gap by file plus line instead of text. After the change the three rows carry three distinct ids (af542ee2bb2a, c7f4c6659a4f, e004489ea229). A false count is the worst kind of bug in a loop whose whole job is counting toward done.

The second break is the one I am least proud of, because it looked like a success. I had wired a second brain behind the verdict, an agent that calls claude -p to reason over each gap, and the first contrast run reported 28 out of 28 agreement with the deterministic ranker. I almost wrote that number down as a finding. It was a lie. A greedy regex was grabbing the claude -p --output-format json result envelope instead of the model’s inner verdict, so every agent call hit a KeyError and fell back to the deterministic ranker (the brain tally read fallback:KeyError twenty-eight times).

The loop did not crash. It confidently produced a meaningless agreement, deterministic agreeing with deterministic. The fix was small, one file, eight lines added and two removed, shipped as PR #167 and merged at 0f42761. The lesson is bigger than the patch: a silent fallback that looks like success is more dangerous than a loud failure, because nothing asks you to look.

The third thing was not in my code at all. Run #2’s Actions summary came back carrying a GitHub deprecation annotation: the actions I had pinned (checkout@v4, setup-python@v5, upload-artifact@v4) run on Node.js 20, which GitHub is forcing up to Node.js 24, with the old runtime scheduled for removal on 2026-09-16. Nothing is breaking yet. But it is a reminder that an unattended loop accrues unattended infra rot underneath it: the platform keeps moving whether or not anyone is watching the run. I logged it and did not act on it, which is its own small discipline, because not every observed problem is this week’s problem.

The real answer to what broke, though, is none of those. The loop never reached DONE. It drained cleanly, twenty pending down to two over four weekday passes, and then the heartbeat simply stopped. Not with an error, not with a cap, not with a failed push. During the v1.0.0 release reconcile the workflow and its tooling were lifted off master under a “release tree wins” merge, and because the release tree did not carry the path, the loop vanished with no delete commit to mark it. The four runs are still real history, still ancestors of master, but the schedule that produced them is gone. Two gaps froze exactly where the last run left them, and nothing flagged that the loop was no longer alive. That is the quietest failure mode of all: not a loop that errors, but a loop that is silently no longer there.

What we keep, what we changed, what surprised us

What we keep. The deterministic brain stays the daily default. It is cheap, it is offline, and it is reproducible: the same gap log produces the same verdicts on every run, which is exactly what you want from the thing that wakes up unattended each weekday. The other keeper is a property I deliberately did not automate away: the loop never auto-acts. Across all four runs the passed set stayed empty. The triage routes every gap into a human inbox and merges nothing on its own. A loop that drains a backlog into decisions a person still makes is the right amount of autonomy to trust this early.

What we changed. Two things, both forced by the breakage above. The id scheme moved from a hash of the row text to file plus line, so identical-looking gaps stop collapsing into one verdict. And the agent brain’s parser was fixed to read the model’s inner verdict out of the claude -p result envelope instead of grabbing the envelope itself (PR #167). Neither change touched the daily heartbeat’s behavior, since the daily default is deterministic and the agent brain stays opt-in.

What surprised us. Once the parser was fixed and the agent brain actually cast its own verdicts, it did not simply reproduce the keyword ranker. On a small subset of gaps it disagreed with the deterministic routes, and when I read the disagreements the agent’s routes were the better ones: in one case it recognized that a prop shape was correct per the spec and routed the gap to won’t-fix, where the keyword brain had parked on the word “defer”. The agent reasons over the gap; the keyword brain matches strings. That is the durable finding.

Here is the honesty caveat that has to travel with it: the subset was six gaps, a single run, and claude -p ran with no seed or temperature control, so the exact count of disagreements wobbles from run to run. The qualitative claim (reasons versus matches) holds. The specific number does not, so I am not going to print it as if it were stable.

I should be equally careful about cost. The metered spend of those agent runs lives only in my session scratch space and the Pool 2 meter, so I can only report it roughly: on the order of a couple dozen metered claude -p calls in the first full 28-gap contrast, and a handful more in the patched six-gap re-run. Treat those as reported, not measured. I am not going to promote a rough read of a meter into a hard figure, and the per-gap agent verdicts behind them live in the same place, so they get the same caution.

The biggest surprise. It was not in any single run. It was the quiet death from the last section: the loop drained almost to done and then disappeared, and nothing told me. I had built memory, a stop condition, and a heartbeat, and I had not built the one check that would have caught this, which is whether the loop is still scheduled at all. An unattended loop needs a meta-check on its own pulse. “Is the heartbeat still alive?” is itself a thing to monitor, and it is the piece I will carry into the next loop I retrofit.

Retrofit automation loop: the four-step recipe

The recipe is short because the work is mostly recognition. A retrofit automation loop is any manual ritual plus the three things that make a ritual a loop: memory, a checkable stop, and a heartbeat. Find the ritual, then add the three pieces in order.

  • 1. Find the recurring manual ritual. The one you do “next session”, on a cadence, by hand. If you are its scheduler, it is a candidate.
  • 2. Give it a state file. A board it reads and writes, so progress survives between runs and a second run does not redo the first.
  • 3. Give it a machine-checked stop condition. A script that answers done or not-done without your judgment. Pick the ritual whose “done” is smallest and most checkable.
  • 4. Give it a heartbeat. A schedule that wakes it. Make empty runs cheap so the cadence costs almost nothing when there is no work.

The takeaway is one line: find the recurring manual ritual in your own harness, because that ritual is a loop waiting for a heartbeat.

Where this leaves us

For the first time one of our “loops” actually passed the test we wrote for it. It had a heartbeat, a state file, and a checkable stop, and it ran unattended for four weekday passes, draining the backlog from twenty pending toward two without me prompting a single turn. That is real, and it is smaller and more satisfying than the victory-lap version I almost wrote instead.

Then it taught its last lesson by disappearing. The v1.0.0 release lifted the workflow off master, and the heartbeat stopped without an error to announce it. So the honest landing is not “the loop is running”, it is the meta-lesson: an unattended loop needs a check that the loop itself is still alive, because the quietest failure is the one where the heartbeat is simply gone. The next part packages this shape into a reusable kit, loop-devkit-fixkit-blueprint, and that pulse check is the first thing it has to carry.

FAQ

What is a retrofit automation loop?

It is an existing manual ritual converted into a real loop by adding three things: a state file for memory, a machine-checked stop condition, and a heartbeat that wakes it on a schedule. You are not building automation from scratch. You are wrapping a ritual you already run so it can run itself.

How do I tell a real loop from a manual ritual?

Ask one question: does it run without you prompting each turn? If you are the scheduler, the memory, or the thing that decides when it is done, it is a ritual. A real loop wakes itself, remembers progress in a state file, and stops on a condition a script can check.

Which ritual should I convert first?

The one with the smallest machine-checkable “done”, not the most painful one. A ritual whose “done” is a judgment call forces you to debug the loop and the definition of done at once. Pick a ritual where a script can answer done or not-done, and where a real backlog is waiting so the first run is not empty.

What usually breaks in the first week of a scheduled loop?

The failure modes worth watching on any scheduled loop are false “done”s, state-file drift between runs, hitting an iteration or budget cap, and silent fallbacks that look like success. Our own first week hit four of these in different shapes: a false count from an id collision, a silent fallback that looked like success, slow infra rot under the runner, and the quietest one, the loop being lifted out of the schedule entirely during an unrelated release. The pattern is that the dangerous failures are the silent ones, the breakages that do not raise an error. See “What broke in week one” above for the full account of each.

Do I need a separate branch for the loop’s state?

It depends on where the schedule fires. Our heartbeat runs on master, because the Actions schedule only fires from the default branch, while the source data lives on develop. So the loop reads the data read-only from develop at run time and keeps its own state on master. If your data and your schedule already share a branch, you do not need to split them.