State File Pattern: Memory Outside the Context Window

TL;DR

  • The model forgets everything between runs; the loop must not. A small markdown state file on disk is the spine that turns repeated runs into one continuous process.
  • Keep the schema to four greppable buckets: tried, passed, open, and blocked. Small and on-disk beats clever and in-context.
  • Prove it with the resume test: kill the session, rerun, and confirm the loop picks up where it stopped. A state file you have not killed-and-resumed is unproven.

📊 Result-Proof: A real loop ran two of five units, then stopped. A cold rerun read the state file and resumed at unit three (create-baz), not unit one. Final state: all five passed. Bare-loop run, 2026-06-14; no model was invoked, so there are no token figures to report.

Your scheduled loop ran last night. The session is gone, the context window with it, and this morning’s run has no idea what last night already did. That is the problem the state file pattern solves. You will give a loop a four-field markdown state file, run it, kill it mid-list, rerun it cold, and confirm it resumes instead of restarting.

Prerequisites:

  • You have skimmed Part 2 (loop anatomy) and Part 3 (stop conditions) of this series.
  • You can run a loop locally: a scheduled prompt, or a bare claude -p loop with an iteration cap.
  • You are comfortable with git, grep, and a markdown file in your repo.

The frame for the whole series stays the same:

A loop is a recursive goal: you define a purpose and a verifiable stop condition, and the system iterates agents against it until the condition holds — without you prompting each turn.

Why is the context window the wrong place to keep loop state?

The context window is volatile, and a loop needs durable memory. Compaction trims it mid-run, a session death wipes it, and re-feeding it every run costs tokens. Addy Osmani frames long-running agents as hitting three walls, context, state, and self-verification; this article owns the state wall. Durable progress belongs on disk, where a cold process can read it.

You have probably read that your AI agent forgets everything, and that the fix is a project memory file. That is true, and it solves a real problem, but a different one than a loop has. A CLAUDE.md or MEMORY.md file fixes forgetting between sessions: the conventions, architecture, and house rules an agent should reload every time it starts. Our own notes on keeping that file healthy and why it matters cover that half well.

A loop has a second kind of forgetting, between runs. What did last night’s run already try? Which units passed? Where did it get stuck? Project memory does not answer those questions, because they are about operational progress, not house rules. Same word, “memory,” two different jobs. This article is about the second one: the run-progress file that lets tonight’s run continue last night’s work.

Key insight: Context is where the agent thinks; the repo is where the loop remembers. Conflating the two is why “it worked last night” does not survive to morning.

How do you design a state schema: tried, passed, open, blocked?

The minimum useful loop state is four greppable buckets, kept small enough to read on one screen. State-file patterns in the wild converge on the same three properties: small, greppable, and on disk. The schema below is the whole contract. A unit of work lives in exactly one bucket, and the loop moves it as work happens.

# Loop State: <task name>
# A unit lives in exactly one bucket. Read fresh each run; written each iteration.


## tried
## passed
## open
- create-foo
- create-bar
- create-baz


## blocked

Each bucket earns its place. open is the work queue, the list of units not yet done. passed is the resume anchor, the proof a unit is finished so the next run skips it. tried records attempts that did not pass, so the loop does not bang on the same dead end forever. blocked is the human-escalation lane, units that need a person before the loop can move them.

Keep it greppable. grep -A99 '## open' state.md tells you the queue; grep -c '^- ' per section gives you a count. The trade-off is real: richer schemas with priorities, timestamps, and owners buy you reporting, but they cost write-discipline, and a loop that cannot maintain its own file honestly is worse than a small one. Start at four fields. Add a fifth only when a run actually needs it.

Key insight: A state schema you cannot grep in one command is already too big for a loop to maintain honestly.

State file vs. a board (Linear/GitHub Issues): when does each win?

A local markdown file wins for a single agent that needs fast, greppable, in-repo memory; a board like Linear or GitHub Issues wins when humans need visibility or work routes across tools. The decision is not about which is better. It is about who reads the state and how often.

Use a file when the loop is the only reader and writer, the work lives in one repo, and you want a grep-able artifact that travels with the code in git history. The file is fast, it has no API rate limit, and it diffs cleanly in a PR. Use a board when several people watch the loop, when work needs an audit trail outside the repo, or when a finding has to become a ticket someone triages by hand.

Many loops use both, and that is the honest answer for most teams. The file is the loop’s working memory, read fresh every run. The board is the human-facing inbox, where the loop posts the units it marked blocked. Keep the machine-readable spine local and cheap; push only what a person must see to the slower, shared system.

Key insight: The board is for the humans watching the loop; the state file is for the loop watching itself.

What state-file patterns already work in production?

The same small file shows up everywhere serious loops run. Anthropic’s guidance for long-running agents persists progress to claude-progress.txt and tracks a feature_list.json whose entries stay passes: false until verified, with git history itself treated as memory. Their failure-mode list even names “dirty state” as a way overnight loops go wrong, which is exactly what a disciplined state file guards against.

Open-source loop runners landed on the same shape. snarktank/ralph keeps its memory in progress.txt plus prd.json, so a fresh agent each lap reads them to learn what is done. continuous-claude relays state between runs through a SHARED_TASK_NOTES.md baton file. The HumanLayer RPI approach goes further: it resets the conversation each iteration and reloads the spec, leaning on the filesystem and git as the memory that survives the reset. (Treat the RPI specifics as reported rather than first-hand; the pattern is what matters here.)

We rely on the same idea in-house. A failure-log convention like the one in our failure-log pattern post doubles as loop-state precedent: a tried list is a failure log a machine can resume from. Observability JSONL logs (one event per line, written by a small observe.py), memory.md decision logs, and gap-log files all serve the same role, a durable record a cold run can read. The shared property is not the format. It is that the record lives on disk and gets read fresh.

Key insight: Four independent loop systems converged on the same artifact: a small file on disk, read fresh every run. That convergence is the signal.

Hands-on: how do you make a loop resume where it stopped?

Give a real loop the four-field state file, run it, stop it mid-list, then rerun it cold and watch it continue. The run below is honest about its limits: it is a bare-loop fallback, not /goal. /goal is an interactive grader that cannot be captured cleanly in a headless run, so this uses a plain iteration-capped loop. No model was invoked; the work units are deterministic file-creates, because the point is to prove the resume mechanism, not to stage a graded transcript.

The loop reads state.md, picks the first open unit, creates a file for it, moves it to passed, and writes the file back. It holds nothing in memory between iterations. Run one stops after two units, with an iteration cap standing in for a kill:

Terminal window
$ python3 loop.py run-1 2
[run-1] iter 1: completed create-foo -> marked passed
[run-1] iter 2: completed create-bar -> marked passed
[run-1] finished: 2 unit(s) this run

State after run one: two passed, three still open.

## tried


## passed
- create-foo
- create-bar


## open
- create-baz
- create-qux
- create-quux


## blocked

Now rerun cold, a fresh invocation with no memory of run one:

Terminal window
$ python3 loop.py run-2 99
[run-2] iter 1: completed create-baz -> marked passed
[run-2] iter 2: completed create-qux -> marked passed
[run-2] iter 3: completed create-quux -> marked passed
[run-2] no open units left; loop complete
[run-2] finished: 3 unit(s) this run

There is the proof. Run two’s first action is create-baz, the third unit, not create-foo. The fresh process read state.md, saw foo and bar already in passed, and resumed from the first open unit. The two files from run one keep their original timestamps; run two never touched them. Final state: all five in passed, open empty.

To be plain about scope: this minimal run only exercised the open to passed move. The tried and blocked buckets are part of the schema but were never hit here, because every unit succeeded on the first attempt. I am not going to invent a blocked example to fill them in. In a real loop, a failed verification is what writes tried, and a unit that needs a human is what writes blocked.

Key insight: A state file you have never killed-and-resumed is a hope, not a mechanism. The resume test is the only proof.

How do you keep a loop state file healthy over time?

Loop-state hygiene is about who writes and when, not about trimming for readability. Pruning a project memory file is curation: a human removes stale conventions so the agent reads less noise. A loop state file is different, because an automated process writes it between runs, with no human in the moment to catch a bad write. Three rules keep it trustworthy.

First, name a single writer. If two agents update the same file in the same run, they race, and the file ends up describing a state that never existed. One writer per file is the rule; the moment you want parallel workers, you need isolation, which is the subject of the next part. Second, archive completed runs. When every unit is passed, move that block to an archive file or let git history hold it, so the live file stays one screen long and the next run reads only what is open.

Third, let stale entries decay. An open unit that no run has touched for several laps is not really open; it is stuck. Promote it to blocked so a human sees it, rather than letting the loop skip past it forever. These three rules are specific to a file a machine writes on a schedule. They are not the same as keeping a hand-edited memory file lean.

Key insight: The dangerous state file is the one nobody owns. Name the single writer, or the loop corrupts its own memory.

What you have, and what is next

You now have a four-field state file and a loop that passes the resume test. The model still forgets between runs; the repo does not, and that is the whole point. A passed entry is only as trustworthy as the check that set it, so pair this with a real stop condition a different model can verify from Part 3.

Watch for the failure modes: two agents writing the same file (isolation fixes it), a file that grows without bound (archive fixes it), and a fake passed (verification fixes it). The first of those, running more than one agent without collisions, is exactly where this series goes next, in loop-parallel-worktrees-subagents.

FAQ

What is a state file pattern for AI agent loops?

A state file pattern is a small markdown or JSON file on disk that records what a loop has tried, passed, left open, and blocked. The loop reads it fresh at the start of every run and writes it back as work happens, so progress survives between sessions even when the context window is gone.

Why not just keep loop state in the context window?

Context is volatile. Compaction trims it mid-run, a session death wipes it, and re-feeding it every run costs tokens. A file on disk is durable, greppable, and free to re-read, which is what a loop needs to continue work across separate runs.

State file or a board like Linear or GitHub Issues?

Use a file for a single agent that needs fast, greppable, in-repo memory. Use a board when humans need visibility or when work routes across tools and needs an audit trail. Many loops use both: the file as the loop’s working memory, the board as the human-facing inbox.

How do I know my loop will actually resume?

Run the resume test. Kill the session mid-run, rerun the loop cold, and confirm it continues from the passed units instead of restarting from the first one. A state file you have not killed-and-resumed is unproven, no matter how clean the schema looks.