Skip to content

fix: eight things that stopped a second repository getting through the loop - #151

Merged
thedancingdeveloper merged 10 commits into
mainfrom
fix/context-budget-and-starved-target
Aug 4, 2026
Merged

fix: eight things that stopped a second repository getting through the loop#151
thedancingdeveloper merged 10 commits into
mainfrom
fix/context-budget-and-starved-target

Conversation

@thedancingdeveloper

@thedancingdeveloper thedancingdeveloper commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Eight fixes from one exercise: importing rdpapp — a Rust/TypeScript product
with a 634 KB gateway file and a 272 KB browser entry point — and driving items
through the loop until they landed. Closes #150, #154, #156, #157, #159.

Based on fix/validator-rejects-valid-patches (#145), not main, because the
code these change only exists there.

Every one was found by watching a real item fail. Two items now go end to
end that could not before.


Context: what each role could see

1. The planner had never seen the repository (the most consequential).
Its entire job is naming the files an item will change, and it was given the
brief and nothing else — so it could only name a path the brief had already
quoted. The first item that described a surface rather than a path replied
"I do not have the repository tree or actual file paths" with an empty target
list, and the implementer then wrote a diff against a stylesheet it had not
been shown. It now gets the tracked paths, bounded by the same budget.

2. A target too large to supply was dropped silently (#150). The
implementer was asked to change a file it had never seen. Now the item stops
before that call is paid for — escalated / context_unavailable, blocked,
costing no attempt — and --context-budget / $HARNESS_CONTEXT_BUDGET exist,
because the budget was a constant with a context_policy nothing ever passed.

3. …but only a missing primary target is fatal. The first version of
that guard blocked whenever any named target did not fit, which on a
repository with two large files lost an item whose real target had been
supplied. The planner orders targets by importance; supporting files are now
named in the prompt with an instruction not to change them.

4. The reviewer got the diff and nothing else (#156). Two of three reasons
in a rejection were "the diff does not show whether…", which no diff can —
while the implementer had 700,000 characters of the same repository. It now
gets the touched files, and a file too large to include is named as absent.

Instructions: what each role was told

5. The reviewer was still told to reason about "the diff alone." Supplying
the files was only half of it — with 637,317 characters in hand including the
whole target file, it still wrote "I cannot verify there isn't another path…".

6. The implementer was not told which checks judge it (#154). A correct
change was refused by cargo fmt --all -- --check, a command it had never
seen.

7. A retry did not know why the last attempt failed (#157). Three of four
attempts at one item were refused by the same formatter, each unaware the
previous had been. requeue keeps last_error deliberately; nothing passed it
on. Not resumption — D11 is untouched, no prior diff is fed back, and the
prompt says so in as many words.

8. The reviewer rejected for things that were not the work (#159). It could
not confirm "no other file changed" (a diff is the whole change) or that the
tree type-checked (Checks: passed reads as the author's claim; the harness
ran them). And it overruled an explicit scope statement — verifying every
requirement, quoting the exclusion back, and rejecting because the wider
problem remained. The prompt now states both facts and gains its first "do not
reject if": scope is the task's to set.


Evidence

Two items completed plan → implement → apply → checks → commit → review → done, each on its own branch. The second is the direct test of #8 — same
model, same diff, third attempt:

I cannot verify from this change alone that main.tsx has been updated to
use them, but that was explicitly out of scope. — APPROVED

What these do not solve is #158: 8 of 23 model diffs did not apply, and the
rate tracks target file size (0/1 against 258 bytes, 8/22 against 272 KB and
634 KB). That is a protocol question under D10 and is deliberately untouched.

Also corrects the reason-kind list in schemas.py and USAGE.md, four kinds
behind outcomes.py, and documents what each role is shown.

Gates

All four from the repository root with TMPDIR on a fast volume: pytest
green, ruff check, ruff format --check, mypy (full-project strict) clean.

Fourteen new tests, each asserting behaviour rather than a flag: that the
implementer role is never called when its primary target is missing, that a
supporting one is named instead of fatal, that the planner gets paths and not
contents, that a project with no checks tells its reviewer it is the only gate.

🤖 Generated with Claude Code

@thedancingdeveloper thedancingdeveloper changed the title fix: stop an item whose target does not fit, rather than paying to guess (#150) fix: four things that stopped a second repository getting through the loop (#150, #154, #156, #157) Aug 4, 2026
@thedancingdeveloper thedancingdeveloper changed the title fix: four things that stopped a second repository getting through the loop (#150, #154, #156, #157) fix: six things that stopped a second repository getting through the loop Aug 4, 2026
@thedancingdeveloper thedancingdeveloper changed the title fix: six things that stopped a second repository getting through the loop fix: eight things that stopped a second repository getting through the loop Aug 4, 2026
@thedancingdeveloper
thedancingdeveloper changed the base branch from fix/validator-rejects-valid-patches to main August 4, 2026 23:03
sprooty and others added 10 commits August 4, 2026 23:03
Found importing rdpapp, whose gateway is one 612 KB source file. The planner
named it correctly; the 60,000-character context budget could not hold it; the
selector recorded "named target exceeds remaining content budget" and moved
on, and the fallback then filled the whole budget with unrelated files —
migration SQL and a vendored patch directory. The implementer was called
anyway and asked to change a file it had never seen. It would have answered,
because a model short of evidence writes a plausible diff rather than refusing,
and the failure would have surfaced as a patch that did not apply.

Two changes, and they are separate ideas:

- A named target that cannot be supplied now stops the item BEFORE the
  implementer is called: escalated / context_unavailable, blocked, and costing
  no attempt, because no attempt could succeed and retrying will not make the
  file smaller. The message names the file, its size, the budget and the flag.
- The budget is configurable at last — --context-budget and
  $HARNESS_CONTEXT_BUDGET. It was a constant, so a repository with large files
  could not be worked on at all, and no single number is right for every
  project.

The omission reasons become constants, because the executor now has to tell
the two apart rather than only print them.

Also corrects the reason-kind list in the API schema and USAGE, which had
fallen four kinds behind the taxonomy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reviewer's prompt has always carried `Checks: {checks}`. The
implementer's carried nothing, so the model writing the change was graded by
gates it was never shown.

Measured on rdpapp. The implementer produced exactly the right change — the
validation function in the right module, wired into main before serving, with
a test for the rejection and a test for the case that must keep working — and
`cargo fmt --all -- --check` refused it over line wrapping. That is one
attempt and two model calls spent discovering something the harness knew
before it asked.

Naming the commands does not weaken the gate: it still runs, and it still
refuses. It stops the gate being a secret.

Empty when a project configured no checks, so a run without them reads exactly
as it did before — asserted, because a prompt that grows a stray blank section
is how a fix like this quietly changes every other run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reviewer is asked whether a change is wired in where it should be, and
whether anything unrelated moved, and was given only the change.

Measured on rdpapp. A correct diff — the right helper, called from main before
serving, with tests for both the rejection and the case that must keep working
— was rejected, and two of the three reasons were of this form:

  "The diff does not show whether main's returned error is surfaced in the
   intended way, whether the rest of startup avoids side effects before this
   check … or that there are no other startup routes bypassing the check."

No diff can show any of that. And the prompt lists "the task cannot be judged
from what you were given" as grounds to reject — so a diff-only reviewer
rejects for the shape of its own prompt rather than for a defect in the work.
Meanwhile the implementer had been shown 700,000 characters of the same
repository.

The touched files now follow the diff, at their post-change state, bounded by
the same context budget. A file too large to include is named as absent: a
reviewer that does not know its view is partial will treat it as complete,
which is worse than one that knows.

This strengthens the gate rather than weakening it. The reviewer's answer stays
its own, and nothing about what it may reject has changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A retry re-planned from the brief with no memory of what had just happened, so
it repeated the same mistake blind. Measured on rdpapp: three of four attempts
at one item were refused by `cargo fmt --all -- --check`, each one unaware the
previous had been, and each costing a planner call and an implementer call
against a 612 KB file.

The material was already there. `requeue` deliberately keeps `last_error` —
"it is the only record of why the item failed" — and `WorkRecord` carries it to
the executor. Nothing passed it to the model doing the work.

**This is not a resumption**, and the prompt says so in as many words. D11 and
the attempt machinery are untouched: a decided attempt still ends
resumability, a retry still re-plans against the current brief, and nothing
here is treated as progress. What changes is only that the new attempt knows
what the last one was refused for, which is what a person retrying would read
first.

Bounded at 4,000 characters, because a check's output can be a whole build log
and the useful part is where the tool says what it objected to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n scope

Three of the four fixes on this branch change what a model is given, which is
what it is judged on. That belongs in USAGE next to the other guarantees
rather than only in the commits that made it true.

The last paragraph is the one that cost the most to learn and is not a harness
behaviour at all: a brief that does not say what is out of scope collects
rejections that are each individually reasonable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Supplying the touched files was half the fix. The prompt still framed the
answer as "anything the diff claims that the diff alone does not show", so the
reviewer went on writing that section as though the diff were all it had.

Measured after the previous commit landed. The reviewer received 637,317
characters — the whole of the 612 KB file the change touched — and rejected
with:

  "I cannot verify there isn't another snippet read/query path elsewhere that
   should also have been extended … I cannot verify that the new row type is
   fully compatible with all existing callers; the diff does not show the
   surrounding type signatures."

Every one of those was answerable from the file it was holding.

So section 2 now says *what you were given* rather than *the diff alone*, and
says plainly that the files are there and must be read before claiming
something could not be verified. The reject-if list keeps "the task cannot be
judged from what you were given" — it is a real ground and it should stay —
with the two qualifications that make it honest: after looking, and not for
evidence the task never asked for.

Nothing about the reviewer's authority changed. It may still reject anything;
it is asked to do so on what is in front of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The planner's entire job is to name the files an item will change. It was
given the brief and nothing else, and told — correctly — not to invent a path.
So it could only ever name a path the brief had already quoted.

That held up for three items whose briefs quoted paths, and broke on the first
one that described a surface instead:

  "I do not have the repository tree or actual file paths, so naming …"

with an empty targets list, which is the honest answer to an impossible
question. Target-first context selection then had no target, the implementer
received relevance-guessed files — a terminal adapter and a vendored Kerberos
patch — and wrote a diff against a stylesheet it had not been shown. It did not
apply.

The tracked paths now follow the brief. Paths only: the planner needs to know
what exists, and reading it is the implementer's job, which has its own budget.
A listing that will not fit says how many paths are missing rather than
stopping silently at an arbitrary letter of the alphabet.

Bounded by the same context budget, so a repository with a very large tree
cannot turn the cheapest call in the pipeline into the most expensive one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard added earlier blocked an item when any named target did not fit. On
a repository with more than one large file that is too strict, and it cost a
real item:

  FAIL R3: the planner's target(s) src/db.rs (144480 bytes),
  docs/rdpappv2/current-state.md (21829 bytes) do not fit …

The file the change actually belonged in — 634 KB — had been supplied. What did
not fit was a supporting file and a design document, neither of which the item
was going to edit. Blocking there loses the work to protect a file nobody was
touching.

The planner is asked to order its targets by importance, so the first usable
one is the file the work is in. Losing that is still fatal and still escalates,
for the original reason: the implementer would be asked to change a file it has
not been shown, and it will answer rather than refuse. Losing a supporting file
is not fatal — it is named in the prompt, with an instruction not to change it
and to say so if the task cannot be done without it.

Which keeps the property that mattered: nothing is ever silently asked to edit
a file it has not read.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An item that added two constants to one 5.6 KB file was rejected. The
reviewer's own section 1 confirmed the change was correct, correctly named,
correctly grouped and within scope. It rejected anyway, for two reasons:

  "The diff does not show whether web/src/main.tsx remains unchanged in the
   actual working tree beyond what was provided here."

  "I cannot verify that the file still type-checks, because no build output
   was provided beyond the claim 'Checks: passed.'"

Both are answerable, and the harness was the one withholding the answers.

A diff produced from the repository IS the whole change — a file absent from
it is unchanged, and that is a property of how the diff was made, not a claim
by whoever wrote the code. The prompt now says so.

And `Checks: passed` reads exactly like the author asserting their own work is
fine, which a reviewer told to assume the work is wrong will discount — as this
one did, in as many words. The harness ran those commands, on that tree, after
the change and before the reviewer. The prompt now names them and says who ran
them.

A project with no checks is told that too, plainly, rather than left to infer
from silence that there were some.

Neither is a softening. Both remove uncertainty that was never real, so the
reviewer can spend its scepticism on the work.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An item declared two constants in a contract file and said, in a paragraph
headed "Scope, stated so it can be judged", that replacing the corresponding
literals in a 272 KB file was deliberately out of scope and why.

The reviewer verified every stated requirement — the right file, the right
names, the right values, the right grouping, no other file touched, the
typecheck passing — quoted the exclusion back in its own answer, and rejected:

  "the broader problem statement ties those constants to repeated 14 literals
   in web/src/main.tsx, and that code remains unchanged, so the repository
   still has the inconsistency the task described … the implementation itself
   is fine"

That is the reviewer overruling the plan author about what the item was for.
The gate exists to catch work that does something adjacent to what was asked,
or claims more than it did; rejecting work for doing exactly what was asked is
the same failure inverted, and it is more expensive, because the answer is to
write the item again identically.

So the prompt gains its first "do not reject if": where a task states what is
out of scope, judge the change against that. Disagreeing with the scope is a
note for a person, and a note is the right size for it.

Nothing else about the reviewer's authority moves. It may still reject work
that is wrong, incomplete against what the task did ask, or overreaching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedancingdeveloper
thedancingdeveloper force-pushed the fix/context-budget-and-starved-target branch from ed7e195 to 5e2aa6e Compare August 4, 2026 23:07
@thedancingdeveloper
thedancingdeveloper merged commit 83a5060 into main Aug 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: a planner target larger than the context budget is dropped, and the implementer is asked to change a file it cannot see

1 participant