Repository navigation
Treat a project's own agent instructions as information (for discussion) - #53
Open
anakanemison wants to merge 1 commit into
Open
anakanemison wants to merge 1 commit into
anakanemison wants to merge 1 commit into
Conversation
Projects are starting to ship instructions for their own coding agents (CLAUDE.md, AGENTS.md, .claude/, .cursor/). They are written for everyday work on the project, and some contradict what a migration must do. The skill says nothing about them, so whether an agent follows such a directive, ignores it, or sets it aside and says so is left to chance. Observed in any2bazel-lab: YugabyteDB ships a src/AGENTS.md saying to use only its build script and never to run cmake or test binaries directly. That is exactly the information a migration needs (the script sets up what the build depends on), phrased as a prohibition a migration and its verification have to break. In the one pair observed, the guided agent never opened the file and the unguided one read it and ran cmake anyway without saying why. No run has yet shown an agent following such a directive to the migration's cost. Adds a short paragraph to "Confirm scope": read them as information, do what the migration requires where they conflict, and record each directive set aside, with the reason, in any2bazel.json under "set_aside" (record-only; config.load ignores unknown keys; listed with the other fields). In the lab's later runs with this change, agents on pono, QGIS and YugabyteDB recorded the directives they set aside, each with a reason. This is a proposal for discussion. Whether setting aside a project's own instructions is better than following them is a judgment, and the evidence is a handful of runs. Lab-Proposal: P24 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 18 of 23 of a stacked series (tracking issue: #35). Based on
U17-done-means-reproducible, the previous part: review and merge that first.Kind: discussion. Lab proposal: P24.
What the migration runs showed
yugabyte-db's
CLAUDE.mdsays to use onlyyb_build.sh. Agents now record the directives they set aside, with reasons (runs 084, 089, 091).The change
Projects are starting to ship instructions for their own coding agents
(CLAUDE.md, AGENTS.md, .claude/, .cursor/). They are written for everyday
work on the project, and some contradict what a migration must do. The
skill says nothing about them, so whether an agent follows such a
directive, ignores it, or sets it aside and says so is left to chance.
Observed in any2bazel-lab:
YugabyteDB ships a src/AGENTS.md saying to use only its build script and
never to run cmake or test binaries directly. That is exactly the
information a migration needs (the script sets up what the build depends
on), phrased as a prohibition a migration and its verification have to
break. In the one pair observed, the guided agent never opened the file and
the unguided one read it and ran cmake anyway without saying why. No run
has yet shown an agent following such a directive to the migration's cost.
Adds a short paragraph to "Confirm scope": read them as information, do what the
migration requires where they conflict, and record each directive set
aside, with the reason, in any2bazel.json under "set_aside" (record-only;
config.load ignores unknown keys; listed with the other fields).
In the lab's later runs with this change, agents on pono, QGIS and
YugabyteDB recorded the directives they set aside, each with a reason.
This is a proposal for discussion. Whether setting aside a project's own
instructions is better than following them is a judgment, and the
evidence is a handful of runs.
Testing
pytest tests/at this commit: 196 passed (130 onmain).🤖 Generated with Claude Code