Skip to content

fix: preserve review provenance and validate loop readiness #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
roodboi merged 2 commits into from
Sep 14, 2026
Merged

fix: preserve review provenance and validate loop readiness #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eaa720d
fix: preserve review provenance and validate loop readiness
roodboi Sep 14, 2026
5a381d9
fix: scope source freshness to configured Git paths
roodboi Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/writeback-evolution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,28 @@ receipt. Global and plugin changes always remain proposal-only.

Manual writeback remains useful, but it is no longer the only source of review
signal. Setup creates `reconciliation.json` in the selected canonical root.
Keep the portable source definition in the project's canonical root and include
it in version control. A missing definition can be restored with `fclt ai review
init --project --dry-run --json`, followed by the same command without `--dry-run`.
Initialization preserves existing valid configuration and does not reset writebacks,
queues, or cursors. Run `fclt ai loop preflight --project --json` afterward: readiness
requires valid source configuration, a valid source selection, and writable state.
Missing or invalid configuration is deterministic and is not retried by the loop.

Machine-specific scheduler identity, queues, cursors, and history belong in fclt's
OS application-data store. Review mirrors live under the global review root with
project identity. Do not move project evidence into the global queue to solve a
missing project configuration. Shared operating instructions and reusable source
recipes may be global; each project's source selection and capability decisions
remain explicit and project-scoped.

The loop reuses source writebacks when creating proposals, preserving their original
context instead of copying them into synthetic capability gaps. A linked pending
proposal does not resolve its signal family. Use an explicit terminal disposition
only after outcome evidence supports it. Successful scans whose observed latest
source timestamp is already covered do not become stale merely because the source
is quiet; newer uncovered activity and unverified old cursors still warn.

Run a bounded review window before deciding that nothing is pending:

```bash
Expand Down
4 changes: 4 additions & 0 deletions src/activity-action-contract.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@ export function activityActionClass(args: {
item: LoopQueueItem;
proposal?: AiProposalRecord | null;
}): ActivityActionClass | null {
// Linked signals remain open, but only the proposal owns its decision action.
if (args.item.kind === "signal" && args.item.proposalId) {
return null;
}
if (args.item.state === "resolved" || args.item.state === "deferred") {
return null;
}
Expand Down
9 changes: 9 additions & 0 deletions src/ai-cli.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1104,6 +1104,15 @@ describe("ai CLI", () => {
expect(draftOut.errors).toEqual([]);
expect(draftOut.logs.join("\n")).toContain("Drafted EV-00001");

const reviewOut = await captureConsole(async () => {
await aiCommand(["evolve", "review", "EV-00001", "--json"]);
});
expect(reviewOut.errors).toEqual([]);
expect(JSON.parse(reviewOut.logs.join(""))).toMatchObject({
id: "EV-00001",
status: "in_review",
});

const acceptOut = await captureConsole(async () => {
await aiCommand(["evolve", "accept", "EV-00001"]);
});
Expand Down
69 changes: 47 additions & 22 deletions src/ai.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3489,18 +3489,24 @@ async function writebackCommand(argv: string[]) {
evidence: parseEvidence(commandArgs),
});
if (commandArgs.includes("--json")) {
console.log(JSON.stringify(portableWritebackRecord(record), null, 2));
await writeCliOutput(
JSON.stringify(portableWritebackRecord(record), null, 2)
);
return;
}
console.log(`Recorded writeback ${record.id}`);
console.log(JSON.stringify(portableWritebackRecord(record), null, 2));
await writeCliOutput(
JSON.stringify(portableWritebackRecord(record), null, 2)
);
return;
}

if (sub === "list") {
const rows = await listWritebacks({ rootDir });
if (commandArgs.includes("--json")) {
console.log(JSON.stringify(rows.map(portableWritebackRecord), null, 2));
await writeCliOutput(
JSON.stringify(rows.map(portableWritebackRecord), null, 2)
);
return;
}
console.log(`writebacks root: ${rootDir}`);
Expand All @@ -3527,7 +3533,7 @@ async function writebackCommand(argv: string[]) {
? await groupWritebacks({ rootDir, by: byValue })
: await summarizeWritebacks({ rootDir, by: byValue });
if (commandArgs.includes("--json")) {
console.log(JSON.stringify(rows, null, 2));
await writeCliOutput(JSON.stringify(rows, null, 2));
return;
}
for (const row of rows) {
Expand All @@ -3547,7 +3553,9 @@ async function writebackCommand(argv: string[]) {
if (!row) {
throw new Error(`Writeback not found: ${id}`);
}
console.log(JSON.stringify(portableWritebackRecord(row), null, 2));
await writeCliOutput(
JSON.stringify(portableWritebackRecord(row), null, 2)
);
return;
}

Expand All @@ -3558,8 +3566,12 @@ async function writebackCommand(argv: string[]) {
throw new Error("writeback link requires an id and --issue");
}
const row = await linkWritebackIssue(id, issue, { rootDir });
console.log(`Linked ${row.id} to ${issue}`);
console.log(JSON.stringify(portableWritebackRecord(row), null, 2));
if (!commandArgs.includes("--json")) {
console.log(`Linked ${row.id} to ${issue}`);
}
await writeCliOutput(
JSON.stringify(portableWritebackRecord(row), null, 2)
);
return;
}

Expand All @@ -3586,8 +3598,12 @@ async function writebackCommand(argv: string[]) {
nextTrigger: parseStringFlag(commandArgs, "--next-trigger"),
expectedOutcome: parseStringFlag(commandArgs, "--expected-outcome"),
});
console.log(`Updated disposition for ${row.id}`);
console.log(JSON.stringify(portableWritebackRecord(row), null, 2));
if (!commandArgs.includes("--json")) {
console.log(`Updated disposition for ${row.id}`);
}
await writeCliOutput(
JSON.stringify(portableWritebackRecord(row), null, 2)
);
return;
}

Expand All @@ -3600,8 +3616,14 @@ async function writebackCommand(argv: string[]) {
sub === "dismiss"
? await dismissWriteback(id, { rootDir })
: await promoteWriteback(id, { rootDir });
console.log(`${sub === "dismiss" ? "Dismissed" : "Promoted"} ${row.id}`);
console.log(JSON.stringify(portableWritebackRecord(row), null, 2));
if (!commandArgs.includes("--json")) {
console.log(
`${sub === "dismiss" ? "Dismissed" : "Promoted"} ${row.id}`
);
}
await writeCliOutput(
JSON.stringify(portableWritebackRecord(row), null, 2)
);
return;
}

Expand Down Expand Up @@ -3639,7 +3661,7 @@ async function evolveCommand(argv: string[]) {
asset: parseStringFlag(commandArgs, "--asset"),
});
if (commandArgs.includes("--json")) {
console.log(JSON.stringify(assessment, null, 2));
await writeCliOutput(JSON.stringify(assessment, null, 2));
return;
}
console.log(`recommendation: ${assessment.recommendation}`);
Expand All @@ -3664,7 +3686,7 @@ async function evolveCommand(argv: string[]) {
asset: parseStringFlag(commandArgs, "--asset"),
});
if (commandArgs.includes("--json")) {
console.log(JSON.stringify(proposals, null, 2));
await writeCliOutput(JSON.stringify(proposals, null, 2));
return;
}
for (const proposal of proposals) {
Expand All @@ -3678,7 +3700,7 @@ async function evolveCommand(argv: string[]) {
if (sub === "list") {
const rows = await listProposals({ rootDir });
if (commandArgs.includes("--json")) {
console.log(JSON.stringify(rows, null, 2));
await writeCliOutput(JSON.stringify(rows, null, 2));
return;
}
for (const row of rows) {
Expand All @@ -3696,7 +3718,7 @@ async function evolveCommand(argv: string[]) {
if (!row) {
throw new Error(`Proposal not found: ${id}`);
}
console.log(JSON.stringify(row, null, 2));
await writeCliOutput(JSON.stringify(row, null, 2));
return;
}

Expand All @@ -3723,8 +3745,10 @@ async function evolveCommand(argv: string[]) {
note: parseStringFlag(commandArgs, "--note"),
allowEarly: commandArgs.includes("--allow-early"),
});
console.log(`Verified ${row.id} as ${effectiveness}`);
console.log(JSON.stringify(row, null, 2));
if (!commandArgs.includes("--json")) {
console.log(`Verified ${row.id} as ${effectiveness}`);
}
await writeCliOutput(JSON.stringify(row, null, 2));
return;
}

Expand Down Expand Up @@ -3795,8 +3819,9 @@ async function evolveCommand(argv: string[]) {
: sub === "promote"
? "Promoted"
: "Applied";
console.log(`${verb} ${row.id}`);
console.log(JSON.stringify(row, null, 2));
await writeCliOutput(
`${commandArgs.includes("--json") ? "" : `${verb} ${row.id}\n`}${JSON.stringify(row, null, 2)}`
);
return;
}

Expand Down Expand Up @@ -3840,7 +3865,7 @@ async function reviewCommand(argv: string[]): Promise<void> {
dryRun: commandArgs.includes("--dry-run"),
force: commandArgs.includes("--force"),
});
console.log(
await writeCliOutput(
json
? JSON.stringify(result, null, 2)
: `${result.created ? "Initialized" : "Using"} reconciliation config ${result.path}`
Expand All @@ -3850,7 +3875,7 @@ async function reviewCommand(argv: string[]): Promise<void> {
if (sub === "status") {
const { reconciliationStatus } = await import("./reconciliation");
const result = await reconciliationStatus({ homeDir, rootDir });
console.log(
await writeCliOutput(
json
? JSON.stringify(result, null, 2)
: `reconciliation: ${result.configured ? (result.coverageState ?? "not-run") : "not-configured"}\nconfig: ${result.configPath}\nstate: ${result.statePath}`
Expand All @@ -3872,7 +3897,7 @@ async function reviewCommand(argv: string[]): Promise<void> {
sourceIds: parseRepeatedFlag(commandArgs, "--source"),
incremental: commandArgs.includes("--incremental"),
});
console.log(
await writeCliOutput(
json
? JSON.stringify(result, null, 2)
: [
Expand Down
82 changes: 74 additions & 8 deletions src/evolution-loop.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1024,7 +1024,7 @@
);
expect(stale.coverageComplete).toBe(true);
expect(stale.status).toBe("complete");
expect(stale.freshness.state).toBe("stale");

Check failure on line 1027 in src/evolution-loop.test.ts

View workflow job for this annotation

GitHub Actions / test

error: expect(received).toBe(expected) 

Expected: "stale"
Received: "current"

      at <anonymous> (/home/runner/_work/fclt/fclt/src/evolution-loop.test.ts:1027:35)
expect(freshnessItem).toMatchObject({
kind: "coverage",
state: "blocked",
Expand Down Expand Up @@ -1685,7 +1685,7 @@
expect(item?.approvalRequired).toBe(false);
});

it("records retry failure state and audit history without hiding the error", async () => {
it("records permanent configuration failure and audit history without retrying", async () => {
const project = await makeProject();
await enableEvolutionLoop({
...project,
Expand All @@ -1700,21 +1700,21 @@
now: () => new Date("2026-01-03T00:00:00.000Z"),
});
expect(failed.status).toBe("failed");
expect(failed.attempts).toHaveLength(3);
expect(failed.attempts).toHaveLength(1);
expect(await Bun.file(failed.artifactPath).exists()).toBe(true);
const state = JSON.parse(
await readFile(
facultAiEvolutionLoopStatePath(project.homeDir, project.rootDir),
"utf8"
)
);
expect(state.lastFailure.attempts).toBe(3);
expect(state.lastFailure.attempts).toBe(1);
const audit = await readFile(
facultAiEvolutionLoopAuditPath(project.homeDir, project.rootDir),
"utf8"
);
expect(audit).toContain('"status":"failed"');
expect(audit).toContain('"attempt":3');
expect(audit).toContain('"attempt":1');
});

it("keeps proposal action locators in failed-run activity snapshots", async () => {
Expand Down Expand Up @@ -1793,7 +1793,7 @@
});

expect(failed.status).toBe("failed");
expect(failed.attempts).toHaveLength(3);
expect(failed.attempts).toHaveLength(1);
expect(failed.attempts[0]?.error).toContain("missing-source");
const history = await queryActivityHistory({
homeDir: project.homeDir,
Expand Down Expand Up @@ -2448,7 +2448,7 @@
const canonicalSignal = third.queue.find(
(item) => item.kind === "signal" && item.familyId === familyA
);
expect(canonicalSignal?.state).toBe("resolved");
expect(canonicalSignal?.state).toBe("open");
expect(canonicalSignal?.proposalId).toBe(aliasProposal!.id);
expect(canonicalSignal?.familyAliases).toContain(familyB!);
expect(
Expand All @@ -2470,7 +2470,7 @@
const postMergeSignal = fourth.queue.find(
(item) => item.kind === "signal" && item.familyId === familyA
);
expect(postMergeSignal?.state).toBe("resolved");
expect(postMergeSignal?.state).toBe("open");
expect(postMergeSignal?.proposalId).toBe(aliasProposal!.id);
expect(postMergeSignal?.familyAliases).toContain(familyB!);
expect(await listWritebacks(project)).toHaveLength(1);
Expand Down Expand Up @@ -2641,7 +2641,7 @@
first.queue.filter(
(item) => item.kind === "signal" && item.state !== "resolved"
)
).toHaveLength(0);
).toHaveLength(2);
expect(
writebacks.flatMap((entry) => entry.issueLinks ?? []).sort()
).toEqual(["EXAMPLE-101", "EXAMPLE-102", "EXAMPLE-201", "EXAMPLE-202"]);
Expand Down Expand Up @@ -2737,6 +2737,72 @@
expect(report.mutations.every((mutation) => !mutation.applied)).toBe(true);
});

it("reuses source writebacks and keeps pending proposal families unresolved", async () => {
const project = await makeProject();
await Bun.write(
join(project.rootDir, "reconciliation.json"),
JSON.stringify({
version: 1,
sources: [{ id: "writebacks", type: "writebacks" }],
})
);
const row = await addWriteback({
...project,
kind: "capability_gap",
summary: "Keep integration checks read-only.",
suggestedDestination: "@project/instructions/CHECKS.md",
evidence: [{ type: "test", ref: "source-evidence" }],
});
await setWritebackDisposition(row.id, "apply-local", {
...project,
target: "@project/instructions/CHECKS.md",
expectedOutcome: "Checks remain read-only",
});
await enableEvolutionLoop(project);
const report = await runEvolutionLoop({ ...project, since: "2020-01-01" });
expect(report.status).toBe("complete");
expect(await listWritebacks(project)).toHaveLength(1);
const proposals = await listProposals(project);
expect(proposals).toHaveLength(1);
expect(proposals[0]?.sourceWritebacks).toEqual([row.id]);
expect(report.queue.find((item) => item.kind === "signal")?.state).not.toBe(
"resolved"
);
await setWritebackDisposition(row.id, "resolve-watch", {
...project,
target: "@project/instructions/CHECKS.md",
expectedOutcome: "Implemented and verified",
});
await rejectProposal(proposals[0]!.id, {
...project,
reason: "Already implemented and verified",
});
const resolved = await runEvolutionLoop({
...project,
since: "2020-01-01",
});
expect(resolved.queue.find((item) => item.kind === "signal")?.state).toBe(
"resolved"
);
expect(
resolved.mutations.some((item) => item.type === "create-proposal")
).toBe(false);
expect(await listWritebacks(project)).toHaveLength(1);
expect(await listProposals(project)).toHaveLength(1);
});

it("records deterministic missing configuration once instead of retrying it", async () => {
const project = await makeProject();
await enableEvolutionLoop(project);
await rm(join(project.rootDir, "reconciliation.json"));
const report = await runEvolutionLoop(project);
expect(report.status).toBe("failed");
expect(report.attempts).toHaveLength(1);
expect(report.attempts[0]?.error).toContain(
"Reconciliation config not found"
);
});

it("covers signal, proposal, explicit apply, regression reopen, and verified improvement end to end", async () => {
const project = await makeProject();
const writeback = await addWriteback({
Expand Down
Loading
Loading