Skip to content

Repair the factory feedback comment (step 1) - #156

Merged
walker-tx merged 2 commits into
mainfrom
walker/interactive-unconverged-feedback
Aug 11, 2026
Merged

Repair the factory feedback comment (step 1)#156
walker-tx merged 2 commits into
mainfrom
walker/interactive-unconverged-feedback

Conversation

@walker-tx

Copy link
Copy Markdown
Member

Implements section 5 of docs/feedback-threads.md, and records the section 6 spike result. Step 2 (review threads) stays unbuilt.

The problem

Three defects made the factory comment worse than the work behind it.

Defect Evidence
The scope check printed more questions than the run found The hubspot comment printed 10 questions where 5 exist
Every open-question bullet lost its body The extractor truncated 53 of 53 bullets in this repository at the first line break
The factory asked questions the run had already refuted The snowflake run reported external.md is missing after the same run stated external.md exists

The change

  • text-similarity.ts (new). It decides when two questions say the same thing. The threshold is 0.6. The measured band is narrow: the highest score between two genuinely different questions is 0.5882, and the lowest score between a true duplicate pair is 0.6000. A test pins both sides, so a silent retune fails the suite.
  • scope-gate.ts keeps the whole bullet. It joins each continuation line.
  • mergeOpenQuestions merges a dossier bullet into the report question that already says it. The report text always wins.
  • findings.ts drops a lint finding that a review round refuted. The rule is narrow on purpose: a <file>.md is missing. problem, and only when a disputed line claims that same file exists.
  • format-scope-check.ts suppresses duplicate soft questions.
  • format-pipeline-review.ts collapses the nits into a <details> element. The old threshold never fired: the highest nit count in all 60 run records is 9, and the branch needs more than 12.
  • format-summary.ts (new) posts a short issue comment with a link. It measures 446 characters against 3800 for the full comment.

Results

Definition of done Result
The hubspot record produces 5 questions 10 → 5
The snowflake record drops the refuted findings 5 unresolved → 3 kept
No open-question bullet ends mid-sentence Bullet lengths go from 69/68/69/68/69 to 572/531/279/414/351
The issue comment holds a summary and a link Done
The spike result is in section 6 Done

Tests go from 112 to 157. tsc --noEmit is clean.

Three deviations from the design document

Each one is measured, and each is recorded in disagreements_with_design terms in the commit message.

  1. 5.2 asks for a prefix rule. No prefix pair exists in any soft or open_questions array across all 60 run records, so the rule takes 10 questions in and returns 10. Containment overlap replaces it.
  2. 5.2 also says "keep the longer text". That would delete the pause. isMaterialOpenQuestion fires on the short report paraphrase and not on the longer dossier bullet. The merge keeps the report text instead.
  3. 5.4 asks to extend partitionFindings to the scope path. That function keys on target plus where, and the scope path carries bare strings.
  4. 5.5 asks to shrink both long copies. cmd-pr.ts:101 passes an empty prUrl, because the pull request does not exist yet at that moment. The pull request body stays the canonical copy. Only the issue comment shrinks.

The document also stated the hubspot baseline as 9 questions. The real baseline is 10: 1 decision plus 9 soft.

The spike

A file-level review thread does not go Outdated after a commit rewrites the file. A line-level control thread on the same file does, which proves the test worked. Section 7.3 needs no change.

The spike ran on a throwaway pull request, not on #143. guide/issue-142-hubspot is the live resume source for issue #142, so a spike commit would change what the next factory run reads. Every spike artifact is cleaned up: pulls/comments returns [], #143 sits at the same sha with 0 review comments, and the throwaway branch is deleted.

Section 6 records the required field set, the verbatim 422 error text, the identifiers the API returned, and one correction: REST position does not become null on an outdated thread. The field that changes is line.

Two items for later, not fixed here

  • A defect the design document does not record. Three findings carry severity: "blocker" with no dimension, so isGate demotes them to "Optional nits". They are in the two google-big-query records. Collapsing the nits hides them one click deeper. A test pins the current behaviour, so a later fix shows up as a test change.
  • The salesforce record cannot be repaired retroactively. The old extractor truncated its entries so far that the highest score between any two is 0.4286. Section 5.1 fixes it for future runs only.

🤖 Generated with Claude Code

walker-tx and others added 2 commits August 11, 2026 14:35
Implement section 5 of docs/feedback-threads.md, and record the section 6
spike result. Step 2 (review threads) stays unbuilt.

The hubspot scope check printed 10 questions where 5 exist. The extractor
truncated all 53 open-question bullets in the repository at the first line
break. The snowflake pipeline review asked 2 questions that the same run had
already refuted.

Changes:

- Add text-similarity.ts. It decides when two questions say the same thing.
  The threshold is 0.6. The measured band is narrow: the highest score
  between two different questions is 0.5882, and the lowest score between a
  true duplicate pair is 0.6000. A test pins both sides.
- scope-gate.ts keeps the whole bullet. It joins each continuation line.
- mergeOpenQuestions merges a dossier bullet into the report question that
  already says it. The report text always wins, because isMaterialOpenQuestion
  fires on the report paraphrase and not on the longer dossier bullet.
- findings.ts drops a lint finding that a review round refuted.
- format-scope-check.ts suppresses duplicate soft questions.
- format-pipeline-review.ts collapses the nits into a details element.
- format-summary.ts posts a short issue comment with a link.

Three deviations from the design document, each with evidence:

- 5.2 asks for a prefix rule. No prefix pair exists in any of the 60 run
  records, so the rule removes nothing. Containment overlap replaces it.
- 5.4 asks to extend partitionFindings to the scope path. That function keys
  on target plus where, and the scope path carries bare strings.
- 5.5 asks to shrink both long copies. cmd-pr.ts passes an empty prUrl,
  because the pull request does not exist yet at that moment. The pull request
  body stays the canonical copy. Only the issue comment shrinks.

The spike answer: a file-level review thread does not go Outdated after a
commit rewrites the file. A line-level control thread on the same file does.
Section 7.3 needs no change.

Tests go from 112 to 157. Every fixture is a copy under pipeline/src/,
because pipeline-ci.yml triggers only on the pipeline path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@walker-tx
walker-tx merged commit 972c872 into main Aug 11, 2026
1 check passed
@walker-tx
walker-tx deleted the walker/interactive-unconverged-feedback branch August 11, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant