Repository navigation
Critique-and-revise pass on cover letters, with a quality eval - #25
Merged
Merged
Conversation
A critique pass reviews a tailored document against a recruiter's rubric for the ad and rewrites only what falls short. For letters the rubric is the swap test for 'why them', evidence mapped to the ad's requirements, the opening, the ending and the language. For CVs it is relevance, evidence, summary, projects and bullet order. A rewrite is kept only if the document check finds no new unsupported claim in it and it is at most 15% longer, so the page fit holds. The review (issues found, kept or not) goes out with the document. DOC_CRITIQUE picks the documents it runs on (cv, letter); letters only by default. evals/test_doc_quality_eval.py measures it: the graduate profile is tailored to six adverts with the real model, the critique runs on each draft, and a judge compares draft and revision blind in both orders. Over three runs the critiqued letter beat its draft every time (4-2, 5-1, 2-1 with 3 ties); CVs were mixed (1-3, 4-0, 2-0), so the CV critique stays off. The eval fails if a kept rewrite adds an unsupported claim or if letters stop winning. Tests turn the critique off by default (one more model call per document) and cover it explicitly: a rewrite that holds up is kept, one that invents a claim or runs long is dropped, and no issues leaves the document alone.
When the critique's rewrite was kept, the Checks strip says how many things the review fixed for this ad, and its details list them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2, part 2: a critique-and-revise pass on tailored documents, measured by a new quality eval. On for cover letters only, by default.
What it does
After a document is written, the model reviews it against a recruiter's rubric for the ad and rewrites only what falls short.
A rewrite is kept only if the document check finds no new unsupported claim in it and it is at most 15% longer, so the page fit holds. The result carries
review: {issues, kept}, and the job workspace's Checks strip shows "Reviewed for this ad: N fixes" with the list in its details.DOC_CRITIQUEpicks the documents it runs on (cv,letter, comma-separated); the default isletter.Why letters only
api/evals/test_doc_quality_eval.py(opt-in,WINDRUSH_LLM_EVAL=1) tailors the graduate profile to six adverts with the real model, runs the critique on each draft, and has a judge compare draft and revision blind, in both orders.No kept rewrite added an unsupported claim. Letters improved every run; CV rewrites were mostly reorderings and once dropped a real result, so the CV critique stays off. The eval fails if a kept rewrite adds an unsupported claim or if letters stop beating their drafts.
Tests