Require session screenshots and make feedback evals reviewable - #6
Draft
AndreasInk wants to merge 4 commits into
Draft
AndreasInk wants to merge 4 commits into
AndreasInk wants to merge 4 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clarification on iOS/macOS 27 could run without the screenshot: the sheet was text-only, and notification analysis silently fell back after losing its image. This change captures the mounted app window before feedback UI or permission prompts, carries that image through the session, and preserves the tester response without model analysis when the image is unavailable. The existing screenshot provider remains an authoritative override; screenshots stay in memory.
The previous evaluation could reward asking on every image and relied on text-only judging. Replace it with 16 matched screenshot cases, baseline/candidate generation through the same image pipeline, image-based judge calibration, raw local artifacts, matched-case statistics, and blinded review plus a six-reply pilot template. The minimal prompt remains the Release/default prompt pending human acceptance. The Debug demo supports
--betafeedback-evaluate-candidateand labels the selected variant onscreen and in report context; the candidate is not claimed to be better.Validation:
screenshotGuidanceFallsBackWithoutPromisingANotificationtest still expects outdated notification copy; its two assertions remain unchanged.Keep this draft pending blinded human ratings and six real tester replies. The local run is from bd37f07; subsequent changes freeze the corpus for report refreshes and add an explicit Debug-only candidate launch argument. Both were verified separately; prompt strings, output schema, preprocessing, and explicit-variant eval paths are unchanged. Physical permission-denial, relaunch, and repeated-screenshot behavior still need hands-on verification; install/launch receipts do not establish those behaviors. macOS visual inspection was blocked by the locked Mac.