Skip to content

Require session screenshots and make feedback evals reviewable - #6

Draft
AndreasInk wants to merge 4 commits into
mainfrom
codex/developer-actionable-feedback-prompts
Draft

AndreasInk wants to merge 4 commits into
mainfrom
codex/developer-actionable-feedback-prompts

Conversation

@AndreasInk

@AndreasInk AndreasInk commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Clarification on iOS/macOS 27 could run without the screenshot: the sheet was text-only, and notification analysis silently fell back after losing its image. This change captures the mounted app window before feedback UI or permission prompts, carries that image through the session, and preserves the tester response without model analysis when the image is unavailable. The existing screenshot provider remains an authoritative override; screenshots stay in memory.

The previous evaluation could reward asking on every image and relied on text-only judging. Replace it with 16 matched screenshot cases, baseline/candidate generation through the same image pipeline, image-based judge calibration, raw local artifacts, matched-case statistics, and blinded review plus a six-reply pilot template. The minimal prompt remains the Release/default prompt pending human acceptance. The Debug demo supports --betafeedback-evaluate-candidate and labels the selected variant onscreen and in report context; the candidate is not claimed to be better.

Validation:

  • Package Debug and Release builds, iPhone demo build, and macOS demo build passed.
  • Demo installed and launched on Andreas’s iPhone 16 Pro Max before opening this PR; updated again with the final candidate pilot launch option. The newest install succeeded; candidate-mode launch is awaiting iPhone unlock (CoreDevice returned Locked).
  • Swift tests: 71/72 passed. The pre-existing screenshotGuidanceFallsBackWithoutPromisingANotification test still expects outdated notification copy; its two assertions remain unchanged.
  • Seven evaluator runner regression tests passed, including child processes that create new sessions or outlive their parent.
  • Fresh local run completed all 40 jobs: eight calibration controls and 32 baseline/candidate generations and judgments, with all 16 case pairs matched, no timeouts and no missing artifacts. Both variants matched the ask/stop labels on 8/16 cases. Judge calibration failed: it gave all eight controls usefulness 3, failed to flag the explicit setup-code request or invented server rejection, and tied all four good/bad rankings. Judge scores cannot establish improvement. Raw screenshots, outputs, and tester data stay outside Git.

Keep this draft pending blinded human ratings and six real tester replies. The local run is from bd37f07; subsequent changes freeze the corpus for report refreshes and add an explicit Debug-only candidate launch argument. Both were verified separately; prompt strings, output schema, preprocessing, and explicit-variant eval paths are unchanged. Physical permission-denial, relaunch, and repeated-screenshot behavior still need hands-on verification; install/launch receipts do not establish those behaviors. macOS visual inspection was blocked by the locked Mac.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant