feat(cli): measure a composition against a reference with compare --against

[miguel-heygen]

What

Adds hyperframes compare --against <reference>: measure one composition against an external reference video or still, at up to 8 sampled times.

Each run emits three things:

• a reference-over-replica contact sheet (--out),
• a red/cyan deviation overlay per sampled time (<out>-overlay-NN.png): agreement grey, reference-only ink red, replica-only ink cyan,
• numbers per time: ssim, meanAbsDiff, meanSignedDiff, and ink bounding-box deltas dw / dh / dcx / dcy / scale.

--fail-under <ssim> turns it into a gate (exit 1 when the worst sampled SSIM is below the threshold). Existing multi-variant compare behaviour is unchanged.

Why

Every gate we ship is self-referential. lint and check audit a composition against its own rules, so a scene that is inverted, mis-scaled, or missing its hero element passes both under a green check. When the brief comes with a reference artifact (a video being rebuilt or recut, an approved cut, a design still, the previous accepted render), nothing in the CLI compares the build to it, and agents end up hand-rolling a one-off ffmpeg measuring rig, or eyeballing composites.

How

• packages/cli/src/utils/referenceDiff.ts: pure frame math (ink bounding box from a median-luma threshold, bounds deviation, mean absolute and mean signed difference, red/cyan overlay buffer, ffmpeg SSIM args + log parsing). No I/O, so it unit-tests without a browser or a built workspace.
• packages/cli/src/capture/compareAgainstReference.ts: orchestration. Reference frames via ffmpeg, replica frames from one browser session seeked with AUDIT_SEEK_OPTIONS (the producer bridge, the same seek target render drives), replicas normalized to reference dimensions so the sheet, the overlay and SSIM all read the same pixels.
• SSIM comes from ffmpeg's own ssim filter rather than a reimplementation. No new dependencies: sharp and ffmpeg are already required.
• No default threshold, deliberately. The floor depends on content (below), so a guessed threshold either passes everything or fails every video-backed build.
• Skills: /hyperframes-cli gains a "ground the gate in a reference" section, references/compare-and-batch.md gains the full contract with a read-the-numbers guide, /remotion-to-hyperframes points at it for mid-build iteration between full renders, and the /hyperframes router now says to read only the routed workflow's SKILL.md and none of its siblings.

The floor, measured rather than assumed

A replica is a live browser paint; a reference is a decoded compressed video. That decode gap is a floor no correction goes below, and it depends on content. Compared against their own renders:

Content	Self-comparison SSIM
Flat graphics and type	0.998 to 0.999
Photographic video layers	~0.93 at --quality high, ~0.89 at draft

meanSignedDiff exists to make that legible: on the video-backed case, 1.6 of a 2.1% mean absolute difference is uniform bias, not structural error. The CLI prints it as diff 2.1% (bias +1.6%).

An earlier revision of this branch documented the video-backed floor as seek drift. That was wrong and is corrected here: probing which reference frame the live seek lands on shows it lands exactly on the right one (SSIM peaks at frame 210 for t=7 and falls off on both sides, 0.881 / 0.893 / 0.888). The residual is encode loss plus browser-versus-FFmpeg colour conversion.

Test plan

Unit tests cover the pure math and the argument parsing (ink bounds, bounds deviation, mean absolute and signed difference, overlay channel mapping, SSIM log parsing, --at / --fail-under validation and caps).

End to end, against real renders:

• registry/examples/nyt-graph (graphics only) rendered at --quality high, compared against its own render at --at 0,3,7,11,14: SSIM 0.9987 to 0.9999, ink deltas all zero.

• Same project against an unrelated video: SSIM 0.2892, meanAbsDiff 62.5%.

• Gate exit codes verified directly: mismatch + --fail-under 0.9 exits 1, match + --fail-under 0.9 exits 0.

• registry/examples/kinetic-type (video-backed) against its own high-quality render: 0.9353 / 0.9316 / 0.9301 with bias +1.6% / +1.1% / +1.0%, and against the draft render 0.8901 / 0.8930 / 0.8955. Repeat runs are bit-identical, so the number is stable.

• Frame-landing probe (above) run against neighbouring reference frames 200 through 220.

• packages/cli suite: 2501 passed, 2 failed. Both failures are the pre-existing transcribe.test.ts cases that also fail on main.

• Unit tests added/updated

• Manual testing performed

• Documentation updated (if applicable)

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
hyperframes	🟢 Ready	View Preview	Aug 8, 2026, 1:35 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.