Skip to content

fix(fixtures): bind synthetic provenance to the fixture byte determinants - #107

Merged
abrichr merged 2 commits into
mainfrom
codex/synthetic-fixture-builder-determinants
Aug 27, 2026
Merged

fix(fixtures): bind synthetic provenance to the fixture byte determinants#107
abrichr merged 2 commits into
mainfrom
codex/synthetic-fixture-builder-determinants

Conversation

@abrichr

@abrichr abrichr commented Aug 27, 2026

Copy link
Copy Markdown
Member

What this changes

synthetic-provenance.json recorded lock_path and lock_sha256, a sha256 of the entire uv.lock, and tests/test_synthetic_fixtures.py asserted it. This drops both fields and bumps the provenance schema to v2.

Why

The lock hash does not determine the fixture bytes. The generator reads four things: the source spec, its own source, the repository's writer code, and the SQLite and SQLAlchemy versions. All four are already pinned precisely. The spec and the generator are recorded by sha256 and asserted against the files. The two library versions are recorded by value, and require_exact_builder() raises if the running interpreter does not match them.

Worse, the coupling was self-referential. The lock hash went into the provenance, the provenance is embedded in the recording config, and the config is inside the sealed database. So a lock rewrite changed the sealed binary fixtures by definition, whether or not it touched anything the generator reads.

That is not hypothetical. #96 adds a linux extra holding a single platform-gated PyGObject dependency. uv sync re-locks, and the result:

  • adds two Linux-only wheels, pygobject and pycairo
  • migrates the lock file format from version = 1 to revision = 3, about 2,195 lines changed
  • changes zero resolved versions of anything already locked, SQLAlchemy included

I checked the causality directly rather than reasoning about it. With the environment resolved and installed from the new lock, and the recorded hash held at the old value, all five generated artifacts in both fixtures regenerate byte for byte identical. The lock's contents contributed nothing. Only the recorded hash did.

Every dependency PR from here on would have had to rewrite two sealed binary fixtures for no reason. Note also that .github/workflows/test.yml pins astral-sh/setup-uv@v7 at version: "latest", so the lock format tracks whatever uv ships that week. The fixture seal should not.

What still catches a real change

check_generated() is untouched and stays the authority. It regenerates all five artifacts into a temp directory and compares every byte. It runs on the 3.12 lane, where the SQLite and SQLAlchemy versions match, so it does the real comparison there rather than the guard-fires check.

A dependency change that genuinely moves the bytes fails that comparison with generated bytes are stale. A dependency change that does not move the bytes now passes, which is the correct outcome and the point of the change.

Two tests were added so this is asserted rather than assumed:

  • test_a_changed_source_spec_makes_the_committed_fixtures_stale moves one action coordinate by a single pixel in a copy of a source spec and requires the byte check to fail.
  • test_every_committed_fixture_artifact_is_byte_checked flips one byte in each of the five generated artifacts in turn and requires the byte check to fail each time.

Both skip on a builder that cannot reproduce the committed bytes.

Verification

  • pytest tests/ --ignore=tests/test_browser_bridge.py: 611 passed, 13 skipped, locally on Python 3.12.13, SQLite 3.53.1, SQLAlchemy 2.0.52.
  • ruff check openadapt_capture/: clean.
  • No consumer of the v1 provenance schema exists anywhere else in the workspace.

Fixtures were regenerated with the committed generator, not hand-edited.

abrichr and others added 2 commits August 27, 2026 18:10
…ants

The synthetic fixture provenance recorded a sha256 of the whole uv.lock, and
tests/test_synthetic_fixtures.py asserted it. That hash does not determine the
fixture bytes. The generator reads the source spec, its own source, the
repository's writer code, and SQLite plus SQLAlchemy. Every one of those is
recorded already: the spec and generator by sha256, the two library versions by
value, with require_exact_builder() refusing any other writer.

Because the lock hash was written into the provenance, and the provenance is
embedded in the recording config, the hash reached the sealed database bytes.
That made it self-referential. Any lock rewrite forced a rewrite of two sealed
binary fixtures, whether or not the rewrite touched anything the generator
reads. Adding a platform-gated optional dependency did exactly that: it added
two Linux-only wheels and migrated the lock file format, changed no resolved
version of anything, and still broke the fixture suite.

Drop lock_path and lock_sha256, and bump the provenance schema to v2 so a
reader sees the narrower shape. The byte oracle is unchanged and remains the
authority: check_generated() regenerates all five artifacts and compares them
byte for byte, so a dependency change that really does move the bytes still
fails.

Two tests now prove that oracle bites:

- a one-pixel edit to a source spec makes the committed fixtures stale
- one flipped byte in any of the five generated artifacts is caught

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two new tamper tests assert that check_generated() raises 'generated
bytes are stale'. On a builder that does not reproduce the committed bytes
exactly, that message could report the builder instead of the tamper, and the
assertion would hold for the wrong reason.

Python 3.12 with the required SQLite and SQLAlchemy is the environment that
test_public_fixtures_match_the_source_generator_byte_for_byte proves
reproduces every committed byte, so it supplies the control. Restrict both
tamper tests to it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@abrichr
abrichr merged commit d00afdd into main Aug 27, 2026
12 checks passed
@abrichr
abrichr deleted the codex/synthetic-fixture-builder-determinants branch August 27, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant