Skip to content

fix(benchmark): read recording.db, the format the recorder writes - #19

Merged
abrichr merged 1 commit into
mainfrom
fix/benchmark-loader-recording-db
Aug 28, 2026
Merged

fix(benchmark): read recording.db, the format the recorder writes#19
abrichr merged 1 commit into
mainfrom
fix/benchmark-loader-recording-db

Conversation

@abrichr

@abrichr abrichr commented Aug 28, 2026

Copy link
Copy Markdown
Member

load_real_capture_data opened <recording>/capture.db and ran SELECT * FROM capture LIMIT 1, then read started_at, ended_at, screen_width and screen_height. openadapt-capture PR #28 replaced that format on 2026-07-17. The current recorder writes recording.db, and the two schemas share no table and no column:

pre-#28 current
capture.started_at recording.timestamp
capture.ended_at gone; derive from the newest event
capture.screen_width / screen_height recording.monitor_width / monitor_height
events action_event
screenshots/*.png screenshot rows holding png_data blobs

So the benchmark viewer raised FileNotFoundError on every recording anyone has made since. #17 fixed the same defect in the catalog scanner and left this file alone.

One reader, not two

src/openadapt_viewer/recording_db.py is new and owns the filename, the column names, the derived end time and the refusal to read a legacy capture. The scanner and the benchmark loader both read through it, so scanner.py loses about 90 lines and gains an import. A second translator living here would have put format knowledge in two places inside a repository that does not own the format.

Legacy support is absent on purpose. openadapt-capture ships scripts/migrate_legacy_capture.py and states that current code cannot load a legacy capture. A capture.db directory raises LegacyCaptureError carrying the conversion command, which is the same thing the scanner already prints.

The run's config gains frame_count, event_count and task_description, counted from the database. Those are what let a loaded run be checked against its source file rather than only against itself.

What the tests prove

tests/test_real_data_loader.py loads the two recordings openadapt-capture commits under examples/captures and asserts the values the loader reports, with every expectation queried here from the same file. CI already checks that repository out and sets $OPENADAPT_CAPTURE_EXAMPLES, so tests/capture_examples.py now holds the lookup that test_scanner_recording_db.py had inline.

Twenty of the twenty-one new tests fail on main. The one that passes asserts that an empty directory gains no recording.db, which was already true.

What the tests do not prove

The episodes half is not verified, and the test module says so in its own docstring rather than leaving a green run to imply otherwise.

load_real_capture_data also requires an episodes.json beside the recording. Neither committed recording carries one, and no producer writes the dialect this parser reads. The parser wants string steps, an episode_id like episode_001, and a screenshots.key_frames list. openadapt-ml's segmentation pipeline serialises EpisodeExtractionResult, whose steps are EpisodeStep objects, whose episode_id is a UUID, and which has no screenshots key at all. Writing a fixture to match the parser would only have proved that the parser agrees with whoever wrote the fixture, so the tests use test_episodes.json, which is already committed here, and assert against recording.db for everything that matters: start time, derived end time, duration, display geometry, platform, task description, frame count and event count.

That mismatch is a separate defect and is not fixed here. Guessing at a format with no sample in hand is how this one started.

Also

examples/capture.db is deleted. It is a zero-byte file, committed, named after the retired format, and read by nothing. sqlite3.connect creates a database at whatever path it is handed, which is how a file like that appears; both lookups in the new reader call is_file() before connecting.

`load_real_capture_data` opened `<recording>/capture.db` and ran
`SELECT * FROM capture LIMIT 1`, then read `started_at`, `ended_at`,
`screen_width` and `screen_height`. openadapt-capture PR #28 replaced that
format on 2026-07-17 with `recording.db`, a SQLAlchemy database whose
`recording` table shares no column with it:

  capture.started_at      -> recording.timestamp
  capture.ended_at        -> (gone; derive from the newest event)
  capture.screen_width    -> recording.monitor_width
  capture.screen_height   -> recording.monitor_height
  events                  -> action_event
  screenshots/*.png       -> screenshot rows holding png_data blobs

So the benchmark viewer raised FileNotFoundError on every recording the
current recorder has produced. #17 fixed this in the catalog scanner and
left this file alone, because the loader also requires an `episodes.json`
that no committed recording carries.

There is now one reader. `openadapt_viewer/recording_db.py` owns the
filename, the column names, the derived end time and the refusal to read a
legacy capture; the scanner and the benchmark loader both go through it.
Adding a second, divergent translator here would have put format knowledge
in two places in a repository that does not own the format. Legacy support
is deliberately absent for the same reason: openadapt-capture ships
`scripts/migrate_legacy_capture.py` and states that current code cannot load
a legacy capture, so a `capture.db` directory now raises LegacyCaptureError
carrying the conversion command, matching what the scanner prints.

The run's `config` gains `frame_count`, `event_count` and
`task_description`, counted from the database. They are what makes a loaded
run checkable against its source file rather than only against itself.

tests/test_real_data_loader.py loads the two recordings openadapt-capture
commits under examples/captures and asserts the values the loader reports,
queried from the same file. Twenty of its twenty-one tests fail without this
change.

It does not verify the episodes half, and says so in its own docstring. The
loader's `episodes.json` dialect has string `steps`, an `episode_id` like
`episode_001` and a `screenshots.key_frames` list. openadapt-ml's pipeline
serialises `EpisodeExtractionResult`, whose `steps` are objects, whose
`episode_id` is a UUID, and which has no `screenshots` key. No producer
writes the dialect this parser reads, so the fixture used is the one already
committed here, and every assertion that matters is made against
recording.db instead.

examples/capture.db goes with it: a zero-byte file, committed, named after
the retired format and read by nothing. sqlite3.connect creates a database
at any path it is handed, which is how a file like that appears. Both
lookups here use is_file() before connecting.
@github-actions

Copy link
Copy Markdown
Contributor

📸 Generated Screenshots Preview

Screenshots have been generated. Download the artifacts to preview them.

Generated Files:

  • demo_new_controls.png (0.10 MB)
  • demo_new_events.png (0.09 MB)
  • demo_new_full.png (0.12 MB)
  • turn-off-nightshift_controls.png (0.10 MB)
  • turn-off-nightshift_events.png (0.10 MB)
  • turn-off-nightshift_full.png (0.12 MB)

🔗 Download screenshots artifact

@abrichr
abrichr merged commit 8d70daf into main Aug 28, 2026
5 checks passed
@abrichr
abrichr deleted the fix/benchmark-loader-recording-db branch August 28, 2026 04:21
abrichr added a commit that referenced this pull request Aug 28, 2026
The first pass verified everything against openadapt-viewer==0.2.0 from PyPI
and documented two bugs that main has already fixed. #15 replaced the
hardcoded DEFAULT_CAPTURE_PATH with $OPENADAPT_CAPTURE_RECORDING, and #17 and
#19 moved the readers from the pre-#28 capture.db to recording.db. Reciting
the wheel's behaviour also tripped tests/test_no_hardcoded_paths.py, which
bans an absolute home path in any tracked file including documentation.

Everything is now re-run against an editable install of this branch: the demo
screenshot, the component signatures in docs/COMPONENTS.md, the benchmark
invocation, and the offline behaviour with cdn.jsdelivr.net aborted in a
headless browser.

The legacy-capture path replaces the stale hardcoded-path bullet, because it is
a live failure: LegacyCaptureError subclasses FileNotFoundError, so the
fallback in generate_benchmark_html catches it, load_benchmark_data returns a
run with zero tasks, and the CLI prints "Generated:". The migration command
that recording_db raises never reaches the person who needs it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
abrichr added a commit that referenced this pull request Aug 28, 2026
* docs: rewrite the README around one worked example

The old README opened with six bolded-lead feature bullets, then documented
capabilities this package does not have. The audio-transcript section described
a feature of openadapt-capture: the string "transcript" appears nowhere in
openadapt-viewer's source or in the 0.2.0 wheel. The four README screenshots
were produced by scripts/generate_readme_screenshots.py, which calls
openadapt_capture.visualize.html.create_html, so they showed another package's
output with captions pointing at a transcript panel that is not in the images.
The synthetic-demo section told the reader to open synthetic_demo_viewer.html
and linked SYNTHETIC_DEMOS_EXPLAINED.md; neither file exists in the repository.
SEARCH_FUNCTIONALITY.md was linked at the root and lives under docs/.

Everything in the new file was run against openadapt-viewer==0.2.0 installed
from PyPI into an empty venv, and the pasted output is that run's output. The
component signatures move to docs/COMPONENTS.md, read out of the installed
package with inspect.signature rather than copied from the source tree.

The screenshot is the real output of `openadapt-viewer demo`, regenerated by
scripts/generate_demo_screenshot.py.

Four behaviours that the old README's "works offline, no server required" claim
covered up are now written down: the page fetches Alpine from jsdelivr and the
task list does not render without it; `benchmark` with no --data resolves an
absolute path on one developer's machine; the capture viewer emits
repo-relative href/src for episode_timeline.css and .js; and __version__ still
reports 0.1.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: describe main, not the released 0.2.0

The first pass verified everything against openadapt-viewer==0.2.0 from PyPI
and documented two bugs that main has already fixed. #15 replaced the
hardcoded DEFAULT_CAPTURE_PATH with $OPENADAPT_CAPTURE_RECORDING, and #17 and
#19 moved the readers from the pre-#28 capture.db to recording.db. Reciting
the wheel's behaviour also tripped tests/test_no_hardcoded_paths.py, which
bans an absolute home path in any tracked file including documentation.

Everything is now re-run against an editable install of this branch: the demo
screenshot, the component signatures in docs/COMPONENTS.md, the benchmark
invocation, and the offline behaviour with cdn.jsdelivr.net aborted in a
headless browser.

The legacy-capture path replaces the stale hardcoded-path bullet, because it is
a live failure: LegacyCaptureError subclasses FileNotFoundError, so the
fallback in generate_benchmark_html catches it, load_benchmark_data returns a
run with zero tasks, and the CLI prints "Generated:". The migration command
that recording_db raises never reaches the person who needs it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: drop two capability claims the generator does not implement

The offline section said benchmark --standalone embeds Plotly. The flag is
dead: cli.py passes it to generate_benchmark_html, which passes it into
_generate_viewer_html, whose body never reads it. PageBuilder is built with
include_alpine=True and no include_plotly, so the Plotly branch never runs
and the two renderings are byte identical. Claiming an offline escape hatch
inside the section about not being offline-safe is the worst place for it.

dark_mode is stored in PageBuilder.__init__ and never read again. The sun
button in the header is what switches the palette. Say that instead.

Add the screenshot-path bullet: the benchmark viewer writes a real
recording's screenshots as absolute local paths, so mailing that file loses
the images. Only demo inlines them as data URIs.

The demo's pass and fail values come from an unseeded random.random(), so
90.0% is not impossible, only unlikely. Soften the caption.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant