Skip to content

Isolate the test suite from real application data and the OS keyring - #62

Merged
sgf36 merged 1 commit into
mainfrom
test/conftest-data-isolation
Sep 13, 2026
Merged

sgf36 merged 1 commit into
mainfrom
test/conftest-data-isolation

Conversation

@sgf36

@sgf36 sgf36 commented Sep 11, 2026

Copy link
Copy Markdown
Owner

Problem

The pytest suite had no isolation from the developer's own application data. app/config.py resolves APP_DATA_DIR once, at import, from EASYPOST_DESKTOP_DATA_DIR or else platformdirs.user_data_dir, and nothing set the variable for pytest (no conftest.py, pytest.ini or pyproject setting). Several suites clear tables outright — tests/test_tracking_states.py runs DELETE FROM trackers, others clear shipments and similar — so python -m pytest tests/ on a machine that also runs the app wiped its local tracker and shipment records for both test and production modes. Importing app.core.client also built ClientManager at module level against the real OS keyring.

Fix

tests/conftest.py, which pytest imports before collecting any test module beside it:

  • points EASYPOST_DESKTOP_DATA_DIR at a per-session temporary directory, set unconditionally so a developer's exported override is not inherited;
  • installs an in-memory keyring backend (the _ScreenshotKeyring pattern from packaging/make_screenshots.py). It stubs the backend, not module attributes, because a name bound by from x import f is not reached by patching x.f;
  • imports app.config and refuses to start the session if its data directory resolved anywhere else — a path computed before the conftest ran cannot be moved, so a warning would not protect anything;
  • removes the temporary directory at the end of the session.

Proof by observation

tests/test_suite_isolation.py checks what the app itself does, not the variable conftest set:

Check Positive control (clean interpreter, no conftest)
PRAGMA database_list after init_db() names a file inside the session directory, not the real per-user one app.config.DATABASE_PATH is the real platformdirs path, outside the session directory
save_settings() writes settings.json inside the session directory (same control)
keyring.get_keyring() is the session backend, not a keyring.backends.* class; a credential saved through credential_store lands in it and is read back through app.core.client's by-name binding keyring.get_keyring() is a keyring.backends.* platform backend
A pytest run that imports app.config first exits non-zero with "Refusing to run the tests" The identical run without the early import passes

Results

  • Full suite locally (Windows, Python 3.14): 602 passed, 6 skipped. The latest green main CI run (bc53957) was 595 passed, 6 skipped on both legs; the difference is exactly the 7 new tests, and --collect-only gives 608 with and without the conftest, so no existing test was dropped.
  • SHA-256, size and modified time of every file in the real %LOCALAPPDATA%\EasyPostDesktop and %APPDATA%\EasyPostDesktop were recorded before the run and are identical after it.
  • No easypost-tests-* directory is left behind.

No application code changes.

🤖 Generated with Claude Code

Several suites clear tables outright (DELETE FROM trackers, shipments and
others), and app/config.py resolves APP_DATA_DIR at import from
EASYPOST_DESKTOP_DATA_DIR or else the real per-user data directory. Nothing
set the variable for pytest, so running the suite on a machine that also runs
the app wiped its local records in both modes. Importing app.core.client also
built ClientManager against the real OS keyring.

tests/conftest.py now, before any app module is imported, points the data
directory at a per-session temporary directory (unconditionally, so an
exported override is not inherited) and installs an in-memory keyring backend
(at the backend, since names bound by `from x import f` are not reached by
patching x.f). If app.config has already resolved anywhere else, the session
refuses to start.

tests/test_suite_isolation.py observes the file SQLite opened, the settings
file written and the backend a credential lands in, each with a positive
control in a clean interpreter, plus the refusal and its control.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sgf36
sgf36 merged commit b67ca59 into main Sep 13, 2026
4 checks passed
@sgf36
sgf36 deleted the test/conftest-data-isolation branch September 13, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant