Skip to content

fix(tests): evict the whole git_service package family — 13 leaked failures in test_ent615 (0.9.5 M1) - #2861

Merged
obasilakis merged 2 commits into
devfrom
fix/m1-git-service-stub-leak
Sep 17, 2026
Merged

obasilakis merged 2 commits into
devfrom
fix/m1-git-service-stub-leak

Conversation

@dolho

@dolho dolho commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Fixes #2859

0.9.5 work order M1 — the red dev (backend-unit-test failing on 3a47f74).

Root cause

#2487 made services.git_service a package. Three test helpers still did sys.modules.pop("services.git_service") before re-importing it with stubbed deps — test_1704_git_service_plugins.py, test_data_paths_allowlist.py, test_data_paths_gitignore.py. Evicting only the parent leaves services.git_service.<sub> cached, and Python binds a cached submodule onto a new parent only on first load — so the re-imported package had no .token_scrub / .conflicts attribute, and every later git_service.<sub> read in test_ent615_token_free_remotes.py failed with AttributeError: module 'services.git_service' has no attribute 'token_scrub'. 13 failures whenever one of those files runs first (random order under pytest-randomly); the file passes alone.

Fix

Evict the whole services.git_service* family (the shape test_reset_preserve_state_guardrails.py already uses) at the three sites, and make every parent-only _restore_sys_modules fixture family-aware (1704 and both data_paths files — a parent restored without its children is the same half-state). Second commit: the fourth polluter from #2859, test_2075_detect_git_dir.py (swept the family but recorded only the package key for undo, so the submodules its own re-import created outlived teardown) — now the linter-sanctioned _STUBBED_MODULE_NAMES + _restore_sys_modules pair.

Regression guard (tests/unit/test_2859_git_service_eviction.py): the mechanism on a throwaway package (parent-only eviction loses the submodule attribute; family eviction rebinds it) + an AST scan of tests/unit that fails on sys.modules.pop("services.git_service") / del sys.modules["services.git_service"] / monkeypatch.delitem(sys.modules, "services.git_service") outside the family shape — self-tested on the exact lines the polluters carried, proven red by reintroducing the pop on a copy.

Verification

  • The issue's repro table (-p no:randomly, each polluter then ent615): 13 failed before on each; after → test_1704 102, test_data_paths_allowlist 101, test_data_paths_gitignore 93, test_2075 101 passed, 0 failed.
  • tests/lint_sys_modules.py: no new violations.
  • Full unit/ with the CI flags (-n auto --dist loadfile -p randomly, seed 12345), first commit: 16521 passed, 31 skipped, 0 failed (11 min). The 13 IDs are not added to tests/unit_failure_baseline.txt.

🤖 Generated with Claude Code

dolho and others added 2 commits September 17, 2026 12:54
…parent — 13 leaked failures in test_ent615 (0.9.5 M1)

#2487 made `services.git_service` a package. Three test helpers still did
`sys.modules.pop("services.git_service")` before re-importing it with stubbed deps.
Evicting only the parent leaves `services.git_service.<sub>` cached, and Python's
import re-binds a cached submodule onto a NEW parent only on first load — so the
re-imported package had no `.token_scrub` / `.conflicts` attribute, and every later
`git_service.<sub>` read in test_ent615_token_free_remotes.py failed with
`AttributeError: module 'services.git_service' has no attribute 'token_scrub'` (13
failures on dev; the file passes alone). Evict the family (the shape
test_reset_preserve_state_guardrails.py already uses) in test_1704_git_service_plugins,
test_data_paths_allowlist and test_data_paths_gitignore, and make 1704's autouse
restore family-aware for the same reason.

Reproduced: `pytest unit/test_data_paths_allowlist.py unit/test_ent615_token_free_remotes.py`
→ 13 failed; after: 118 passed in both orders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#2859 names four files; the first commit covered three. test_2075_detect_git_dir.py
swept the family but recorded only the package key for undo, so the submodules its
own re-import CREATED outlived monkeypatch's teardown — same half-state, same 13
failures. It now carries the lint_sys_modules.py-sanctioned pair
(_STUBBED_MODULE_NAMES + an autouse _restore_sys_modules) restoring the whole family;
the two data_paths files get the same family-aware restore as 1704 (their parent-only
restore fixture was a second leak behind the pop).

tests/unit/test_2859_git_service_eviction.py: (1) the mechanism on a throwaway
package — parent-only eviction loses the submodule attribute, family eviction rebinds
it; (2) an AST guard over tests/unit that fails on `sys.modules.pop("services.git_service")`,
`del sys.modules["services.git_service"]` or `monkeypatch.delitem(sys.modules,
"services.git_service")` outside the family shape (self-tested on the exact lines the
polluters carried; proven red by reintroducing the pop on a copy).

Issue's repro table (-p no:randomly, each file then ent615): 102 / 101 / 93 / 101 passed,
0 failed. lint_sys_modules.py: no new violations.

Fixes #2859

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dolho

dolho commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

/review Report

Branch: fix/m1-git-service-stub-leakdev (merge-base 3a47f74)
Files Changed: 5 (+219/−3), tests only
Scope: CLEAN — intent: #2859 (four polluters restore the whole services.git_service* family; a named regression test; no baseline entries). Delivered: exactly that. No product code touched.
Plan Completion (issue ACs): 3 done / 1 unverifiable

  • DONE — four files restore package + submodules: test_1704_git_service_plugins.py:52-66, test_data_paths_allowlist.py:56-70, test_data_paths_gitignore.py:53-67 (family snapshot + family-aware _restore_sys_modules), test_2075_detect_git_dir.py:36-59 (new _STUBBED_MODULE_NAMES + autouse family restore, teardown after monkeypatch's undo).
  • DONE — repro table passes: 102 / 101 / 93 / 101 passed, 0 failed (each polluter then ent615, -p no:randomly).
  • DONE — regression test named for the issue: test_2859_git_service_eviction.py — mechanism on a throwaway package + AST guard; red when the parent-only pop is reintroduced (proven on a copy).
  • UNVERIFIABLE from the diff — push-side pytest (push, absolute failures) green on dev: needs the merge. tests/unit_failure_baseline.txt untouched (0-line diff). Local full suite with CI flags, seed 12345: 16521 passed / 0 failed.

Critical Findings (block merge)

None.

Informational Findings (review required)

[I1] Incomplete-fix completeness (4.14): a fifth parent-only eviction lives outside the guard's reach (Confidence: 8/10)
File: tests/git_sync/test_s7_reserve_instance_id.py:107
Evidence: sys.modules.pop("services.git_service", None) at module level, followed by from services import git_service and from services.git_service import provisioning.
Issue: same pattern as the four fixed files. It is outside tests/unit/, so it is not in the backend-unit-test job and not scanned by the new guard (TESTS_UNIT.glob("test_*.py"), test_2859_git_service_eviction.py:118). Harmless today — that directory runs alone — but it is the exact line the guard exists to catch, and a future pytest tests/ invocation would leak it into whatever imports the package next.
Suggestion: follow-up (not this PR — #2859 scopes to tests/unit): family-evict there too, and consider widening the guard's glob to tests/**/test_*.py.

[I2] Test gap (4.9): the guard proves the static rule; the runtime half is proven only by the throwaway-package mechanism test (Confidence: 6/10)
File: tests/unit/test_2859_git_service_eviction.py:36-70
Issue: the mechanism test exercises CPython import semantics on pkg2859, not on services.git_service itself. If a future services/git_service/__init__.py stopped from .x import …-ing its submodules, the real package could regress in a way the throwaway package wouldn't show. The static guard still catches the cause (parent-only pop), so this is belt-and-braces.
Suggestion: optional — one assertion that, after a family eviction + import services.git_service, hasattr(gs, "token_scrub") and hasattr(gs, "conflicts") (the two attributes ent615 reads). Cheap; ties the guard to the real package.

[I3] Consistency (4.7): three copies of the same family-restore block (Confidence: 5/10 → appendix)
Files: test_1704:52-66, test_data_paths_allowlist:56-70, test_data_paths_gitignore:53-67 (+ a fourth variant in test_2075).
Issue: identical snapshot/restore code in four files. A conftest fixture would be one copy — but lint_sys_modules.py explicitly sanctions the per-file _STUBBED_MODULE_NAMES + _restore_sys_modules pair, so the duplication is the linter's own shape. Not worth churn in a red-dev fix.

Clean Categories

  • 4.1 SQL / 4.4 credentials / 4.10 frontend / 4.11 perf / 4.12 enums / 4.15 product bar — no product code in the diff.
  • 4.2 concurrency — the CI job runs -n auto --dist loadfile; sys.modules is per-worker-process, and loadfile keeps a file on one worker, so the family restore cannot race another file's import. Verified: workflow line -n auto --dist loadfile.
  • 4.3 auth — n/a.
  • 4.8 error handling — fixtures use try/finally; the AST guard raises on a parse error (fail-closed) rather than skipping a file.
  • 4.13 docs — no architecture/flow surface changed. A learnings.md entry for this class ("evicting a package from sys.modules must take its submodules") is warranted — not added on this branch per instruction; note for the author.
  • lint_sys_modules.py: no new violations (the 2075 pair and the guard's own tests use the sanctioned shapes).

Summary

  • Critical: 0 — none found
  • Informational: 2 (+1 appendix) — I1 is a follow-up outside this PR's stated scope; I2 optional
  • Scope: clean

Review complete. Next: /validate-pr 2861 for the docs/process pass; file the tests/git_sync sibling (I1) as a follow-up.

obasilakis added a commit that referenced this pull request Sep 17, 2026
The rc5 bump moves to its own chore PR, as #2838 did for rc4, so this fix
can merge on its own and the bump is not coupled to it. Review finding I1 on
#2862: the bump must not land ahead of #2861 while dev is red.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@obasilakis obasilakis mentioned this pull request Sep 17, 2026
3 tasks

@obasilakis obasilakis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at fc8480cce. No blocking findings.

  • Repro table on this head (-p no:randomly, each file then ent615): 1704 → 102 passed, allowlist → 101, gitignore → 93, 2075 → 101, 0 failed. The same 1704 → ent615 pair on dev fails 13, so the probe does detect the leak.
  • Guard bites: putting sys.modules.pop("services.git_service", None) back into test_data_paths_gitignore.py turns test_no_test_evicts_only_the_git_service_package red.
  • CI head jobs: 16524 passed, 0 failed on all three seeds.
  • Teardown order in test_2075 is right: the autouse fixture is set up before monkeypatch, so it tears down after monkeypatch's undo.

Non-blocking notes:

  • _ALLOWLIST exempts all of test_2075_detect_git_dir.py, so a future parent-only eviction added elsewhere in that file would pass the guard.
  • The guard only knows services.git_service, but the trap applies to any package. The one other parent-only pop of a package, services.agent_client in test_voice_tools.py:237, only runs when the key was absent before the test and a stub was installed, and test_voice_toolstest_circuit_breaker passes (103). So not a live leak today.

@obasilakis
obasilakis merged commit f31ed2d into dev Sep 17, 2026
25 of 26 checks passed
obasilakis added a commit that referenced this pull request Sep 17, 2026
Reverts the split in ea0649a. Agreed with the reviewer to keep the rc5 bump
here and merge in order instead: #2861 (green dev) first, then this PR, then
tag v0.9.5-rc5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
obasilakis added a commit that referenced this pull request Sep 17, 2026
…first boot serves again (#2862)

* fix(onboarding): the getting-started checklist updates without a reload

The store fetched the checklist once per page load, so a milestone reached
mid-session (creating an agent from the dashboard, or chatting, scheduling or
connecting a channel on another page) stayed unticked until a browser refresh.

Re-read on every mount, so returning to the dashboard picks up milestones
reached elsewhere, and whenever the agent count changes, since agents are
created and deleted from the dashboard itself.

Refs Abilityai/trinity-enterprise#238

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(deploy): make the rendered Caddyfile readable by the caddy user

The 1-Click first boot runs start.sh under `umask 077`. Since the Caddyfile
is rendered to Caddyfile.new and moved into place, the new file came out
0600 root, the `caddy` service user could not read it, the restart failed,
and provisioning stopped before Trinity started: a droplet created from the
v0.9.5-rc4 snapshot never served HTTPS. Overwriting in place had kept the
package's 0644, so rc3 booted.

chmod 0644 before the move (the file holds no secrets). The new test runs
provision_caddyfile under umask 077 and asserts the installed mode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: set version to 0.9.5-rc5

Bumps VERSION to 0.9.5-rc5 and points trinity-do-create.sh's default
TRINITY_IMAGE_TAG at v0.9.5-rc5 (tied to VERSION by
test_2380_installer_release_pin), so the v0.9.5-rc5 tag can be cut off dev
with the Caddyfile permission fix for the DigitalOcean Marketplace snapshot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(onboarding): don't re-ask an absent checklist surface on every forced read

The checklist now forces a read on every mount, sidebar expand and fleet-size
change. On OSS (404) and unentitled (403) builds that turned one swallowed
request per session into one per event, each with a console 404, for a card
that never renders. A 404/403 now marks the surface absent for the session;
a genuine failure (5xx) stays retryable.

Addresses review finding I1 on #2862.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "chore: set version to 0.9.5-rc5"

The rc5 bump moves to its own chore PR, as #2838 did for rc4, so this fix
can merge on its own and the bump is not coupled to it. Review finding I1 on
#2862: the bump must not land ahead of #2861 while dev is red.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: set version to 0.9.5-rc5 (restored)

Reverts the split in ea0649a. Agreed with the reviewer to keep the rc5 bump
here and merge in order instead: #2861 (green dev) first, then this PR, then
tag v0.9.5-rc5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge-train: validated individually against dev 3a47f74 (2-PR run, no train branch). Repro confirmed: each of the four polluters → test_ent615 goes 13 failed → 0 failed; random-order git_service slice clean on seeds 12345/67890/99999.

@vybe
vybe deleted the fix/m1-git-service-stub-leak branch September 17, 2026 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants