Skip to content

Derive installer file behavior from declared owners - #2765

Merged
trissim merged 16 commits into
mainfrom
fix/installer-declared-file-behavior
Oct 23, 2026
Merged

trissim merged 16 commits into
mainfrom
fix/installer-declared-file-behavior

Conversation

@trissim

@trissim trissim commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The installer classified undeclared files by opening each as SQLite. The 150,195-file frozen copy contained 119,821 undeclared resources, so that guess dominated the full-root report. File behavior now comes from declared owners; unknown files receive RootFile byte behavior. The whole-root changed-file check is unchanged.

content_owner and SQLiteFile.is_database are deleted. Backup capture, digest comparison, tree copying and restore pass owner mappings through their operations. CoordinationStore.observed still validates its explicitly declared coordinator through a read-only SQLite snapshot; it no longer asks a generic file-type classifier. Backup JSON formats are unchanged.

Whole-root measurement

Same frozen copy of the live root, nice 19, before applying the retirement carry:

Before (50af024) After (22db4da)
bus_files seconds 39.921 9.336
regular files 150,195 150,195
reported resources 130,153 130,153
undeclared resources 119,821 118,277

Both runs produced the same sorted-path SHA256: 772e8a02e8a8b27d4a456ce83f04892617a62c20b718b91d99153201f473fa82. No files were deleted or skipped to obtain that timing. Data and results are under /home/ts/.cache/agent-scratch/install20-scoped-measurement/. These numbers describe the recorded comparison, not an install-23 wall-time claim. The current head merges origin/main at 791e5a2. The timing above predates the review fixes below.

The real agent_comms.installer.step_worker applied retirement under maintenance on a separate selected-input copy. StoreBackup.restore restored every selected resource, including SQLite WAL data, publisher signals and the registry pair. That earlier proof selected 69 resources; subsequent source-history reading established that the PR48 marker must be retained, so it was removed from deletion. That saved run predates this review correction. This correction used only the requested Ruff and Black commands; no tests or live-root operations were run.

Nine SQLite verdicts

The retirement implementation is tools/cutover/carry_retired_stores.py:12–31. Its declarations live beside it in tools/cutover/retired_store_files.py and are deleted with the carry; nothing in src imports them. These files have no writer, so a backup restores them as plain bytes; published stores retire their change signals through the same owner (retired_store_files.py:29–40).

File Verdict and writer/replacement evidence Retirement declaration
todos.sqlite3.before-allocation-1792621659 Unreferenced allocation checkpoint. Exact-name and before-allocation git searches across src/tools/all refs found no writer. Current TodoStore owns todos.sqlite3; the carry removes the snapshot. tools/cutover/retired_store_files.py:43–44
transcript_reply_index.v4.sqlite3 Superseded reply cache. 9e58b1cf5^:src/agent_comms/view_unread.py:40,495 declares version 4 and builds its name. Current version is 7 at view_unread.py:44,404. tools/cutover/retired_store_files.py:47–48
wake_candidate_index.da2261cd24bb.sqlite3 Superseded candidate cache. dff45f4 derives the stem from WakeCandidateIndex; e9a0fb3e3:typed_table.py:1301 derives the schema hash. 6a41c2f replaces this store with bus recipient-index reads; current wake_candidate_index.py:49–58 owns that view. tools/cutover/retired_store_files.py:51–52
bus_page_index.0f89d7bc6f52.sqlite3 Superseded JSONL page cache. e9a0fb3 changes the fixed filename to a schema-derived name; dff45f4 derives the stem. fee622f deletes the producer. tools/cutover/retired_store_files.py:55–56
bus_page_index.sqlite3 The preceding fixed-name version: e9a0fb3e3^:bus_page_index.py:114. Same removed producer. tools/cutover/retired_store_files.py:55–56
bus_route_counts.v6.sqlite3 Superseded route cache. 563a9df9e^:src/agent_comms/bus_route_counts.py:343–344 writes the versioned file; 563a9df moves the projections into bus transactions. tools/cutover/retired_store_files.py:59–60
wake_candidates.sqlite3 Fixed-name predecessor. e9a0fb3 changes WakeCandidateIndex.filename into a derived stem; 6a41c2f removes the separate store. tools/cutover/retired_store_files.py:51–52
private_bus_checkpoint.sqlite3 Retired wire/checkpoint store. fee622f56^:src/agent_comms/private_bus_checkpoint.py:1051–1054 declares WritableCheckpoint; fee622f deletes it after consumers move to the SQLite bus. tools/cutover/retired_store_files.py:63–64
owner_release_receipts.json/owner_releases.sqlite3 Obsolete nested placement. 90b6dfb changes OwnerReleaseStore from a JSON-path constructor to a root constructor. Current owner_lifecycle.py:240–242,281 uses the root-level owner_releases.sqlite3; no current reader names the nested path. tools/cutover/retired_store_files.py:67–69

Exact git log -S<full filename> -- src found no literal writers for the generated hash/version names or the nested path; the table identifies their generating declarations instead of pretending those literal searches found them.

Other requested directories

  • turns/: the reported undeclared examples are orphaned .json.lock names. StoredTurnRecord already declares their filename family (turn_lease.py:535), and PatternFile.files_in already declares orphan store-lock names. RootFile.owned_files now takes member identity from each owner's files_in, including absent-primary companions (locked_store.py:167–169). No turn history is deleted.
  • diagnostics/: OwnerProcessOutput owns the existing startup writer and its filename at diagnostics.py:32–52. OwnerLifecycle uses that owner directly.
  • sessions/ and imported_sessions/: SavedSessionFile composes the NativeSessionLocks writer fence into member paths, transient paths and orphan-lock enumeration (native_session_files.py:225–236). Session bytes are retained. Remaining selected-tool-ledger entries belong to Move selected tool consumption into the input coordination store #2764's migration.
  • .native-pi-agent/: NativeAgentSettings declares the settings file and prepares the directory used by NativePi.build (native_session_files.py:263–273; native_pi.py's prepare_directory call). Settings remain byte files; no auth or history is deleted.
  • PR48 uncertainty marker: explicitly retained by RetainedNativeWriteMarker at native_session_files.py:276–279. The removed writer deliberately kept <session>.pr48-writer.lock on uncertainty (8f9fd69c9^:stack/native-writer-policy.md:43 and native-session-storage.patch). It is not a retired-store member and is not deleted by this carry.

The measured tree still had six undeclared files outside selected-tool-ledger after classification: two pi-web-search extension assets and four old activity/display files. This PR leaves them visible to the full-root report rather than guessing their type or silently removing them.

Source evidence and assumptions

AST inventory covered 949 Python modules across Core src/tools/tests and paired Toad src, with zero parse omissions. It found the coordinator validation caller in addition to the installer callers. Static names do not prove resolution of arbitrary runtime rebinding.

Assumed:

  • The allocation checkpoint is unreferenced retained backup data; its original creator was not found. It is snapshotted before retirement, never hand-removed.
  • Retired stores have no concurrent writer. Their carry-owned declarations select each database, WAL, SHM, journal and publisher signal as separate byte artifacts. The carry counts primary stores; backup capture and restore use byte behavior regardless of whether the carry module is imported. Live declared SQLite stores retain their existing snapshot behavior.
  • The timing comparison covers the frozen live-root copy and the recorded heads. The commander measures actual install latency in install 23; under 30 seconds is not claimed here.
  • The frozen root has no retained archive catalogue; this proof does not exercise multi-root archive migration. Unknown paths remain byte resources by the requested contract.

Review fixes

The declarations moved from src to tools/cutover/retired_store_files.py:1; its module says to delete it with the carry. The snapshot copy_group override is deleted. Companion names derive from SQLiteFile.files and PublishedSQLiteFile._source_paths, not a second suffix list. primaries_in keeps the carry's changed-store count separate from the physical files selected for backup.

Both backup tests now use ExternalTranscript.path_in(root) instead of declaring temporary SQLite subclasses. Removed the coordinator's unused import and formatted the three requested files.

Commands on exactly coordination_database.py, tools/cutover/retired_store_files.py and tools/cutover/carry_retired_stores.py:

  • ruff check --fix: All checks passed!
  • black: 3 files left unchanged.
  • ruff check: All checks passed!
  • black --check: 3 files would be left unchanged.

The published branch's concurrent move/main merge was merged without overwriting it; the final source keeps byte behavior for unwritten retired artifacts.

@trissim
trissim marked this pull request as ready for review October 23, 2026 12:36
Copilot AI balanced review requested due to automatic review settings October 23, 2026 12:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions

Copy link
Copy Markdown

debt-tools at 2bad394, every finding:

grew Unformatted:tools/cutover/carry_retired_stores.py +1

@trissim

trissim commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Handoff at 4a9bfeb.

Done: content_owner and SQLiteFile.is_database are removed; RootFile.owned_files gives undeclared files byte behavior (src/agent_comms/locked_store.py:157-188). Backup copy/digest/restore thread owner mappings through (src/agent_comms/installer/backup.py:260-410). Startup logs, native settings, session writer locks and retained PR48 uncertainty evidence have declarations (diagnostics.py:32-52; native_session_files.py:220-279). Nine obsolete SQLite stores and their signals are declared in retired_store_files.py:12-65 and retired by tools/cutover/carry_retired_stores.py:11-26. PR body has the per-file verdicts and measured same-tree walk improvement: 39.921 to 9.336 seconds, identical 150,195-file path hash. Main through 7017468 is merged. No tests run for this order; review belongs to the commander fork.

Left: exact-head review and install-23 qualification. Preserve the PR48 marker: it records uncertain native work and must not be deleted. The timed copy had no archive catalogue; multi-root archive ownership propagation is not qualified by that proof. Retired declarations must remain available while retained backups need their SQLite behavior.

Exact next edit: inspect the review verdict first and apply its specific source correction; if no correction is requested, make no further source edit and hand this pushed head to the commander for install 23. Do not launch a competing installer or test run. InstallSpeedPlan is also pending the commander's evidence shape: PhaseEvidence currently has no timing reader; InstallCommands writes timing records at installer/commands.py:44-46, serialized into result.json through installer/main.py:27 and installer/launch.py:142-147. Do not replace timing evidence with a grep.

@github-actions

Copy link
Copy Markdown

debt-src at 2bad394, every finding:

grew LintFindings:src/agent_comms/coordination_database.py +2
grew LintFindings:src/agent_comms/retired_store_files.py +3
grew Unformatted:src/agent_comms/retired_store_files.py +1

Ratchet test and others added 7 commits October 23, 2026 08:46
The eight retired store classes had no writer or reader in src, so they move
to tools/cutover/ beside carry_retired_stores.py and are deleted with it.
Nothing restores them by declaration: with no writer, a plain byte copy
restores them exactly, so the published-source copy_group goes too.

Drop the SQLiteFile import orphaned by deleting is_database, wrap the long
coordinator snapshot line, and make the backup tests copy the declared
ExternalTranscript store instead of declaring a throwaway SQLiteFile that
RootFile.declarations_in would see in every later test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trissim
trissim merged commit 7a5939d into main Oct 23, 2026
6 of 8 checks passed
trissim pushed a commit that referenced this pull request Oct 23, 2026
Keeps main's RegistryStore.digest and the GivenThreadText title in
test_interrupted_rollback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants