Repository navigation
Harden Python lifecycle and bound replication metadata - #108
Merged
Merged
Conversation
forhappy
marked this pull request as ready for review
October 24, 2026 06:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python task/object saturation previously surfaced as generic handler failures, and replication coverage metadata accumulated across historical activation epochs. This preserves typed overload responses, bounds the coverage cache, and removes per-invocation Python dispatch closures while adding durable-write and native/Python read qualification. Cleanup exceptions now remain failures even when their Python truth value is false. Startup confirmation also retires safely during shutdown and fencing, while caller cancellation preserves node-owned confirmation work.
Changes:
None. An exception class with__bool__returning false previously allowed failed context cleanup to disappear from the shutdown report. Keep the failed invocation registered for the final cleanup pass.ReplicationOptions.coverage_cache_entriesandCELLAR_LTX_COVERAGE_CACHE_ENTRIES. Zero disables caching. Evicted entries use the durable-store lookup path; failed lookups are not cached as successful zero coverage. Repeated updates preserve the maximum watermark without growing eviction metadata.nested,initial, anddirect-initialread paths; reject direct snapshot measurements that submit SDK operations.Validation:
Snapshot failure diagnostics pass the full local snapshot suite (44 passed, two intentional ignores), strict Clippy, formatting, layout, documentation, dependency-policy and diff checks. A temporary expired-query injection exercised the failure-reporting path; it does not attribute or resolve the earlier
VersionUnavailableincident.The free-threaded lifetime failure, at
c72b1c78, exposed a native confirmation marker surviving transition toInactive. Four deterministic core regressions failed before the fix and now pass, covering preserve, caller cancellation, fencing and late inventory. All 34 ordinary core tests and 18 real embedded lifecycle/fencing cases pass locally across both ABIs, including five lifetime repetitions per ABI. Three native lifecycle/recovery tests, strict core Clippy, formatting, layout, documentation, dependency-policy and diff checks also pass.The prior-head SDK run, at
64f566e1, failednative_snapshot_catchup_resumes_follower_reads: the restored follower query returnedVersionUnavailable. Its cause remains unresolved. A separate Linux diagnostic run passed all 44 snapshot tests (two intentional ignores) and 200 focused repetitions with failure tracing; those passing repetitions do not close the original failure. Final-head CI must be evaluated separately.Reproduced 19,488 historical bundle-epoch markers after 20,000 epochs with only 512 live bundles. Five owning-module regressions now pass, covering bounded historical retention, complete GC, shared epoch membership, arrival ordering, and real persisted-row reload/corruption refusal. All three bundle-scan/recovery memory tests, 26 native log tests, six replication tests, strict Clippy and repository checks pass locally.
The cleanup regression reproduces
runtime_failed: falsebefore the fix. Eager and suspended-handler failure regressions now require a failed runtime report and complete resource cleanup. All 24 runner tests pass on CPython 3.14.5 and 3.14.5t, as do real embedded-node lifetime, global-close, cancellation, cleanup-error and full-mailbox cleanup cases on both ABIs.Reproduced the full-SDK metrics failure locally, corrected its stale expected inventory, then passed the focused regression and all 23 native Raft operations tests. The assertions retain exact cardinality and add archive metric identity checks.
The dispatch change passes all 22 runner tests and real embedded-node contract, delivery and schedule cases on CPython 3.14.5 and 3.14.5t.
Independent verification of the Linux dispatch comparison confirms matched native/baseline/candidate controls, raw scheduled outcomes, CPU placement, binary/source identity and all 18 joined hosts. Saturated nested-read throughput improved approximately 2–3% on the standard runner; the free-threaded result ranged from near unchanged to 1.4% better. Local alternating comparisons also passed all response and shutdown checks. These counter-read measurements do not establish native parity or production capacity.
Four focused cache tests pass, including fixed-population epoch churn, bounded eviction metadata, disabled caching, and real-store eviction/outage/retry with restored SQLite contents. The churn-bound regression fails against the previous unbounded implementation.
All 26 native log tests and six native replication tests pass. Three baseline harness tests pass. Strict Clippy for the affected library/integration targets, Rust formatting, module layout, documentation, dependency-policy and diff checks pass.
A real native-host churn run on the cache fix acknowledged 8,192 writes across 128 cells and 64 activation rounds. After killing the owner and deleting its local disk, the replacement recovered all 128 counters, acknowledged another 128 writes, and completed shutdown. This development-profile run validates recovery, not throughput or total memory stability.
The complete Raft multigroup suite and 12 repeated repair runs pass locally and in the Linux fixture qualification; the suite retains its one intentionally ignored test.
Earlier changes passed 17 compiled native-client tests, 22 runner tests on both Python ABIs, and real embedded-node saturation regressions.
The bounded FastAPI write run, at
3d5a7ea1, verified 35,200 scheduled updates across both ABIs: 6,538 acknowledged and 28,662 undispatched. Acknowledged values survived restart, undispatched orders stayed unchanged, and all eight hosts drained. Heavy client-capacity drops preclude a 1,000 updates/s claim.The S3 recovery/performance run, also at
3d5a7ea1, verified 473,571 acknowledged writes, 6,144 payload observations and 120 complete shutdowns across both ABIs. At concurrency 32, direct Python/native throughput was approximately 95% for writes and 83% for reads on the standard runner, and 93%/78% on the free-threaded runner. These disposable RustFS, shared-runner measurements have separate native controls; absolute rates across ABI runners are not comparable.The matched offered-read run passed both ABIs. Standard Python and native reached the 15,000 requests/s offered ceiling; the separate free-threaded runner retained a throughput gap. Reaching the offered ceiling does not establish saturated parity.
Independent verification of both earlier sustained-churn jobs, at
b5ba54ad, accounts for 1,056,768 acknowledged writes, 8,192 recovered cell instances and eight clean replacement shutdowns. Rust allocator usage and RSS continued growing in both native and Python controls, so recovery success does not qualify memory stability. The same bounded-cache workload now passes both ABIs: another 1,056,768 acknowledged writes, 8,192 recovered cell instances and eight joined replacements, with verified process-to-case identities. Late native allocation growth fell to near zero and final native allocated memory was about 13–15 MB; Python allocated-block counts were flat late in the run. RSS retained some growth and each primary observation spans about 10–14 minutes, so this does not establish arbitrary-duration memory stability.Qualification limits: final-head CI is required. The cache fix bounds one confirmed source of historical metadata growth; it does not establish stable total process memory. Long-running qualification, remaining historical failure attribution, production backend/SLO testing, and native read-performance parity remain open. Earlier unbounded-residency seeding failures remain failed evidence. Generated reports, raw evidence, and binaries are excluded from the PR.