Skip to content

Harden Python lifecycle and bound replication metadata - #108

Merged
forhappy merged 13 commits into
mainfrom
codex/python-write-load-qualification
Oct 24, 2026
Merged

forhappy merged 13 commits into
mainfrom
codex/python-write-load-qualification

Conversation

@forhappy

@forhappy forhappy commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Python task/object saturation previously surfaced as generic handler failures, and replication coverage metadata accumulated across historical activation epochs. This preserves typed overload responses, bounds the coverage cache, and removes per-invocation Python dispatch closures while adding durable-write and native/Python read qualification. Cleanup exceptions now remain failures even when their Python truth value is false. Startup confirmation also retires safely during shutdown and fencing, while caller cancellation preserves node-owned confirmation work.

Changes:

  • Retain bounded failure diagnostics for restored snapshot reads: original error and elapsed time, independently timed node status calls, transfer attempts, and a bounded joined shutdown attempt. The original read deadline and assertions remain unchanged.
  • Retire startup confirmation before preserve shutdown or node fencing changes cell phases. Ignore late inventory and owner-read results, keep the owned-cell count unknown after an aborted confirmation, and preserve queued confirmation when its last joined caller cancels. The native invariant checks remain enabled.
  • Release node-log eviction markers with the last retained bundle referencing their cell/epoch. Count distinct epoch membership once per evicted bundle; keep conservative live cutoffs even when later rows have lower transaction IDs. This removes historical metadata without discarding the upper bounds used by compaction.
  • Check completed-task exceptions explicitly against None. An exception class with __bool__ returning false previously allowed failed context cleanup to disappear from the shutdown report. Keep the failed invocation registered for the final cleanup pass.
  • Replace the per-invocation four-lambda dispatch dictionary with direct event argument selection. Preserve invocation, task and context semantics while avoiding the dictionary, function objects and closure cells.
  • Preserve HTTP 503 overload envelopes and unknown mutation outcomes at Python admission boundaries.
  • Bound remembered (cell, epoch) coverage to 4,096 entries by default, configurable through ReplicationOptions.coverage_cache_entries and CELLAR_LTX_COVERAGE_CACHE_ENTRIES. Zero disables caching. Evicted entries use the durable-store lookup path; failed lookups are not cached as successful zero coverage. Repeated updates preserve the maximum watermark without growing eviction metadata.
  • Add one-shot FastAPI PATCH workloads with finite resident-cell budgets, exact response validation, and restart verification of acknowledged, undispatched, and uncertain outcomes. Retain every scheduled outcome, including client-capacity drops.
  • Compare native and Python nested, initial, and direct-initial read paths; reject direct snapshot measurements that submit SDK operations.
  • Place the independent Raft traffic leader outside the repair fixture’s held data lane, preserving the original repair/traffic counts and assertions.
  • Update the native metrics fixture for the archive metrics added in Reduce multi-Raft image I/O and archive overhead #98: retain exact counts of 26 operations, 34 counters and 634 performance series; verify the archive names and identical series across 1 and 32 groups.
  • Document ABI concurrency, cache configuration, benchmark operation, and qualification limits.

Validation:

  • Snapshot failure diagnostics pass the full local snapshot suite (44 passed, two intentional ignores), strict Clippy, formatting, layout, documentation, dependency-policy and diff checks. A temporary expired-query injection exercised the failure-reporting path; it does not attribute or resolve the earlier VersionUnavailable incident.

  • The free-threaded lifetime failure, at c72b1c78, exposed a native confirmation marker surviving transition to Inactive. Four deterministic core regressions failed before the fix and now pass, covering preserve, caller cancellation, fencing and late inventory. All 34 ordinary core tests and 18 real embedded lifecycle/fencing cases pass locally across both ABIs, including five lifetime repetitions per ABI. Three native lifecycle/recovery tests, strict core Clippy, formatting, layout, documentation, dependency-policy and diff checks also pass.

  • The prior-head SDK run, at 64f566e1, failed native_snapshot_catchup_resumes_follower_reads: the restored follower query returned VersionUnavailable. Its cause remains unresolved. A separate Linux diagnostic run passed all 44 snapshot tests (two intentional ignores) and 200 focused repetitions with failure tracing; those passing repetitions do not close the original failure. Final-head CI must be evaluated separately.

  • Reproduced 19,488 historical bundle-epoch markers after 20,000 epochs with only 512 live bundles. Five owning-module regressions now pass, covering bounded historical retention, complete GC, shared epoch membership, arrival ordering, and real persisted-row reload/corruption refusal. All three bundle-scan/recovery memory tests, 26 native log tests, six replication tests, strict Clippy and repository checks pass locally.

  • The cleanup regression reproduces runtime_failed: false before the fix. Eager and suspended-handler failure regressions now require a failed runtime report and complete resource cleanup. All 24 runner tests pass on CPython 3.14.5 and 3.14.5t, as do real embedded-node lifetime, global-close, cancellation, cleanup-error and full-mailbox cleanup cases on both ABIs.

  • Reproduced the full-SDK metrics failure locally, corrected its stale expected inventory, then passed the focused regression and all 23 native Raft operations tests. The assertions retain exact cardinality and add archive metric identity checks.

  • The dispatch change passes all 22 runner tests and real embedded-node contract, delivery and schedule cases on CPython 3.14.5 and 3.14.5t.

  • Independent verification of the Linux dispatch comparison confirms matched native/baseline/candidate controls, raw scheduled outcomes, CPU placement, binary/source identity and all 18 joined hosts. Saturated nested-read throughput improved approximately 2–3% on the standard runner; the free-threaded result ranged from near unchanged to 1.4% better. Local alternating comparisons also passed all response and shutdown checks. These counter-read measurements do not establish native parity or production capacity.

  • Four focused cache tests pass, including fixed-population epoch churn, bounded eviction metadata, disabled caching, and real-store eviction/outage/retry with restored SQLite contents. The churn-bound regression fails against the previous unbounded implementation.

  • All 26 native log tests and six native replication tests pass. Three baseline harness tests pass. Strict Clippy for the affected library/integration targets, Rust formatting, module layout, documentation, dependency-policy and diff checks pass.

  • A real native-host churn run on the cache fix acknowledged 8,192 writes across 128 cells and 64 activation rounds. After killing the owner and deleting its local disk, the replacement recovered all 128 counters, acknowledged another 128 writes, and completed shutdown. This development-profile run validates recovery, not throughput or total memory stability.

  • The complete Raft multigroup suite and 12 repeated repair runs pass locally and in the Linux fixture qualification; the suite retains its one intentionally ignored test.

  • Earlier changes passed 17 compiled native-client tests, 22 runner tests on both Python ABIs, and real embedded-node saturation regressions.

  • The bounded FastAPI write run, at 3d5a7ea1, verified 35,200 scheduled updates across both ABIs: 6,538 acknowledged and 28,662 undispatched. Acknowledged values survived restart, undispatched orders stayed unchanged, and all eight hosts drained. Heavy client-capacity drops preclude a 1,000 updates/s claim.

  • The S3 recovery/performance run, also at 3d5a7ea1, verified 473,571 acknowledged writes, 6,144 payload observations and 120 complete shutdowns across both ABIs. At concurrency 32, direct Python/native throughput was approximately 95% for writes and 83% for reads on the standard runner, and 93%/78% on the free-threaded runner. These disposable RustFS, shared-runner measurements have separate native controls; absolute rates across ABI runners are not comparable.

  • The matched offered-read run passed both ABIs. Standard Python and native reached the 15,000 requests/s offered ceiling; the separate free-threaded runner retained a throughput gap. Reaching the offered ceiling does not establish saturated parity.

  • Independent verification of both earlier sustained-churn jobs, at b5ba54ad, accounts for 1,056,768 acknowledged writes, 8,192 recovered cell instances and eight clean replacement shutdowns. Rust allocator usage and RSS continued growing in both native and Python controls, so recovery success does not qualify memory stability. The same bounded-cache workload now passes both ABIs: another 1,056,768 acknowledged writes, 8,192 recovered cell instances and eight joined replacements, with verified process-to-case identities. Late native allocation growth fell to near zero and final native allocated memory was about 13–15 MB; Python allocated-block counts were flat late in the run. RSS retained some growth and each primary observation spans about 10–14 minutes, so this does not establish arbitrary-duration memory stability.

Qualification limits: final-head CI is required. The cache fix bounds one confirmed source of historical metadata growth; it does not establish stable total process memory. Long-running qualification, remaining historical failure attribution, production backend/SLO testing, and native read-performance parity remain open. Earlier unbounded-residency seeding failures remain failed evidence. Generated reports, raw evidence, and binaries are excluded from the PR.

@forhappy forhappy changed the title Qualify FastAPI durable updates under offered load Preserve Python overload errors and qualify durable FastAPI updates Oct 24, 2026
@forhappy
forhappy marked this pull request as ready for review October 24, 2026 06:17
@forhappy forhappy changed the title Preserve Python overload errors and qualify durable FastAPI updates Preserve Python overload errors and qualify durable writes and snapshot reads Oct 24, 2026
@forhappy forhappy changed the title Preserve Python overload errors and qualify durable writes and snapshot reads Bound replication coverage and qualify Python overload, writes and reads Oct 24, 2026
@forhappy forhappy changed the title Bound replication coverage and qualify Python overload, writes and reads Bound replication coverage and reduce Python request overhead Oct 24, 2026
@forhappy forhappy changed the title Bound replication coverage and reduce Python request overhead Bound replication metadata and harden Python request handling Oct 24, 2026
@forhappy forhappy changed the title Bound replication metadata and harden Python request handling Harden Python lifecycle and bound replication metadata Oct 24, 2026
@forhappy
forhappy merged commit 47f4a89 into main Oct 24, 2026
23 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant