Skip to content

Commit ebc8150

Browse files
committed
Implement persisted BM25 and bounded whole-partition operation flows
Checkpoint the complete R26 working-tree scope on main. Native compiler/analyzer reports zero errors and warnings with unchanged final source guard; scoped whitespace and three import repairs completed. Current full formatter verification, ordinary Release image and new runtime qualification remain pending. Preserve original Linux failures, KL034 acceptance and the honest 19-of-104 count; this checkpoint does not close the new gates.
1 parent 5649dff commit ebc8150

111 files changed

Lines changed: 3374 additions & 39 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# ADR-127: bounded native current BM25 corpus statistics
2+
3+
Status: Accepted current implementation decision. Original runtime and performance qualification remain OPEN.
4+
5+
## Decision and reason
6+
7+
KL-058 retains one replaceable exact-Terms statistics head per full partition/collection/field/tokenizer/authenticated-subject identity in native ZoneTree family `bm25-corpus-current`. This reduces repeated exact statistics calculation only when every current canonical dependency matches. [Bm25CorpusCurrent](../Features/Search/Bm25CorpusCurrent.md) is the canonical alias/field/phase/bounds and REQ/AC inventory; [ADR-018](ADR-018-global-rank-fusion.md) continues to own distributed statistics and final ranking.
8+
9+
Use the original generated four-field DistributedTextStatisticsV1 and exact tokenizer/scorer. Relocate the pure internal type mechanically to Core's Search Contracts, preserving namespace, alias and IDs; Query already consumes Core and receives its existing friend access. Do not add a duplicate type, scorer or Core-to-Query cycle. Public SDK/MCP/SQL contracts remain unchanged.
10+
11+
Only a server-verified private native Orleans RF3 maintenance publication may write the head. Generic client commands, envelopes and public MCP descriptors reject phase45. Fresh persisted administrator and target-subject corpus permissions, exact source dependencies and expected-generation CAS are verified before first effects. Existing ordered node-local commit/apply ownership and RF3 acknowledgement remain authoritative.
12+
13+
Search reads the head within its original scoped view. Exact current Terms/policy/schema/resource digest/outbox/registered physical owner/incarnation/full atomic placement admit unchanged scoring. Missing, unsupported, malformed or stale optimization data uses the same complete exact canonical calculation without writes. This does not change engine selection, current read-cut authority or authorization.
14+
15+
Bound total partition-family raw key/value bytes and records before first write with the existing MaxBatchBytes/MaxScanRecords, independently from existing request-read/cancellation budgets. Preserve exact native first-run and same-ID replay contracts; replicated operation.EvaluatedAt controls deterministic committed expiry, while caller capture retains original actual cancellation/deadline.
16+
17+
Admit publication only after current native family inventory/export/target/backup/census/restore paths handle the head. Preserve complete original values and strict corrupt/unsupported rejection. No legacy readers, previous-format migration, opaque receipt storage, whole-vocabulary dictionary, staging pages or unbounded secondary authority.
18+
19+
Portable header validation cannot recompute historical placement-bound PayloadDigest because the head has no full retained SourcePlacement. Keep strict native shape/key/bounds, original authenticated archive/MAC integrity and exact copied values. Query independently verifies against fresh placement and declines stale restored/moved data; only a real new capture can qualify reuse. No extra head field is introduced.
20+
21+
Fresh receipt authorization fences both current administrator outcome epoch and exact target subject epoch. Revocation/repair at a higher target epoch refuses the original receipt without effects. Document-tail/placement/CAS/quota and inner-expiry checks remain first-run only, independently from fresh receipt authorization.
22+
23+
## Consequences and qualification
24+
25+
Head capture Position/Applied/ReadGeneration are historical originating-view evidence, not values to equate across replicas or against post-publication Store.Position. The phase additionally binds actual AtomicPartitionPlacementResolution; its bounded digest detects full current placement changes without confusing atomic and physical group epochs. Existing distributed witnesses retain their genuine node-local cut.
26+
27+
Genuine native publication, exact result, denied/no-effect, CAS, full/one-over retained limits, reducing replacement, receipt/cold and current-format delivery flows map to every REQ/AC in the canonical feature. These flows supplement mandatory Linux Aspire RF3 SDK/official MCP and fault/resource qualification. Comparable original before/after measurements are required for any CPU, allocation, memory or latency claim; no source review or static configuration is performance proof.
28+
29+
```mermaid
30+
flowchart LR
31+
Canonical[Authorized complete canonical corpus] --> Private[Private native RF3 publication]
32+
Private --> Validate[Fresh authority, dependencies, CAS and retained limits]
33+
Validate --> Head[Bounded current native head]
34+
Search[Fresh scoped Search] --> Match[Exact current dependencies]
35+
Head --> Match
36+
Match --> Scores[Unchanged exact scoring]
37+
Match --> Recalculate[Unchanged canonical calculation]
38+
```
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# ADR-128: Bounded explicit whole-partition physical packing orchestration
2+
3+
Status: Accepted design; implementation and qualification gates remain open.
4+
Owners: KL071 / Client ClusterRouting and Integration ClusterRouting.
5+
Related: ADR-016, ADR-106, REQ-MOVE-001/003/005/007.
6+
7+
## Decision
8+
9+
Compose an explicitly supplied ordered set of existing PartitionMoveRequest operations in the authenticated .NET SDK. The SDK convenience API MoveWholePartitionsAsync returns IAsyncEnumerable<Result<PartitionMoveResult>> using the existing ManagedCode.Communication result contract. It introduces no route, public wire payload, serializer alias/field, operation enum, persisted plan record or separate dispatcher. Every actual request uses existing MovePartitionAsync, server-owned connection execution, fresh persisted administrator authorization, original native CQRS phase progress and the existing node-local RF3 movement authority.
10+
11+
Physical split/merge means moving complete existing atomic partitions between already registered physical groups. It never splits a hot atomic partition or changes logical scope, creates a physical group, transfers Control authority, or grants all-shard atomicity. The caller retains original ordered requests; settled per-partition receipts are the only durable recovery authority.
12+
13+
## Admission, bounds and lifetime
14+
15+
The convenience method snapshots all inputs before any request. Admit 1..64 requests, distinct nonempty stable MoveIds, distinct complete PartitionRefs, one nonempty destination, original Transfer/Resume modes and nonnegative original placement CAS. Preserve Revision0 for the real unassigned fallback. Reject malformed/duplicate/oversized local plans before transport. SDK preflight cannot replace native server authorization, owner registration, byte budgets, CAS, current lineage or admission.
16+
17+
The explicit64 bound is a local memory/in-flight policy: at most64 retained original request references,64 GUID identities and64 complete PartitionRef references. Execution has exactly one awaited existing parent request in flight and no task queue, producer, channel or retained result history. Every individual request keeps the existing actual server byte/work/deadline limits. This cardinality is independent of DatabaseLimits.MaxBatchMutations. Larger caller-selected inventories require explicit bounded sequences of original requests.
18+
19+
Yield each actual original terminal result once in order. Continue only after native successful Retired; yield failure or non-Retired success unchanged then stop. There is no batch result or invented aggregate error. Partial success remains visible. Re-enumerating the original ordered snapshot uses the SAME original MoveIds, destination and CAS: existing native parent reconciliation returns original settled receipts and advances only unfinished moves. Never generate retry IDs, reset phase expiry, reinterpret refusal, or redispatch an uncertain phase through another transport.
20+
21+
The supplied caller token and enumerator cancellation govern the entire SDK enumeration. Await the current original HTTP/native CQRS lifecycle; after cancellation or disposal no next request starts. The caller owns the borrowed SDK client. Server and existing test fixtures own and join their original operation/stream/native node resources; this composition adds no parallel lifetime.
22+
23+
## Complete-flow acceptance
24+
25+
REQ-KL071-PLAN-001 / AC-KL071-PLAN-001: real registered two-RF3 topology and two distinct all-model atomic partitions. Reject65 and duplicate inputs before effects; mutate the caller list after API creation and prove the actual original snapshot remains authoritative for enumeration.
26+
27+
REQ-KL071-PLAN-002 / AC-KL071-PLAN-002: settle the first complete A-to-B transfer, cancel the original enumeration before the second request, prove the second original row revision/owner/epoch and all linked models unchanged while the global directory records only the first legitimate publication, then join six owners/all18 native locks. Cold original per-partition records show Retired first and absent second parent. Reopen and run the SAME plan: exact first receipt replay, actual second transfer, literal native installed receipt/owner/CAS/epoch, published row revision from the original global directory plus the ordered publication ordinal, and final global directory revision and all original SDK/official MCP/Q1 command receipts.
28+
29+
REQ-KL071-PLAN-003 / AC-KL071-PLAN-003: real first same-ID/changed-CAS Conflict is yielded unchanged and stops before the second entry. Replay both original terminal results across full six-owner cold reopen without raw image/counter/body/index changes; no old effect is redispatched.
30+
31+
REQ-KL071-PLAN-005 / AC-KL071-PLAN-005: within the same original complete operation, settle the first original request and await early enumerator disposal; re-enumerate the SAME plan with the API token absent and a linked original enumerator token, then cancel before the second request. Require the exact cancellation token and byte-equal original first terminal receipts across early disposal, enumerator cancellation and the original API-token cancellation/snapshot control. The existing second-partition absent native parent, unchanged placement/models, six-owner cold reopening and full resume/merge/healthy continuation prove no second dispatch. Keep the original case declaration and deadline.
32+
33+
REQ-KL071-PLAN-004 / AC-KL071-PLAN-004: read actual current per-partition placements, execute a reverse B-to-A plan with fresh stable IDs and actual CAS, require literal current ownership/install receipts and full all-model state. Write distinct healthy literal documents under actual current owner epochs, prove real SDK/official MCP/Q1 receipts and documents, then full native cold-image equality on replay. Preserve existing ParentDeadline12m and all actual runtime cleanup bounds.
34+
35+
Supporting automated case: PhysicalShardPartitionMovementRf3Tests.OrderedWholePartitionsResumeOriginalReceiptsAfterPartialCancellationThenMergeCold. Helpers reuse actual existing registration, all-model seeds, native per-partition terminal oracles, complete raw cuts and Aspire-owned kill/restart/cleanup. No fake host or substitute grain. This supports KL071; every persisted-phase crash, endurance, fault and exact-source Linux qualification remains mandatory and open until original evidence is joined.
36+
37+
```mermaid
38+
flowchart TD
39+
Input[Original ordered whole-partition requests] --> Admission[Snapshot and bounded preflight]
40+
Admission --> Parent[Existing authenticated native parent]
41+
Parent --> Result[Actual per-partition terminal]
42+
Result -->|Retired| Next[Next original request]
43+
Next --> Parent
44+
Result -->|Failure or other phase| Stop[Yield unchanged and stop]
45+
Parent -->|Caller cancellation| Join[Join original request and start no next]
46+
Join --> Cold[Actual six-owner cold reopen]
47+
Cold --> Replay[Same IDs and CAS through original native parent]
48+
Replay --> Parent
49+
```
50+
51+
No compilation, runtime, Linux, durability or throughput claim follows from this design. Rollback removes only this SDK facade, its owned admission/helpers/test and requirement mapping; all original single-partition movement contracts remain.

‎docs/Features/BenchmarkComparisons/DotnetProfiling.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Use [dotnet-diagnostics-mcp](https://github.com/pedrosakuma/dotnet-diagnostics)
1212
| REQ-PROFILE-105-004: verify changes with matched evidence. | AC-PROFILE-105-004: matched before/after profiles plus separate clean isolated Linux GitHub benchmark artifacts with equivalent hardware, source identity, topology, scale, operation counts, resources and durability. | Reprofile and performance gates open; diagnostic overhead cannot become a speed claim. |
1313
| REQ-PROFILE-105-005: use diagnostics regularly during development alongside Roslynk. | AC-PROFILE-105-005: root policy selects bounded profiling for runtime symptoms and regressions, requires source mapping and matched reprofile, and distinguishes compiler/documentation work. | Root policy updated; policy review and repository governance pass. This workflow criterion uses explicit policy review rather than an implementation-mirroring test. |
1414

15-
The current local prerequisite build of `tests/KeyLoad.UnitTests/KeyLoad.UnitTests.csproj` exited1 with31 Orleans errors, including inaccessible `AnnProjectionJobProtocol.Identity` (`ORLEANS0107`/`CS0122`), experimental Journaling opt-ins (`ORLEANSEXP005`) and runtime-options/analyzer findings. No current KeyLoad application process was discovered; the matching paths belonged to the diagnostics client. The build failure prevents a source-matched functional application profile. Retain the original log privately under `artifacts/profiling/kl105-development-20261012/`; the task stays in progress and no bottleneck or speed claim follows.
15+
The initial local prerequisite build of `tests/KeyLoad.UnitTests/KeyLoad.UnitTests.csproj` exited1 with31 Orleans errors, including inaccessible `AnnProjectionJobProtocol.Identity` (`ORLEANS0107`/`CS0122`), experimental Journaling opt-ins (`ORLEANSEXP005`) and runtime-options/analyzer findings. No application process was discovered in that initial attempt; the matching paths belonged to the diagnostics client. Preserve that original failure under `artifacts/profiling/kl105-development-20261012/`. Later coherent R24 full ordinary Release and authenticated Linux source01a9c5eb solution builds completed with0 errors/0 warnings; those later compiler results supersede the initial prerequisite failure, but no KeyLoad application profile was collected. The current R26 functional image and bounded profiling workload remain pending. Its original Linux Unit source01a9c5eb artifact independently records69 failures from one RequestCqrs fixture startup and closed native ThreadPool-stall warnings. These warnings identify a symptom, not a measured hotspot or proven cause: attach only to the actual source/DLL/PDB-bound functional runner, retain original quality/loss/overhead evidence, and select no performance repair without that runtime capture. The task stays in progress and no speed claim follows.
1616

1717
Execute the remaining scope in this order: obtain the actual source-matched functional build and owned workload; confirm PID/runtime/DLL/PDB/topology; collect bounded native profiles and unavailable results; map evidence to the owning source; repair a confirmed issue with meaningful operation regressions; repeat matched profiles and qualify clean Linux measurements. Existing ADR-122 and native topology/ownership contracts remain sufficient; no new product boundary or benchmark scenario is selected here. The root owns shared policy/configuration/status and final evidence. A read-only supporting agent reviewed the installed-version upstream schemas and collector interpretation; it owns no process, edits or qualification.
1818

‎docs/Features/BenchmarkComparisons/NativeGateRepair.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,3 +135,19 @@ REQ-NGR-OPENLOOP-TRANSPORT-001: initialize the bounded native GitHub transport o
135135
AC-NGR-OPENLOOP-TRANSPORT-001 maps to REQ-NGR-OPENLOOP-TRANSPORT-001: the real Benchmarks `collect-open-loop` entry must retain its bounded authenticated downloads and native ZIP inventory under the owned output, preserve the original metadata bytes, and validate every selected job and terminal against the exact source/run/attempt/profile. Wrong, incomplete, oversized, mismatched or malformed evidence still refuses; no workload retry or historical fallback. Existing genuine native transport/collector TUnit coverage and exact-source Linux intake are required. Source review and a check-only patch do not qualify actual GitHub intake.
136136

137137
The current collector creates a fresh context but never initialized its transport before calling `downloadArtifact`; `requestNative` rejects that missing transport before launching an HTTP request. The original `bab8d9b` run stopped earlier at matrix job admission, so this confirmed source defect is a later prerequisite and is not claimed as its observed failure. Other reviewed collector profile copies use cohort-only validation or initialize their actual network context before requests. Owning map: `scripts/Features/BenchmarkComparisons/open-loop-github-collect.mjs`, existing native transport/ZIP helpers and the real Benchmarks collection step. Archive/capture immutability, bounds, cancellation, cohort and publication guards remain mandatory; exact-source runtime and full qualification remain OPEN.
138+
139+
## Large genuine native stream regression, 2026-10-12
140+
141+
REQ-NGR-LARGE-STREAM-001: the existing bounded native stream API must retain every original binary byte from genuine Node children at 219,259 bytes, 1 MiB and 8 MiB, for natural asynchronous stdout and synchronous fd1 writers. Admit the exact byte limit and independently validate original expected length/hash/full file content; reject one byte over the same limit, join every original child, seal each capture and permit a literal healthy subsequent capture. Retain the existing small-payload, timeout, archive digest/size, symlink, exclusive-path and ZIP inventory cases.
142+
143+
AC-NGR-LARGE-STREAM-001 maps to REQ-NGR-LARGE-STREAM-001 and AC-ISO-007: six native TUnit cases in IsolatedGitHubFileTests.AcIso007LargeNativeStreamRetainsOriginalBytesRefusesOneOverAndJoinsHealthyContinuation invoke the production streamToFile/validateDownloadedArchive through real bounded Node process ownership. Each covers exact-limit acceptance, one-over refusal, original child exit, immutable file mode, original byte/hash equality, healthy continuation and unchanged process signal-listener counts. The outer native owner retains its original deadline/cancellation/joined cleanup. Execute only in the Linux Benchmarks contract lane; no local benchmark execution.
144+
145+
Owning map: existing ComparisonTests/Features/BenchmarkComparisons/UnitContracts/Cases/IsolatedGitHubFileTests.cs and Processes/IsolatedGitHubFileProgram.cs. This additive binary-stream regression does not alter transport code, provide an HTTP provider response or explain the original malformed Website body. Actual authenticated Website capture and Linux runtime are independent required evidence; all new test outcomes remain OPEN.
146+
147+
```mermaid
148+
flowchart LR
149+
Child[Real bounded native stdout child] --> Exact[Exact-limit original byte capture]
150+
Exact --> Reject[One-over refusal and joined original child]
151+
Reject --> Healthy[Exact healthy continuation and sealed files]
152+
Healthy --> Linux[Original Linux TUnit report]
153+
```

0 commit comments

Comments
 (0)