fix(model): Close two chunking parity gaps with the Python reference - #355
Conversation
Signed-off-by: David Leong <leongdl@amazon.com>
… range Signed-off-by: David Leong <leongdl@amazon.com>
…ent ones Signed-off-by: David Leong <leongdl@amazon.com>
Signed-off-by: David Leong <leongdl@amazon.com>
|
Note on a review finding that has since been withdrawn (the automated reviewer removes superseded comments on each push), recorded because the last commit adds tests for it. The claim was that That does not reproduce. The forward loop already has an arm for it (
The shapes were worth having regardless, so all four are now in the reference-pinned case table and three are in |
Signed-off-by: David Leong <leongdl@amazon.com>
What
Closes two behaviour gaps between
StepParameterSpaceIteratorand the pure-Python reference (openjd-model-for-python), both found while addingchunks_task_count_overrideto the PyO3 bindings (openjd-model-for-python#344).Neither is a conformance failure — the TASK_CHUNKING conformance tests pass before and after. Both are divergences from the reference implementation, which is the contract consumers migrating from Python rely on.
Gap 1 — chunk metadata was reported only for adaptive spaces
chunks_parameter_nameandchunks_default_task_countwere both derived fromadaptive_info, which is only populated when there is no chunk override andtargetRuntimeSeconds > 0. So both returnedNonefor any non-adaptive chunked space:Frame/ 5Frame/ 5Some(1)Frame/ 1Frame/ 5Some(1)Frame/ 1Neither value is unknowable for a static space — both are in the template, or are the override the caller just supplied. A consumer inspecting a static chunked space could not learn which parameter chunks, or at what size.
Fixed by scanning for the chunked parameter unconditionally and keeping the size alongside the name.
set_chunks_default_task_countstays adaptive-only, since only an adaptive size is mutable mid-walk, and the getter still prefers the liveArcvalue when the space is adaptive.Gap 2 — contiguous chunked spaces refused random access
ContiguousChunkNode::getwas a no-op stub andneeds_sequentialincludedhas_contiguous_chunks, soget(i)returnedNonefor any contiguous chunked space whilelen()still reported a count. The reference answersit[i]for the same space, because Python materialises the chunk list up front — which is also why Python cannot handle a large range at all.Implemented
ContiguousChunkNode::chunk_at(index)instead, keeping laziness:Nis only known from the chunk counts before it. That is O(R) in sub-ranges, not O(N) in values, and reuses the same walk ascount_contiguous_chunks_for_range.jtakes one extra value whenceil((j+1)*leftovers/chunk_count) > ceil(j*leftovers/chunk_count), so the number of larger chunks beforejtelescopes toceil(j*leftovers/chunk_count). Offset and size follow in O(1).So a single 100-billion-value interval indexes in constant time rather than O(index) — there is a test indexing chunk 99,999,999 of 100,000,000.
needs_sequentialis now justadaptive, which is the only case where chunk N genuinely is not a function of N.has_contiguous_chunksbecame unused and is removed.Worth knowing for review: because
Iterator::nextis implemented asget(current_index)on the non-sequential path, this change routes every existing contiguous-chunking iteration test through the newchunk_at. Those tests passing is a strong check on the arithmetic, not just the new tests.Testing
cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test --workspace,cargo test --workspace --doc— all clean (27 suites).step_param_space::tests.The random-access test pins values against the reference implementation's actual output, generated by running
divide_int_list_into_contiguous_chunksfromopenjd-model-for-pythonfor the same shapes, rather than only assertingget == iterateself-consistency. Coverage includes uneven splits, exact splits, single-chunk, chunk-per-value, multi-interval ranges with gaps (1-5,8-12), stepped ranges (1-20:2), and mixed (1-3,7,11-15), plus indexing inside aProductNodewhere the child sees a divided index.Specs updated:
specs/model/parameter-space.mdandspecs/model/public-api.mdboth described contiguous chunking as sequential-only.Not addressed here
Two further divergences I measured but deliberately left alone, happy to file separately:
build_chunk_range_exprfront-loads the leftover values (size = small + if i < leftovers), while the reference spreads them (chunk_sizes[(i*chunk_count)//leftovers] += 1). For1-10atdefaultTaskCount: 3that is1-3, 4-6, 7,8, 9,10here versus1-3, 4,5, 6-8, 9,10in the reference — same count, same values, different grouping. NotablyContiguousChunkNodealready implements the reference's spread, so the two chunking paths in this file disagree with each other. No conformance test pins placement (contiguous-unevenhas zero leftovers), which is why it went unnoticed.StaticChunkNode::validate_containmentrequires exact chunk identity, so a chunk minted by the reference is rejected here (4,5and6-8in the example above), while the reference only checks subset-of-range. Arguably this implementation is the better behaviour, but the two disagree.I kept both out of this PR because fixing (1) changes the observable output of a published crate, and (2) is a semantics question rather than a defect. Say the word and I will send either.