Skip to content

fix(model): Close two chunking parity gaps with the Python reference - #355

Merged
mwiebe merged 5 commits into
OpenJobDescription:mainfrom
leongdl:fix/chunk-metadata-and-contiguous-random-access
Sep 1, 2026
Merged

fix(model): Close two chunking parity gaps with the Python reference#355
mwiebe merged 5 commits into
OpenJobDescription:mainfrom
leongdl:fix/chunk-metadata-and-contiguous-random-access

Conversation

@leongdl

@leongdl leongdl commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

What

Closes two behaviour gaps between StepParameterSpaceIterator and the pure-Python reference (openjd-model-for-python), both found while adding chunks_task_count_override to the PyO3 bindings (openjd-model-for-python#344).

Neither is a conformance failure — the TASK_CHUNKING conformance tests pass before and after. Both are divergences from the reference implementation, which is the contract consumers migrating from Python rely on.

Gap 1 — chunk metadata was reported only for adaptive spaces

chunks_parameter_name and chunks_default_task_count were both derived from adaptive_info, which is only populated when there is no chunk override and targetRuntimeSeconds > 0. So both returned None for any non-adaptive chunked space:

space override before reference
adaptive none Frame / 5 Frame / 5
adaptive Some(1) None / None Frame / 1
static none None / None Frame / 5
static Some(1) None / None Frame / 1

Neither value is unknowable for a static space — both are in the template, or are the override the caller just supplied. A consumer inspecting a static chunked space could not learn which parameter chunks, or at what size.

Fixed by scanning for the chunked parameter unconditionally and keeping the size alongside the name. set_chunks_default_task_count stays adaptive-only, since only an adaptive size is mutable mid-walk, and the getter still prefers the live Arc value when the space is adaptive.

Gap 2 — contiguous chunked spaces refused random access

ContiguousChunkNode::get was a no-op stub and needs_sequential included has_contiguous_chunks, so get(i) returned None for any contiguous chunked space while len() still reported a count. The reference answers it[i] for the same space, because Python materialises the chunk list up front — which is also why Python cannot handle a large range at all.

Implemented ContiguousChunkNode::chunk_at(index) instead, keeping laziness:

  • Locating the chunk's interval still walks intervals, since which interval holds chunk N is only known from the chunk counts before it. That is O(R) in sub-ranges, not O(N) in values, and reuses the same walk as count_contiguous_chunks_for_range.
  • The chunk within the interval is computed arithmetically. Chunk j takes one extra value when ceil((j+1)*leftovers/chunk_count) > ceil(j*leftovers/chunk_count), so the number of larger chunks before j telescopes to ceil(j*leftovers/chunk_count). Offset and size follow in O(1).

So a single 100-billion-value interval indexes in constant time rather than O(index) — there is a test indexing chunk 99,999,999 of 100,000,000.

needs_sequential is now just adaptive, which is the only case where chunk N genuinely is not a function of N. has_contiguous_chunks became unused and is removed.

Worth knowing for review: because Iterator::next is implemented as get(current_index) on the non-sequential path, this change routes every existing contiguous-chunking iteration test through the new chunk_at. Those tests passing is a strong check on the arithmetic, not just the new tests.

Testing

  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, cargo test --workspace --doc — all clean (27 suites).
  • 10 new tests in step_param_space::tests.

The random-access test pins values against the reference implementation's actual output, generated by running divide_int_list_into_contiguous_chunks from openjd-model-for-python for the same shapes, rather than only asserting get == iterate self-consistency. Coverage includes uneven splits, exact splits, single-chunk, chunk-per-value, multi-interval ranges with gaps (1-5,8-12), stepped ranges (1-20:2), and mixed (1-3,7,11-15), plus indexing inside a ProductNode where the child sees a divided index.

Specs updated: specs/model/parameter-space.md and specs/model/public-api.md both described contiguous chunking as sequential-only.

Not addressed here

Two further divergences I measured but deliberately left alone, happy to file separately:

  1. Noncontiguous leftover placement. build_chunk_range_expr front-loads the leftover values (size = small + if i < leftovers), while the reference spreads them (chunk_sizes[(i*chunk_count)//leftovers] += 1). For 1-10 at defaultTaskCount: 3 that is 1-3, 4-6, 7,8, 9,10 here versus 1-3, 4,5, 6-8, 9,10 in the reference — same count, same values, different grouping. Notably ContiguousChunkNode already implements the reference's spread, so the two chunking paths in this file disagree with each other. No conformance test pins placement (contiguous-uneven has zero leftovers), which is why it went unnoticed.
  2. Containment strictness. StaticChunkNode::validate_containment requires exact chunk identity, so a chunk minted by the reference is rejected here (4,5 and 6-8 in the example above), while the reference only checks subset-of-range. Arguably this implementation is the better behaviour, but the two disagree.

I kept both out of this PR because fixing (1) changes the observable output of a published crate, and (2) is a semantics question rather than a defect. Say the word and I will send either.

Signed-off-by: David Leong <leongdl@amazon.com>
@leongdl
leongdl requested a review from a team as a code owner September 1, 2026 04:46
Comment thread crates/openjd-model/src/job/step_param_space.rs Outdated
Comment thread crates/openjd-model/src/job/step_param_space.rs Outdated
Comment thread crates/openjd-model/src/job/step_param_space.rs
Comment thread crates/openjd-model/src/job/step_param_space.rs Outdated
Comment thread crates/openjd-model/src/job/step_param_space.rs Outdated
… range

Signed-off-by: David Leong <leongdl@amazon.com>
Comment thread crates/openjd-model/src/job/step_param_space.rs Outdated
…ent ones

Signed-off-by: David Leong <leongdl@amazon.com>
Signed-off-by: David Leong <leongdl@amazon.com>
@leongdl

leongdl commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Note on a review finding that has since been withdrawn (the automated reviewer removes superseded comments on each push), recorded because the last commit adds tests for it.

The claim was that interval_end_index merges intervals only in one direction, so it would still disagree with count_contiguous_chunks_from_sub_ranges when the following sub-range has step > 1 — predicting 1-3,4-8:2 at defaultTaskCount 2 would report len() = 4 but yield 5 chunks.

That does not reproduce. The forward loop already has an arm for it (else if next_sr.start() == last_val + 1 && next_sr.step() > 1 { end += 1; break; }), so the follower's first value is absorbed and the rest stay isolated — the same place the counter's value-by-value walk lands. Measured against divide_int_list_into_contiguous_chunks from the reference:

shape len() iterated chunks reference
1-3,4-8:2 dtc=2 4 4 1-2, 3-4, 6-6, 8-8 identical
1-3,4-8:2 dtc=3 4 4 1-2, 3-4, 6-6, 8-8 identical
1-4:3,5-9:2 dtc=2 4 4 1-1, 4-5, 7-7, 9-9 identical
2-8:2,9-12 dtc=2 6 6 2-2, 4-4, 6-6, 8-9, 10-11, 12-12 identical

The shapes were worth having regardless, so all four are now in the reference-pinned case table and three are in test_len_agrees_with_iteration_across_mixed_step_ranges. If that merge ever does become one-directional, they fail.

Comment thread crates/openjd-model/src/job/step_param_space.rs
Comment thread crates/openjd-model/src/job/step_param_space.rs
Comment thread specs/model/parameter-space.md
Signed-off-by: David Leong <leongdl@amazon.com>
Comment thread crates/openjd-model/src/job/step_param_space.rs
Comment thread crates/openjd-model/src/job/step_param_space.rs
@mwiebe
mwiebe merged commit 17d16ce into OpenJobDescription:main Sep 1, 2026
22 checks passed
@github-actions github-actions Bot mentioned this pull request Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants