Skip to content

test: Add FLOAT floatstring numeric-comparison conformance fixtures - #182

Open
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-float-string-numeric-compare
Open

test: Add FLOAT floatstring numeric-comparison conformance fixtures#182
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-float-string-numeric-compare

Conversation

@leongdl

@leongdl leongdl commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

What

Five fixtures closing the two coverage gaps @jericht raised on #179: no job-level FLOAT analog of that PR's INT <intstring> bounds fixtures, and no string-form numeric comparison for amounts[].min/.max.

Suite delta on both reference implementations: base 656 → 660, FEATURE_BUNDLE_1 55 → 56, 0 failed before and after.

This branch is off merged mainline (3d8811b, #158) and does not depend on #179. The INT fixtures it mirrors are still on that branch, so nothing here references a file that is not live; the spec is cited directly instead.

Finding 1: FLOAT parameter definition bounds and allowedValues

JobFloatParameterDefinition (§2.4) declares minValue, maxValue and allowedValues as <float> | <floatstring> with no extension gate, and the §2 preamble (L151) states these are "constraints that must be enforced during validation". When the bound is authored as a string and the submitted value arrives as a number, the comparison has to be numeric.

I confirmed the gap is real rather than taking it on trust. The three candidate fixtures do not fill it:

Existing fixture Why it does not cover this
base/jobs/1.1--float-parameter-string-coercion-valid Coerces a submitted '5.5', declares no bounds at all
base/job_templates/2.4--float-param-minmax-floatstring Decode-only, no submitted value, bounds "0.0"/"1.0" do not diverge lexically
base/job_templates/2.4--float-param-allowedvalues-floatstring Decode-only, so no membership is ever decided against a value

Four fixtures, paired accept and reject on each axis:

Fixture Construct Asserts
1.1--float-floatstring-bounds-satisfied.test.yaml minValue: '010.0', maxValue: '100.0', submitted 50.5 Accepted, renders 50.5
1.1--float-floatstring-bounds-violation.invalid.test.yaml Same bounds, submitted 5.5 Rejected at job creation
1.1--float-floatstring-allowedvalues-member.test.yaml ['1.5','2.5','04.5'], submitted 4.5 Accepted, renders 4.5
1.1--float-floatstring-allowedvalues-violation.invalid.test.yaml Same list, submitted 3.5 Rejected at job creation

The values are chosen so the two comparison strategies disagree in both directions, which the INT pair only achieves on the accept side:

  • accept side: '50.5' sorts above '100.0' lexically, so a lexical implementation wrongly rejects a valid job;
  • reject side: '5.5' sorts above '010.0' lexically, so a lexical implementation wrongly admits an out-of-range job.

allowedValues carries '04.5' against a submitted 4.5, so membership matches numerically and matches no element textually. That is the sharper half of the membership test and nothing covered it: an implementation that rejects a numerically-equal member passed every fixture in the suite.

Per @jericht's note, a FLOAT default-resolves fixture is not included: it collides with the <floatstring> normalization question and with EXPR/jobs/expr1.3.4--float-passthrough. Numeric comparison is normalization-independent, so these five land live.

Both fixtures use a :END terminator on the asserted output. Without it SCALE:4.5 is a prefix of SCALE:4.50, which is the substring trap the quorum review caught on #158's indexed-property fixture.

Finding 2: amounts min/max string-form comparison

amounts[].min and .max (§3.3.1, L958/960) are <nonnegativefloat> | <nonnegativefloatstring> and <positivefloat> | <positivefloatstring>. Note the string form is @extension FEATURE_BUNDLE_1, so this fixture belongs under FEATURE_BUNDLE_1/, not base/ where the numeric-literal fixture lives.

FEATURE_BUNDLE_1/job_templates/3.3.1--amount-floatstring-min-greater-than-max.invalid.yaml sets min: '010.0', max: '5.0'. '010.0' sorts below '5.0' lexically, so a lexical comparison reads min as under max and accepts the template.

One caveat worth stating: §3.3.1 does not state the min ≤ max constraint in its text. It is established by two live mainline fixtures (base/job_templates/3.3.1--amount-min-greater-than-max.invalid.yaml and 3.3.1--min-greater-than-max.invalid.yaml), both of which use numeric literals only. This fixture adds the string-form axis to a constraint the suite already pins; it does not introduce the constraint. If reviewers would rather the spec said so explicitly first, that is a reasonable separate issue.

Verification

Both reference implementations, every fixture:

  • Python openjd-cli 0.7.7 / openjd-model 0.11.9 / openjd-sessions 0.12.1
  • openjd-rs at main 361b2c5, release build

The error text confirms the comparison is numeric rather than the fixture passing for an unrelated reason. Python: Value (5.5) for parameter Scale must be at least 10.0. openjd-rs: Parameter 'Scale': value 5.5 is less than minimum 10. Both parsed '010.0'.

Mutation checks

Ten fixture-level mutants, ten caught (5 fixtures × 2 implementations). Accept fixtures had their expected output corrupted; reject fixtures had their input made legal. Every one flipped to failing, so no assertion is vacuous.

Five implementation-level mutants against openjd-rs, five caught. These are the ones that decide whether the fixtures catch the defect class they name. The enforcement site is MergedParameterDefinition in crates/openjd-model/src/job/create_job/parameters.rs (Float arm) and the amounts check is crates/openjd-model/src/template/validate_v2023_09/structure.rs.

Mutant Fixture Result
min bound not enforced bounds-violation caught
min compared lexically bounds-violation caught
allowedValues not enforced allowedvalues-violation caught
membership rejects a numerically-equal member allowedvalues-member caught
amounts min ≤ max not enforced amount-floatstring-min>max caught

The lexical mutant is the point of the exercise, and the new negative is the only fixture in the suite that fails against it. Under the two min mutants the accept fixture still passes, so the mutants are changing the specific behaviour rather than breaking everything. Baseline was confirmed clean before and after, and the worktree was restored and checksum-verified between mutants.

One honest limitation: for the allowedvalues-member fixture, openjd-rs normalizes '04.5' to 4.5 at parse time and discards the authored spelling, so that code path structurally cannot compare authored spellings. The mutant injects the observable rejection directly instead of simulating a string comparison. The fixture still pins the required behaviour for any implementation that does retain the text.

Notes for review

Closes the two coverage gaps jericht raised on OpenJobDescription#179: no job-level FLOAT
analog of the INT intstring bounds fixtures, and no string-form numeric
comparison for amounts min/max.

Section 2.4 declares minValue, maxValue and allowedValues as
`<float> | <floatstring>` with no extension gate, and the section 2
preamble makes them constraints validation must enforce. The suite had
template-level accept coverage only: 2.4--float-param-minmax-floatstring
and 2.4--float-param-allowedvalues-floatstring never submit a value, and
1.1--float-parameter-string-coercion-valid declares no bounds. Nothing
decided a string-form FLOAT constraint against a submitted value.

Four job-level fixtures close that, paired accept and reject on both
bounds and allowedValues. Bounds are '010.0' and '100.0' so the compare
diverges lexically from numerically in both directions: 50.5 satisfies
'100.0' numerically but sorts above it lexically, and 5.5 violates
'010.0' numerically but sorts above it lexically. allowedValues holds
'04.5' so membership for a submitted 4.5 can only be decided on the
parsed value.

Section 3.3.1 admits `<nonnegativefloatstring>` and
`<positivefloatstring>` for amounts min and max under FEATURE_BUNDLE_1,
so the fifth fixture pins min '010.0' > max '5.0'. The existing
3.3.1--amount-min-greater-than-max.invalid.yaml uses numeric literals
only, and '010.0' sorts below '5.0' lexically.

Verified against openjd-cli 0.7.7 with openjd-model 0.11.9 and against
openjd-rs at main 361b2c5. Slice results on both: base 656 -> 660,
FEATURE_BUNDLE_1 55 -> 56, 0 failed before and after.

Mutation-checked twice over. Ten fixture-level mutants, ten caught: each
accept fixture with its expected output corrupted, each reject fixture
with its input made legal. Five implementation-level mutants against
openjd-rs, five caught: min bound unenforced, min compared lexically,
allowedValues unenforced, membership rejecting a numerically-equal
member, and amounts min<=max unenforced. The lexical mutant is the one
that matters, and only the new negative catches it.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl requested a review from a team as a code owner September 8, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant