test: Add FLOAT floatstring numeric-comparison conformance fixtures - #182
Open
leongdl wants to merge 1 commit into
Open
test: Add FLOAT floatstring numeric-comparison conformance fixtures#182leongdl wants to merge 1 commit into
leongdl wants to merge 1 commit into
Conversation
Closes the two coverage gaps jericht raised on OpenJobDescription#179: no job-level FLOAT analog of the INT intstring bounds fixtures, and no string-form numeric comparison for amounts min/max. Section 2.4 declares minValue, maxValue and allowedValues as `<float> | <floatstring>` with no extension gate, and the section 2 preamble makes them constraints validation must enforce. The suite had template-level accept coverage only: 2.4--float-param-minmax-floatstring and 2.4--float-param-allowedvalues-floatstring never submit a value, and 1.1--float-parameter-string-coercion-valid declares no bounds. Nothing decided a string-form FLOAT constraint against a submitted value. Four job-level fixtures close that, paired accept and reject on both bounds and allowedValues. Bounds are '010.0' and '100.0' so the compare diverges lexically from numerically in both directions: 50.5 satisfies '100.0' numerically but sorts above it lexically, and 5.5 violates '010.0' numerically but sorts above it lexically. allowedValues holds '04.5' so membership for a submitted 4.5 can only be decided on the parsed value. Section 3.3.1 admits `<nonnegativefloatstring>` and `<positivefloatstring>` for amounts min and max under FEATURE_BUNDLE_1, so the fifth fixture pins min '010.0' > max '5.0'. The existing 3.3.1--amount-min-greater-than-max.invalid.yaml uses numeric literals only, and '010.0' sorts below '5.0' lexically. Verified against openjd-cli 0.7.7 with openjd-model 0.11.9 and against openjd-rs at main 361b2c5. Slice results on both: base 656 -> 660, FEATURE_BUNDLE_1 55 -> 56, 0 failed before and after. Mutation-checked twice over. Ten fixture-level mutants, ten caught: each accept fixture with its expected output corrupted, each reject fixture with its input made legal. Five implementation-level mutants against openjd-rs, five caught: min bound unenforced, min compared lexically, allowedValues unenforced, membership rejecting a numerically-equal member, and amounts min<=max unenforced. The lexical mutant is the one that matters, and only the new negative catches it. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Five fixtures closing the two coverage gaps @jericht raised on #179: no job-level FLOAT analog of that PR's INT
<intstring>bounds fixtures, and no string-form numeric comparison foramounts[].min/.max.Suite delta on both reference implementations: base 656 → 660, FEATURE_BUNDLE_1 55 → 56, 0 failed before and after.
This branch is off merged
mainline(3d8811b, #158) and does not depend on #179. The INT fixtures it mirrors are still on that branch, so nothing here references a file that is not live; the spec is cited directly instead.Finding 1: FLOAT parameter definition bounds and allowedValues
JobFloatParameterDefinition(§2.4) declaresminValue,maxValueandallowedValuesas<float> | <floatstring>with no extension gate, and the §2 preamble (L151) states these are "constraints that must be enforced during validation". When the bound is authored as a string and the submitted value arrives as a number, the comparison has to be numeric.I confirmed the gap is real rather than taking it on trust. The three candidate fixtures do not fill it:
base/jobs/1.1--float-parameter-string-coercion-valid'5.5', declares no bounds at allbase/job_templates/2.4--float-param-minmax-floatstring"0.0"/"1.0"do not diverge lexicallybase/job_templates/2.4--float-param-allowedvalues-floatstringFour fixtures, paired accept and reject on each axis:
1.1--float-floatstring-bounds-satisfied.test.yamlminValue: '010.0',maxValue: '100.0', submitted50.550.51.1--float-floatstring-bounds-violation.invalid.test.yaml5.51.1--float-floatstring-allowedvalues-member.test.yaml['1.5','2.5','04.5'], submitted4.54.51.1--float-floatstring-allowedvalues-violation.invalid.test.yaml3.5The values are chosen so the two comparison strategies disagree in both directions, which the INT pair only achieves on the accept side:
'50.5'sorts above'100.0'lexically, so a lexical implementation wrongly rejects a valid job;'5.5'sorts above'010.0'lexically, so a lexical implementation wrongly admits an out-of-range job.allowedValuescarries'04.5'against a submitted4.5, so membership matches numerically and matches no element textually. That is the sharper half of the membership test and nothing covered it: an implementation that rejects a numerically-equal member passed every fixture in the suite.Per @jericht's note, a FLOAT
default-resolves fixture is not included: it collides with the<floatstring>normalization question and withEXPR/jobs/expr1.3.4--float-passthrough. Numeric comparison is normalization-independent, so these five land live.Both fixtures use a
:ENDterminator on the asserted output. Without itSCALE:4.5is a prefix ofSCALE:4.50, which is the substring trap the quorum review caught on #158's indexed-property fixture.Finding 2: amounts min/max string-form comparison
amounts[].minand.max(§3.3.1, L958/960) are<nonnegativefloat> | <nonnegativefloatstring>and<positivefloat> | <positivefloatstring>. Note the string form is@extension FEATURE_BUNDLE_1, so this fixture belongs underFEATURE_BUNDLE_1/, notbase/where the numeric-literal fixture lives.FEATURE_BUNDLE_1/job_templates/3.3.1--amount-floatstring-min-greater-than-max.invalid.yamlsetsmin: '010.0',max: '5.0'.'010.0'sorts below'5.0'lexically, so a lexical comparison reads min as under max and accepts the template.One caveat worth stating: §3.3.1 does not state the min ≤ max constraint in its text. It is established by two live mainline fixtures (
base/job_templates/3.3.1--amount-min-greater-than-max.invalid.yamland3.3.1--min-greater-than-max.invalid.yaml), both of which use numeric literals only. This fixture adds the string-form axis to a constraint the suite already pins; it does not introduce the constraint. If reviewers would rather the spec said so explicitly first, that is a reasonable separate issue.Verification
Both reference implementations, every fixture:
openjd-cli0.7.7 /openjd-model0.11.9 /openjd-sessions0.12.1main361b2c5, release buildThe error text confirms the comparison is numeric rather than the fixture passing for an unrelated reason. Python:
Value (5.5) for parameter Scale must be at least 10.0.openjd-rs:Parameter 'Scale': value 5.5 is less than minimum 10. Both parsed'010.0'.Mutation checks
Ten fixture-level mutants, ten caught (5 fixtures × 2 implementations). Accept fixtures had their expected output corrupted; reject fixtures had their input made legal. Every one flipped to failing, so no assertion is vacuous.
Five implementation-level mutants against openjd-rs, five caught. These are the ones that decide whether the fixtures catch the defect class they name. The enforcement site is
MergedParameterDefinitionincrates/openjd-model/src/job/create_job/parameters.rs(Float arm) and the amounts check iscrates/openjd-model/src/template/validate_v2023_09/structure.rs.The lexical mutant is the point of the exercise, and the new negative is the only fixture in the suite that fails against it. Under the two min mutants the accept fixture still passes, so the mutants are changing the specific behaviour rather than breaking everything. Baseline was confirmed clean before and after, and the worktree was restored and checksum-verified between mutants.
One honest limitation: for the
allowedvalues-memberfixture, openjd-rs normalizes'04.5'to4.5at parse time and discards the authored spelling, so that code path structurally cannot compare authored spellings. The mutant injects the observable rejection directly instead of simulating a string comparison. The fixture still pins the required behaviour for any implementation that does retain the text.Notes for review
.invalid.test.yamlfixtures are named that way rather than.invalid.yamlbecause the templates are statically valid; the error only fires once the submitted value is bound. Same reasoning as test: Add conformance fixtures for every coercible 2023-09 field #179's INT pair.