Repository navigation
refactor(conformance): the judge is a codeupipe pipeline - #15
Merged
Merged
Conversation
The judge was eight plain functions in run.py. It is now a pipeline in conformance/judging: read the cases, choose the targets, run each target, compare, count, format, and write the results. Each step is a primitive in its own file and has tests that start no process. run.py keeps its command line and the function names that the tests and bench.py use. Each one calls a step of the pipeline, so there is one judge. The new judge was run beside the old one. Their reports are the same byte for byte on all 7 targets, and they gave the same verdict for each of 6550 results, 2700 of them wrong on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(conformance): the Python runner is codeupipe pipelines The runner was 15 plain functions in runner.py. It is now two pipelines in conformance/runners/python/case_runner: one plays a case on a target and one times a scenario. They share the steps that read the target, write the Predicates, build the pipeline and compile it. Each step is a primitive in its own file with tests of its own. runner.py is the command line. The new runner was run beside the old one on each Python target, with all 30 cases and 7 more that were made to fail. The reports are the same byte for byte. In the timing mode the outputs and the refusals are the same, and the times are the same within the noise of a run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(scorecard): use the steps of the new runner The differential test tool used two functions of the old runner file. It now uses the same steps that the runner uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The judge of the conformance suite is now a codeupipe pipeline. It was eight plain functions in
conformance/run.py.This PR is on top of #14. Merge #14 first.
Why
The test tools were the last large part of the repository that was plain code, so the repository's own rule (CUP-I008) needed a comment on each of their functions. This is the first of them written the way the product is written: as primitives.
The pipeline (
conformance/judging/)LoadCasesFilterNoTargetsNamedPredicate+DefaultTargetsFilterRunTargetsFilterJudgeResultsFilterSummarizeFilterFormatMatrixFilterWriteResultsTaplatest.jsonandlatest.mdTwo edges stay plain, each with its reason:
ProcessLauncher(starts a runner) andNodeProbe(asks which Node the machine has).run.pykeeps its command line and the names that the tests andbench.pyuse (load_cases,run_target,judge,default_targets,TARGETS). Each one calls a step of the pipeline, so there is one judge and not two.How it was checked against the old judge
A fault in the judge can make a broken runtime look correct. So the new judge ran beside the old one:
latest.jsonandlatest.mdare the same byte for byte, on all 7 targets.truein the place of1), and both judges failed each of them with the same problems.The old judge is not in this PR. It was used for this comparison only.
What it gains
allow-plaincomments are gone, and 2 are added for the edges. The pinned number in the repository self-check goes from 81 to 75.tests/conformance/test_judge_pipeline.py, 14 tests). Before, the judge was tested only through a full run of a target.Checks
bench.pyuses the same launcher and the same comparison as before.Not in this change
conformance/runners/python/runner.py) andscorecard/are still plain code with comments.🤖 Generated with Claude Code