feat: per-test generators to end golden-test drift and speed up CI#1229
Merged
alandefreitas merged 3 commits intoJun 12, 2026
Merged
Conversation
The `generator` option now accepts a single id, a YAML sequence, or a comma-separated string, and the tool runs each generator in turn, so the corpus is extracted once and rendered by every requested generator. A new built-in `noop` generator runs extraction and reports diagnostics without writing any output, which is useful for checking that extraction succeeds and surfacing warnings.
Each golden test now runs once, with the generator(s) it declares in its own config, instead of being run three times (xml, adoc, html).
|
| Scope | Lines Δ% | Lines Δ | Lines + | Lines - | Files Δ | Files + | Files ~ | Files ↔ | Files - |
|---|---|---|---|---|---|---|---|---|---|
| 🥇 Golden Tests | 99% | 124757 | 9 | 124748 | 709 | 2 | 6 | - | 701 |
| 🛠️ Source | <1% | 356 | 344 | 12 | 8 | 4 | 4 | - | - |
| 🏗️ Build | <1% | 150 | 66 | 84 | 1 | - | 1 | - | - |
| 🧪 Unit Tests | <1% | 137 | 90 | 47 | 5 | - | 5 | - | - |
| 🔧 Toolchain | <1% | 126 | - | 126 | 1 | - | 1 | - | - |
| ⚙️ CI | <1% | 118 | 63 | 55 | 9 | - | 9 | - | - |
| 📄 Docs | <1% | 23 | 19 | 4 | 2 | - | 2 | - | - |
| 🧰 Tooling | <1% | 22 | 19 | 3 | 2 | - | 2 | - | - |
| Total | 100% | 125689 | 610 | 125079 | 737 | 6 | 30 | - | 701 |
Legend: Files + (added), Files ~ (modified), Files ↔ (renamed), Files - (removed)
🔝 Top Files
- test-files/golden-tests/symbols/record/class-template-specializations-1.adoc (Golden Tests): 8421 lines Δ (+0 / -8421)
- test-files/golden-tests/symbols/record/class-template-specializations-1.html (Golden Tests): 8260 lines Δ (+0 / -8260)
- test-files/golden-tests/symbols/variable/function-objects.adoc (Golden Tests): 1304 lines Δ (+0 / -1304)
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1229 +/- ##
========================================
Coverage 83.16% 83.16%
========================================
Files 35 35
Lines 3658 3658
Branches 843 843
========================================
Hits 3042 3042
Misses 409 409
Partials 207 207
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
An automated preview of the documentation is available at https://1229.mrdocs.prtest2.cppalliance.org/index.html If more commits are pushed to the pull request, the docs will rebuild at the same URL. 2026-06-12 06:15:18 UTC |
With golden fixtures no longer drifting across three formats on every change, the pipeline can run the full test set on every build and a rebasing merge queue becomes practical: the queue can rebase a PR onto the current tip and re-test it without the expected outputs churning. All actions are also bumped to their latest releases (cpp-actions to v1.9.5 and the GitHub-maintained actions to their node24 versions) so the runner no longer warns about the deprecated node20 runtime.
alandefreitas
force-pushed
the
feat/per-test-generators
branch
from
c0973e4 to
85f7c54
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes CI faster and stops the golden tests from drifting on unrelated changes. The motivation is mostly about CI. Today every golden test is run three times, once for each output format, and each run keeps its own expected file. A change that affects one format rewrites expected files all over the tree. That drift forces rebases where we have to regenerates the expected output files. Pull requests then end up touching more than a thousand files that have nothing to do with the actual change.
The fix is to associate each test with the format it tests. The expected files don't change because the basic corpus doesn't change that much. This will allow us to enable the github merge queue that will speed up development so we don't block each other anymore.
Changes
A documentation generator can now be chosen per test, and a test can ask for more than one at once. A noop generator is provided for tests that only need to confirm extraction works. With that system, each golden test is tested only once in the cheapest format. About 700 redundant files were expected to be removed. Existing configurations keep working unchanged.
Testing
One golden test run now covers every test with the generator it requested.
Documentation
The configuration reference is generated from the options definition.