Repository navigation
Test hardening: regression tests, guard checks against weakened tests, CODEOWNERS - #828
Merged
Merged
Conversation
…ests, API snapshot, coverage floor, PR template Phases 1-4, 6, 7 of the test hardening plan (logs/test_hardening_2026-10/PLAN.md). - test/regression/: 16 seeded networks ported from the review canaries; golden spike/voltage/trace/weight files (spikes exact, floats atol 1e-6, skipped on a different torch major.minor); invariants (split run, reset vs fresh, state_dict round trip, learning off, batch of copies, dt scaling). 100 passed, 15 strict xfail for known bugs (rule state outside state_dict, MulticompartmentConnection weights not saved, CSRM s_w window, PR #813 and #815 crashes). - test/network/test_all_classes.py: every Nodes, connection, learning rule, feature and model class built and run; 25 strict xfail pin known crashes. Core coverage network 79-90% -> 91%, learning 84-87% -> 95%, models 43% -> 99%. - test/test_api_surface.py + api_surface.json: public signatures pinned; regen scripts refuse to run without --reason. - CI: pytest-cov in dev group, coverage floor script (network 90, learning 94), coverage.xml artifact. - CONTRIBUTING.md, PR template, test/regression/README.md, bindsnet/learning/README.md known limitations. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- test/regression/test_op_counts.py + op_counts.json: kernel dispatch counts for one update() of every learning rule (batch 1 and 4), one DiehlAndCook2015 step (batch 1 and 16) and LIFNodes.forward; fails when a count grows, warns on a drop. regen_op_counts.py refuses without --reason. 32 tests, 3.9 s. - .github/workflows/bench.yml + scripts/bench_ab.py: non-required PR workflow, merge base vs head, alternating rounds, median ratio with bootstrap interval, SLOWER above 1.25; report in the job summary and an artifact. Local no-change run: medians 0.965 to 1.012; a 0.5 ms/step sleep flagged on all workloads. Calibration on GitHub runners still pending. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…WNERS Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts; document the guard checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hananel-Hazan
added
the
tests-changed-ok
…6%, learning 95.2%) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes and why
Contributors increasingly send fixes written by AI tools. Some of these fixes are wrong, and some correct fixes break something else. This pull request adds tests and checks that fail when results, behaviour, the public API or the amount of computation change without the author saying so. It changes no library behaviour and no simulated results.
Tests added (local run: 513 passed, 3 skipped, 40 expected failures, about 30 s; line coverage of
bindsnet/57% → 65%):test/regression/test_golden.py: 16 small seeded networks. Their spikes, voltages, traces and weights are compared with stored reference files. Spikes must match exactly; other values must be within 1e-6. Each network was checked against three deliberate wrong changes.test/regression/test_invariants.py: rules that must always hold. Running one long run equals running it in two halves; resetting a network equals building a fresh one; saving and loading gives the same network; turning learning off freezes the weights; the result does not depend on batch size; learning-rule steps scale correctly with the time step.test/network/test_all_classes.py: every neuron, connection, learning rule, feature, monitor and model class is found automatically and run for a few steps.test/regression/test_op_counts.py: counts the computing operations in one step of each learning rule, neuron type and the Diehl & Cook model. The test fails if the count grows, which catches slowdowns without measuring time.test/test_api_surface.py: a snapshot of the parameters and defaults of 443 public functions and classes.master. They are listed inKNOWN_BUGSand as strict xfails. Each one must be fixed (in 0.4.0) or the class deleted. When a bug is fixed, its test reports it, so a fix cannot go unnoticed.New checks on every pull request (
.github/workflows/test-guard.yml):bug-test: if a pull request changesbindsnet/, at least one test it adds or changes must fail onmasterand pass with the change. Maintainer override label:no-bug-test, for refactors and speed-ups.weakened-tests: lists removed tests, fewer checks, new skips or xfails, wider tolerances, changed reference files, and changedconftest.pyor pytest settings. Fails until a maintainer reviews the list and adds the labeltests-changed-ok.build(.github/scripts/coverage_floor.py:bindsnet/network90%,bindsnet/learning94%). These floors are provisional and will be re-measured from this pull request's CI run..github/CODEOWNERS: changes to the reference files, the guard tests and.github/need a maintainer's approval.bench.yml: an informational speed comparison against the base. It never fails.Each guard check was tested on deliberate cases (temporary commits, since deleted):
weakened-testscaught a removed test, a removed assert, a tolerance widened in a helper function and in a module constant, an added xfail, and an edited reference file. This pull request itself will failweakened-tests, because it adds the reference files. That is expected; it needs thetests-changed-oklabel.Checklist
poetry run pytestlocally and it passed.test/regression/,test/regression/op_counts.json,test/api_surface.json, andKNOWN_BUGSintest/network/test_all_classes.py. All are new, so nothing was regenerated.Still open after merge
KNOWN_BUGS: fix it in 0.4.0 or delete the class.🤖 Generated with Claude Code