Repository navigation
test(repo-environment): give the hostile-string regex test a CI-safe time limit - #147
Conversation
…time limit
The tsc case ("tsc " * 100_000) takes ~0.4 s on a workstation but 2.1-2.4 s on a
loaded CI runner, failing the 2 s limit on main. Without its {0,256} bound the
pattern is quadratic (~150 s at this size), so a 15 s limit still catches a
regression.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #147 +/- ##
=======================================
Coverage 91.38% 91.38%
=======================================
Files 216 216
Lines 26722 26722
Branches 4373 4373
=======================================
+ Hits 24419 24421 +2
+ Misses 1637 1635 -2
Partials 666 666 🚀 New features to boost your workflow:
|
|
@exai-bot /review |
exai-bot
left a comment
There was a problem hiding this comment.
Note
AI-generated review of bc4640dd8a06, requested by @vchamarthi, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 9 of 93 turns, 0.5 of 48 minutes, 230k in / 3k out tokens.
The change raises the time limit in test_script_and_manifest_patterns_are_linear_on_hostile_strings from 2 s to 15 s, because the "tsc " * 100_000 case sat close to 2 s on CI runners. Your own numbers back the choice: about 0.3 s on a workstation and roughly 5–6× that on CI with the bound, against about 150 s without it. That leaves a wide margin on both sides, so the flake fix is sound. I have one minor point about the other cases in the same test.
1 finding is on the changed lines, as comments below.
| assert time.monotonic() - start < 2 | ||
| # Bounded, the `tsc` case takes ~0.4 s on a workstation and over 2 s on a loaded CI runner; unbounded, | ||
| # its pattern is quadratic (~150 s), so the limit leaves room for slow runners. | ||
| assert time.monotonic() - start < 15 |
There was a problem hiding this comment.
Minor (plausible, tests): 15 s limit is applied to every case, including the cheap ones
The 15 s limit applies to all four parametrized cases, but only the tsc case needs it. The rm --a×28 and git clean --a×28 cases are tiny inputs, and a regression there would show up as backtracking blow-up. That blow-up is less certain to be caught under a 15 s limit than under 2 s: a polynomial regression on a 28-token input could finish in under 15 s and pass. This is unconfirmed, because I could not run the patterns. Giving the tsc case its own limit would keep the other cases strict.
Suggested fix:
Add a limit parameter to the parametrization (for example (files, 15) for the tsc case and (files, 2) for the others), or move the tsc case into its own test with the 15 s limit.
napetrov
left a comment
There was a problem hiding this comment.
Fine as an unblock: the analysis (linear vs quadratic, ~150 s unbounded) shows 15 s still catches the regression.
Follow-up suggestion: an absolute wall-clock limit stays runner-dependent. Timing n and 2n and asserting the ratio < ~3 (linear ~2, quadratic ~4) is machine-independent. test_manifest_readers_are_linear_on_repeated_tokens and test_makefile_rules_are_linear have the same 2 s limit and are next to flake; worth converting all three together.
…r raise Main grew four baselined or near-cap items and added three functions over complexity 10 before the gate landed: test_repo_environment.py 1369 -> 1371 (intel#147), manifest._check_task 21 -> 22 (intel#113), ci_local.run 28 -> 29 (intel#146), cmd_repo_validate 10 -> 11 (intel#150), cli_surface._value 17 (intel#88), research_register.check 19 and main 15 (intel#139). Each entry names its PR. --update after the seed lowers nothing. Now 15 files and 98 functions. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
* ci: no-growth ratchet for file length and function complexity A first-party .py file over 800 lines (1000 under tests/) or a function over cyclomatic complexity 10 (ruff's C901 count, computed with ast so the test job needs no ruff) now fails tests/test_growth_ratchet.py unless scripts/growth_baseline.json records it. A recorded value is exact: the code may shrink but not grow, and a shrunk, gone or back-under-cap entry must be lowered or removed (--update does that, never raising). An entry raised or added relative to the base branch needs a rationale of at most 300 characters. The baseline holds the 14 files and 81 functions over the caps today. `check_growth.py --report --markdown` adds the totals to the static job's step summary, non-blocking. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(ci_local): pass resolved base ref to growth check via DOKIMI_GROWTH_BASE ci_local.py resolves the base ref to handle non-origin remotes, but did not pass it to the pytest environment. This caused the growth check to use the wrong baseline or skip the rationale check if the baseline was missing, allowing misleading local CI passes when using --remote with a non-origin remote. Pass the runner's resolved base_ref as DOKIMI_GROWTH_BASE environment variable to the subprocess steps. Add a regression test that verifies the check correctly uses a non-origin remote's baseline. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): baseline evaluate_novelty at 26, as #72 left it on main Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(ci-local): pass resolved base_ref as DOKIMI_GROWTH_BASE env var ci_local.py now passes the resolved base reference to workflow steps as DOKIMI_GROWTH_BASE, allowing check_growth.py to use it without requiring --base argument when ci_local is used to run workflows locally. Adds test to verify the env var is passed to all subprocess steps. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix: restore the files f105136 reverted by mistake f105136 was built on a stale tree and deleted the #72 and #84 changes from this branch. Bring every file back to 6c4cc20; only the test_ci_local.py addition of f105136 stays. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(growth): a raised entry needs a new rationale; test loop/try else and match guards Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(ci_local, check_growth): propagate DOKIMI_GROWTH_BASE to isolated jobs and guard TryStar Two fixes for growth ratchet compatibility: 1. Isolated jobs (package-smoke) did not receive DOKIMI_GROWTH_BASE from base_env, causing the growth check to compare against the wrong baseline. Now included in env. 2. ast.TryStar is only available on Python 3.11+ (introduced with except*). Guarded with hasattr() so the script remains compatible with Python 3.10. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(ci_local, check_growth): cover isolated-job env and drop dead TryStar guard Add a regression test for DOKIMI_GROWTH_BASE reaching the isolated package-smoke job, and a try-star complexity case. ast.TryStar always exists on the supported Python (>=3.11), so the hasattr guard is removed. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * refactor: split functions over the complexity cap Extract helpers from cmd_answers_generate (runtime config, question loading, output path) and from the import-architecture test (import targets, owner module, Tarjan component pop, spawn-import aliases) so each is under the cap. cmd_answers_generate is back under its cap, so its baseline entry is removed. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(answers): cover the helpers answers generate was split into The refactor moved config loading, question loading and output-path choice into helpers that the existing tests never reach, leaving 12 changed lines uncovered (diff-cover gate: 61%, needs 80%). The new tests cover the missing-, empty-, invalid- and non-mapping config cases, the question load error, the output-path precedence, and the command end to end over fake Answerer and documentation-source clients. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): take the #101 growth into the baseline; changelog fragment Main's #101 grew dokimi/novelty/bundle.py to 1016 lines and evaluate_novelty to complexity 29 after the baseline was cut, so the ratchet failed on code this PR does not touch. Raise both entries with a rationale, as the #72 entry already does. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(growth): compare rationale against the fork point, not the base tip The rationale check read the baseline at the tip of origin/<base>. Once main lowers an entry, every branch behind it, and a re-run CI merge commit, was told it raised that entry. Read it at git merge-base HEAD <ref> instead. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(growth): pin the C901 count and guards that no test reached Eleven single-line mutants of check_growth.py passed the whole suite: defs in except/case bodies, async for, nested async defs, class bodies, `(_ as z)`, duplicate names, GITHUB_BASE_REF, cli.py, dot dirs, non-int entries and the rationale strip. Expected counts come from ruff C901 at max-complexity 0. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(growth,answers): pin the match-case counts, the subdirectory baseline read, and the top_k default Three review gaps with no failing test: a decision inside a case body and a refutable as-pattern last case in the complexity count; load_base_baseline when the project sits in a subdirectory of the git root (the ./ prefix); and the answers command's top_k default from retrieval.top_k when the flag is unset. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(answers): make the flag row differ from config and default; fix growth comments The flag row used top_k=3, which is also the built-in default, so a command ignoring --top-k still passed (mutant: if True:). Use 5 with a config of 7. Correct the test and function counts in check_growth.py comments and drop the unused _product_config assignment in answers.py. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): --seed mode, reseed baseline after merging main, document --update after rebase Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(growth): the --seed rationale does not explain a raise against a baselined base After this lands, `--seed` on a branch that grew llm.py wrote the entry at the new size with "pre-existing when the gate was introduced", and the gate passed. The rationale check now treats that text as missing, so only the one-time seeding (base with no baseline) can use it. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * ci(growth): reseed the baseline after the merge of main (main has no baseline) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): seed the baseline after merging main, with a rationale per raise Main grew four baselined or near-cap items and added three functions over complexity 10 before the gate landed: test_repo_environment.py 1369 -> 1371 (#147), manifest._check_task 21 -> 22 (#113), ci_local.run 28 -> 29 (#146), cmd_repo_validate 10 -> 11 (#150), cli_surface._value 17 (#88), research_register.check 19 and main 15 (#139). Each entry names its PR. --update after the seed lowers nothing. Now 15 files and 98 functions. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): re-seed the baseline after merging main again (#73, #115) #73 grew test_judge_validity.py 1423 -> 2235, test_judge_reliability.py 1505 -> 1695, model_compare.py 1296 -> 1297, cmd_eval_judge_validity 25 -> 29, and took degrade.build_pairs (17) and judge_sequential.run_sequential (12) over the cap. #115 grew l0 _build_names 14 -> 15 and shrank _cmake (now under the cap, entry dropped) and Scan.resolve_path (20 -> 19). Each raise names its PR. Now 15 files and 99 functions. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com>
test_script_and_manifest_patterns_are_linear_on_hostile_strings[files2]failed onmainin the py3.11 and py3.12 jobs (run):inspect_repotook 2.12 s and 2.36 s against a 2 s limit. The code under test hasn't changed since #66.The case is a
package.jsonscript of"tsc " * 100_000, which guards the{0,256}bound in_NODE_TYPE_TOOL(dokimi/repos/environment/checks.py).This PR raises the limit to 15 s, which still catches that regression by a wide margin. The test's other three cases are much cheaper.
test_manifest_readers_are_linear_on_repeated_tokensandtest_makefile_rules_are_linearuse the same 2 s limit. They have not failed, so this PR leaves them unchanged.