Repository navigation
Add a research register for ideas, experiments and decisions - #139
Conversation
Records live as Markdown with typed YAML front matter under dokimi/research/records/<type>/<id>.md, validated by research/record.v1.json. scripts/research_register.py checks schemas, paths and cross-references, queries by field or reference, hands out the next id, and exports records, edges and a per-theme idea tree for the plans-and-research map. Negative and inconclusive outcomes are first-class; experiment outcome, decision state and implementation status are separate fields. Seeds 98 records from the project's public PRs/issues and local research notes; AGENTS.md names the register as the one exception to the no-status-reports rule. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #139 +/- ##
==========================================
+ Coverage 91.38% 91.40% +0.02%
==========================================
Files 216 216
Lines 26722 26741 +19
Branches 4373 4377 +4
==========================================
+ Hits 24419 24442 +23
+ Misses 1637 1635 -2
+ Partials 666 664 -2 🚀 New features to boost your workflow:
|
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 5dbb112. The tooling is small and does what the doc says: check passes on the real tree, the doc's example record validates, and the PR touches no package code. The problems are in the record content and in what check does not catch. Seven points are inline; I'd fix all but the superseded_by nit before merge.
What I ran, on a Linux review host at 5dbb112:
make ci-local: 20 passed, 0 failed, 4 skipped; 5322 tests passed, 1 skipped; the changed-lines coverage step passed.- Reproductions for the script findings, with a temporary register and no network.
- A hand check of the 98 records against their own result text, and of the doc's commands against the script.
Checked, no issues
- Docs:
research-register.mdis in themkdocs.ymlnav and linked fromdocs/README.md; the commands, flags and theintel//napetrov/link rule match the script and schema. - Repo rules: the Markdown records carry the SPDX header, the JSON schema is covered by the license ignore for JSON,
jsonschemaandPyYAMLare declared dependencies, and no generated file is committed. - Public content: a grep of the records for local paths, hosts and credentials found nothing.
- Counts: 21 ideas, 27 experiments (12 refuted, 12 supported, 3 inconclusive; none is
planned, unlike the description), 19 findings, 13 decisions, 18 implrefs.
…y, pin more register behaviour export, query and next read the register without checking it, so a record that check rejects crashed them with a traceback, and a nested reference crashed check itself. Every command now runs the schema and reference checks first and exits 1 with the messages. superseded_by is only valid on a superseded decision and cannot name the record itself. Tests cover --ref with --type, misplaced and misnamed files, id gaps, the broken-register path for each command, and unevaluatedProperties on every record type. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 7123967. The script fixes hold: query, next and export now refuse a broken register, a nested reference is reported as a schema error, and the new superseded_by checks and tests are pinned. The record content is where it still falls short. Four of my earlier threads are open, and I found four more points, inline.
What I ran, on a Linux review host at 7123967:
make ci-local: 20 passed, 0 failed, 4 skipped; the changed-lines coverage step reports no lines with coverage information in this diff.- Mutation run on
research_register.pyandrecord.v1.json: 26 script mutants and 5 schema mutants against the 28 register tests. The mutants that matter and survive are listed inline. - A hand check of the experiment outcomes and findings against their result text.
Still open from my earlier review:
dokimi/research/records/experiment/X006.md:8:outcome: supportedcontradicts the result text, the body does not say what the retraction covers, and F006 (high) and F010 still rest on retracted experiments.dokimi/research/records/experiment/X003.md:8: the outcome belongs to the enumeration variant, not to the hypothesis on line 7.dokimi/research/records/finding/F009.md:15: "kappa 0.576" is still not in X009, where 0.576 is an NLI AUC.dokimi/scripts/research_register.py:131: a finding that cites a retracted experiment still passescheck, andexportstill emits the retracted outcome.
Checked, no issues
- Refusal path: removing the early return, or writing the errors to stdout, fails the parametrized broken-register test for all three commands.
- New checks: the self-reference, state-mismatch, missing-target, wrong-type and non-string-reference mutants are all killed.
- Schema: flipping
unevaluatedPropertiesto true for each of the five record types fails a test. - Docs: the commands and flags in
research-register.mdmatch the script.
|
Pushed 7e152f4 for the four new points in the review at 7123967:
The four open threads (X006, X003, F009, and the retracted-experiment check in |
…field queries - X027: the hypothesis (a self-hosted tool enforces the schema) is refuted by the result text, so the outcome is refuted. - F016: drop the "~7% / lost accuracy" clause; no cited experiment has it. - query: exit 2 on a --where clause that is not FIELD=VALUE on a schema field, instead of printing nothing with exit 0. A missing field matches no value rather than the string "None". - tests: export ignores a decision that points at an idea; bad --where clauses and the missing-field case are pinned. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
bf5f52f to
7abf6f7
Compare
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…der rules - X006 refuted, partial retractions under Corrections; X012, X025 the same. F006 lowered to med. - X003 split: criterion wording only (refuted) and X028 criterion plus enumeration (supported). F002 and F003 cite both. - X029 holds the grader agreement F009 quotes; D004 rationale says where +0.007 comes from. X021 gains the compaction follow-up. - Outcome sweep: X002, X008, X021 inconclusive; X013, X016 refuted. - check fails a finding that cites a retracted experiment; export marks retracted experiments; query --ref on a missing id exits 2. - render writes the plans-and-research map (Mermaid mindmap and table) on demand; it is not committed or published. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
YAML reads retracted: true as a bool and str() spells it True, so the documented --where retracted=true matched nothing and exited 0. Query rows also showed a retracted experiment as plain supported. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
load() globbed */*.md, so a record directly under records/, one level too deep or named .markdown was never validated or exported, and check still printed OK. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
The url pattern checked only the prefix, so a valid prefix followed by a space and an internal URL, or by ../ or %2e%2e segments, passed check. Every one of the 37 links in the register still matches. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
The stamp was git rev-parse HEAD of the script's checkout, so a map rendered from uncommitted records named a commit with other contents. Read the commit from the --root tree and add a dirty flag, as the run manifest does. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Redirected stdout uses the locale encoding, so on Windows export wrote F009's ± as byte 0xB1 (not valid UTF-8 JSON), and a title outside cp1252 made export, query and render exit 1 with UnicodeEncodeError. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…nd render No fixture experiment was retracted and no title had brackets or a pipe, so dropping the export flag, the render mark, _label or the table escape left every test green. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at ccc9fa2. All eight of my earlier threads are fixed: the outcomes now match the result text, the retracted-experiment check works and is tested, and the query typo cases exit 2. What is left is a crash in the new retracted check, a broken render, findings and decisions whose text is not in the experiments they cite, and untested reference checks; three nits. Ten points: eight inline, two below.
What I ran, on a Linux review host at ccc9fa2:
make ci-local: 26 passed, 0 failed, 4 skipped; the changed-lines coverage step passes with no measurable lines in this diff.checkon the real tree:OK: 100 records. 71 mutants onresearch_register.pyandrecord.v1.jsonagainst the 52 register tests: 22 survive, the ones that matter are inline.- Reproductions for the script findings, with temporary registers and no network.
- A hand check of every experiment outcome, finding and decision against the records it cites.
Findings that cannot be anchored inline:
- Nit:
exportandrendercrash withFileNotFoundErrorwhengitis not installed._git(research_register.py:189-191) callssubprocess.run(["git", ...]), andcheck=Falsedoes not cover a missing binary. WithPATHemptied,export --root <valid register>raisesFileNotFoundError: ... 'git'. A register outside a git repo is fine (commit: null). Suggestion: catchOSErrorin_gitand return"". F003quotes a range that its own experiments contradict.finding/F003.md:17says wrong_numbers "stayed 0.67-0.82 across every judge configuration measured". F003 cites X002, X003 and X028, and X028's baseline is 0.599. X002 has no wrong_numbers figure, so "wrong names are caught" has no cited support. The 0.47-0.72 range is in X008, which F003 does not cite. Suggestion: cite X008, and restate the range from the cited figures.
I agree with @xaleryb's points on the id clash after next and on the stale counts in the PR description.
Checked, no issues
- Outcomes: X001, X003, X006, X007, X009-X011, X013-X019, X022, X023, X027 and X029 match their hypothesis against their result text; X003/X028 and X006's
## Correctionshold. - Docs build:
mkdocs build --strictpasses; the page is in thenavand linked fromdocs/README.md; the commands and flags match the script. - Repo rules:
record.v1.jsonis outsidedokimi/schemas/, soSCHEMA_VERSIONSdoes not apply; no changelog fragment is needed; SPDX headers are on every record;gitleaksand a grep for local paths and non-GitHub hosts find nothing. - Script: the path and extension check, the UTF-8 reconfigure, the
retractedmark inquery,exportandrender, and the--refand--whereexits are all killed by a mutant.
| if record.fields.get("superseded_by") == rid: | ||
| errors.append(f"{rid}: superseded_by names the record itself") | ||
| for target in ref_values(record, "experiments"): | ||
| if target in records and records[target].fields.get("retracted"): |
There was a problem hiding this comment.
A non-string entry in experiments crashes check with a traceback.
Line 142 skips non-string targets, but this loop does target in records without that guard. With finding/F001.md set to experiments: [[X001]], check raises TypeError: unhashable type: 'list'. experiments: [{a: 1}] and experiments: {a: 1} do the same. query, export and render run check first, so they crash too. The schema error that would explain it is never printed. A scalar experiments: X001 is handled ('X001' is not of type 'array'). The nested-reference case was fixed for REFS fields in the last round; this loop is the new retracted check.
Suggestion: reuse the loop's guard, for target in ref_values(record, "experiments"): if isinstance(target, str) and target in records and ..., and add a test_bad_record_fails row with experiments: [[X001]].
There was a problem hiding this comment.
Fixed in 93452fd: a non-string entry in experiments is now a check error ("is not a record id") instead of a TypeError; tests cover a list and a mapping entry.
| continue | ||
| lines.append(f" {theme['id']}[{_label(theme['title'])}]") | ||
| for idea in theme["ideas"]: | ||
| lines.append(f" {idea['id']}[{idea['id']} {_label(idea['title'])} - {idea['status']}]") |
There was a problem hiding this comment.
A title with a newline breaks the Mermaid mindmap and the table.
_label strips brackets but not newlines, and the schema's text type is only minLength: 1. A folded title (title: > followed by two lines keeps a trailing newline) renders as I001[I001 Folded title over lines\n - open], so the status suffix lands on its own line and the diagram does not parse. title: "multi\nline [x] - done" does the same. The table rows at :250 and :253 only replace |, so they split too.
Suggestion: collapse whitespace in _label and in the table cells with " ".join(text.split()), and add a render test with a multi-line title.
There was a problem hiding this comment.
Fixed in 93452fd: _label and the table title collapse all whitespace, newlines included; a test renders a folded title.
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
| ## Statement | ||
| When both arms pass every paired task, compare must flag no headroom. |
There was a problem hiding this comment.
F018 (confidence: high) states something its only experiment does not show.
The statement is "When both arms pass every paired task, compare must flag no headroom." The only cited record, X024, is a regrade of 1414 answers whose whole result is "0 mismatches." (experiment/X024.md:20). It says nothing about ceilings or headroom. D009 and R022 rest on F018.
Suggestion: cite the experiment that found the zero-delta ceiling case, or add it as a record, or lower the confidence and say it is a design rule.
There was a problem hiding this comment.
Fixed in 93452fd: F018 confidence is now low and the body says X024 is its only source (a regrade with 0 mismatches, not a test of the no-headroom flag).
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
| ## Rationale | ||
| The five dimensions behaved as one factor; averaging diluted correctness. Done in pre-public history. |
There was a problem hiding this comment.
D002 cites F002, which has nothing to do with dimension weights.
D002 is "weight correctness x2; zero-weight specificity" and its rationale is "The five dimensions behaved as one factor; averaging diluted correctness." Its only finding, F002, is about enumerating figures to catch wrong numbers (X003, X028). "Behaved as one factor" appears in no experiment in the register, so a reader following the link finds no support.
Suggestion: add the experiment and finding behind the weighting, or say in the rationale that it comes from pre-public history without a record.
There was a problem hiding this comment.
Fixed in 93452fd: D002 rationale is softened. It says F002 is only judge-prompt context and the weights come from pre-public history.
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
| ## Rationale | ||
| Measured gains negligible or refuted: stratified probe sampling moved agreement by +0.007, inside the grader spread F009 reports, so it does not pay for its probe budget; the LLMLingua shortener was refuted (X009), and the SliCK split showed no gain worth building either. |
There was a problem hiding this comment.
The numbers in D004's rationale are in no record.
"Stratified probe sampling moved agreement by +0.007" and "the SliCK split showed no gain" appear nowhere else: grep -rn "0.007\|SliCK" research/records returns only this file. F009 cites X009 and X029, which cover the LLMLingua shortener and grader agreement. The earlier thread on F009 is fixed; this is the figure that replaced it.
Suggestion: add an experiment for the sampler and the SliCK split and cite it from F009, or drop both claims from the rationale.
There was a problem hiding this comment.
Fixed in 93452fd: the uncited +0.007 and SliCK numbers are removed; D004 says they come from unrecorded local runs.
| "names the record itself"), | ||
| ("idea", "I002", "title: T\ntheme: O\nquestion: Q\nstatus: open\nextra: 1\n", | ||
| "Unevaluated properties"), | ||
| ("finding", "F002", "title: T\nexperiments: [X001]\nconfidence: low\nextra: 1\n", |
There was a problem hiding this comment.
The reference checks for finding, decision and implref are not pinned.
Mutation check: emptying the finding, decision.superseded_by, decision.ideas or implref entries of REFS passes all 52 tests. Only the experiment-to-idea edge has a missing-target and wrong-type case here, and finding.experiments is covered only by the retracted check. Other mutants that survive: dropping the -- . pathspec in the dirty check, _label on the theme title, the pipe escape on an idea title, and sorted on the idea and experiment lists.
Suggestion: add test_bad_record_fails rows for finding.experiments: [X999], implref.decision: F001, decision.findings: [X001] and decision.ideas: [X001], each with a no record or want message.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 999ecf3, which fix the three nits from my review:
308d6fd:dokimi/scripts/research_register.py:155:checknow followssuperseded_byand reports{rid}: superseded_by chain loops back to {rid}; a two-decision cycle test is added.4ac9f70:dokimi/docs/research-register.md:34: the page now says optionallinks(required onimplref).999ecf3:dokimi/scripts/research_register.py:197:_gitcatchesOSErrorand returns an empty string, soexportandrenderno longer crash without git; a test is added.
make ci-local at 999ecf3: 26 passed, 0 failed, 4 skipped.
Not approving yet. Still open: 1 bug finding(s) from my review; 6 risk finding(s) from my review.
|
CI at |
Raise golden_manifest max_docs to 31 for the 31 docs pages. Report a non-string experiments entry as a check error instead of a TypeError, flatten line breaks in titles for the Mermaid map, pin reference checks for finding, decision and implref, lower F018 confidence, soften D002 and D004 rationales, mark R017 and R023 merged, and document renumbering. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
@exai-bot /review |
exai-bot
left a comment
exai-bot
left a comment
There was a problem hiding this comment.
Note
AI-generated review of 44a5a20f46ca, requested by @vchamarthi, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 9 of 95 turns, 0.9 of 48 minutes, 214k in / 5k out tokens.
This adds a file-based research register under dokimi/research/: a JSON Schema, 100 Markdown records, a check/query/next/export/render script, tests and a docs page. I found no correctness or security defects in the script or schema. The reference, supersession, retraction, link and path checks look sound, and the tests cover the failure paths well. The remaining findings are small inconsistencies in the docs and the PR description.
3 findings are on the changed lines, as comments below.
For a human reviewer
- Whether maintainers accept the AGENTS.md exception to the rule against status reports and ADRs in the tree, or want the register in a separate repo
- Domain accuracy of the experiment outcomes, finding confidence levels and decisions in each area owner's records
Not verified
I could not run the tests or the script, so I did not confirm that check passes on the real tree. CI reports that the test suite passes. I also did not verify the factual content of the 100 records against their sources.
| - `next` prints the next free id for a type. Two PRs can claim the same id; `check` reports | ||
| the duplicate on whichever merges second, which renumbers. To renumber: take the new id | ||
| from `next`, rename the file and change its `id:`, then rewrite every reference to the old | ||
| id in that PR's own records (`idea`, `experiments`, `findings`, `ideas`, `superseded_by`, |
There was a problem hiding this comment.
Nit (confirmed, maintainability): Duplicate-id renumbering advice describes a case that cannot occur between two PRs
check requires a record's file name to equal its id, so two PRs that claim the same id add the same path. Git reports an add/add conflict at merge time rather than letting check report a duplicate on whichever PR merges second. The renumbering steps are still useful, but the explanation of when they are needed is misleading. Say that the second PR hits a merge conflict and renumbers then.
There was a problem hiding this comment.
Fixed in a139cd7: the doc now says both PRs add the same file, so the second to merge gets an add/add conflict (check never sees a duplicate) and renumbers then.
| `[tool.setuptools.package-data]`, or the installed wheel goes without it. | ||
| - Design rationale, status and plans go in the PR description or an issue; | ||
| the tree has no ADRs or status reports. | ||
| the tree has no ADRs or status reports. The one exception is the research |
There was a problem hiding this comment.
Nit (confirmed, maintainability): AGENTS.md exception omits implementation references and ideas
The new sentence lists "one record per idea, experiment, finding or decision". The register also holds implref records, which link to PRs and issues. Without the implref type, the exception reads narrower than what the tree contains.
There was a problem hiding this comment.
Fixed in a139cd7: the AGENTS.md exception now reads "idea, experiment, finding, decision or implementation link"; a test pins it.
| idea: I017 | ||
| hypothesis: Generic L0 rules match the product's static check. | ||
| outcome: supported | ||
| links: |
There was a problem hiding this comment.
Nit (plausible, correctness): Outcome supported sits oddly with the recorded result
The hypothesis is that generic L0 rules match the product's static check. The result is product 6/7 against dokimi 5/7 after the change, which looks closer to inconclusive. Consider re-checking this outcome. The retracted flag is not used anywhere; the corrections here and in X012 are written as body text only, as the docs describe.
The PR description's counts (27 experiments, R001..R018) also do not match the tree. The tree has 29 experiments (X001..X029), and the implref ids are non-contiguous, R001 to R030 with gaps.
There was a problem hiding this comment.
Fixed in a139cd7: X025 is now inconclusive (product 6/7 vs dokimi 5/7 after the change). On retracted: the schema field is optional and the record text carries the correction; no change. PR body counts are updated to match the tree.
…md, X025 inconclusive Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
@antonwolfy thanks for 308d6fd, 4ac9f70 and 999ecf3, all kept. The open items from your ccc9fa2 review:
|
…ents measured The 0.67-0.82 range and 'wrong names are caught' had no support in X002, X003 or X028. Cite X008 (0.47-0.72 across wordings) and X028 (0.599 -> 0.712, under the 0.80 gate) instead. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ve GitHub state - render --html writes one self-contained page (no network, no library): collapsible theme > idea > experiment > finding > decision > implref tree, search, filters by type, theme, outcome/state and implementation, a sortable table and a record panel with back-references. - gaps lists accepted decisions with nothing merged and pending implrefs; --live asks GitHub through gh for each implref link, reports recorded status that differs, and lists open issues no record links to. - R024 and R029 are merged on GitHub; their status says so now. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…gineering issues - R031-R035: journey sub-issues intel#96-intel#100 under D011. - D014/R036 catalog integration (intel#138), D015/R037 CLI restructure deferred (intel#94), D016/R038 a run records only the conditions it enforced (intel#148). - R039 links D012 to this PR; R040 links D002 to the judge aggregate code. - D001 is rejected like D004: a 'do not ship' decision is not work to build. - research/untracked.yaml lists open issues that are engineering, not research (intel#19, intel#30, intel#61, intel#136); gaps --live skips them and reports an entry whose issue closed. A proposed implref on an open issue is not drift. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
@exai-bot /re-review |
exai-bot
left a comment
exai-bot
left a comment
There was a problem hiding this comment.
Note
AI-generated review of c874618d2573, requested by @vchamarthi, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 9 of 96 turns, 0.9 of 48 minutes, 257k in / 6k out tokens.
Re-review of c874618d2573: 4 new findings.
Findings
1. Minor (plausible, correctness): HTML payload escaping misses <!-- + <script in a record body
dokimi/scripts/research_register.py
render_htmlonly rewrites</to<\/.- In the HTML tokenizer, a JSON
<script>block still follows the script-data escape rules. A record body (rendered into the payload) that contains<!--and later<scriptputs the parser in the double-escaped state. The real closing</script>of the data block then no longer ends it, and the page breaks. - Bodies are repo-reviewed and the test only covers
</script>, so this is low risk. The fix is cheap.
Suggested fix:
Also replace <!-- with <\u0021-- (or escape every < as \u003c) in the JSON payload, and add a test with <!--<script> in a body.
2. Minor (confirmed, correctness): render --live without --html is silently ignored
dokimi/scripts/research_register.py
--live is accepted by the render subparser, but the Markdown path (render(export(...))) never reads args.live. A user running render --live > map.md believes the map reflects GitHub state, but it doesn't use it.
Suggested fix:
Make --live an error without --html, or move it into a mutually-required group with --html.
3. Minor (confirmed, correctness): --live makes one sequential gh api call per link, with a 60 s timeout each
dokimi/scripts/research_register.py
implementation(live=True) calls github_state for every link of every implref, one after another. There are about 40 implrefs, so it is fine today. If gh hangs, each call can stall for up to 60 s, so a run could take tens of minutes. If gh is missing or unauthenticated, every call returns None and the output quietly reports 0 drift, which looks like success. The script does not report that live data was unavailable.
Suggested fix:
Print a warning when --live produced no state for any link. Optionally cache per URL or lower the timeout.
4. Nit (plausible, style): PR description counts are out of date
dokimi/research/records/decision/D014.md
The description says decisions are D001..D013 and implrefs number 18, but the tree has D001..D016 and R001..R040. It also says check prints OK: 100 records. Update the description so reviewers checking claims by area see the true scope, including the three later decisions and the new implrefs.
|
@exai-bot re-review of c874618 (#139 (review)), finding by finding:
Ready for another look. |
…ve without --html, warn on empty live state Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
…r raise Main grew four baselined or near-cap items and added three functions over complexity 10 before the gate landed: test_repo_environment.py 1369 -> 1371 (intel#147), manifest._check_task 21 -> 22 (intel#113), ci_local.run 28 -> 29 (intel#146), cmd_repo_validate 10 -> 11 (intel#150), cli_surface._value 17 (intel#88), research_register.check 19 and main 15 (intel#139). Each entry names its PR. --update after the seed lowers nothing. Now 15 files and 98 functions. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
* ci: no-growth ratchet for file length and function complexity A first-party .py file over 800 lines (1000 under tests/) or a function over cyclomatic complexity 10 (ruff's C901 count, computed with ast so the test job needs no ruff) now fails tests/test_growth_ratchet.py unless scripts/growth_baseline.json records it. A recorded value is exact: the code may shrink but not grow, and a shrunk, gone or back-under-cap entry must be lowered or removed (--update does that, never raising). An entry raised or added relative to the base branch needs a rationale of at most 300 characters. The baseline holds the 14 files and 81 functions over the caps today. `check_growth.py --report --markdown` adds the totals to the static job's step summary, non-blocking. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(ci_local): pass resolved base ref to growth check via DOKIMI_GROWTH_BASE ci_local.py resolves the base ref to handle non-origin remotes, but did not pass it to the pytest environment. This caused the growth check to use the wrong baseline or skip the rationale check if the baseline was missing, allowing misleading local CI passes when using --remote with a non-origin remote. Pass the runner's resolved base_ref as DOKIMI_GROWTH_BASE environment variable to the subprocess steps. Add a regression test that verifies the check correctly uses a non-origin remote's baseline. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): baseline evaluate_novelty at 26, as #72 left it on main Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(ci-local): pass resolved base_ref as DOKIMI_GROWTH_BASE env var ci_local.py now passes the resolved base reference to workflow steps as DOKIMI_GROWTH_BASE, allowing check_growth.py to use it without requiring --base argument when ci_local is used to run workflows locally. Adds test to verify the env var is passed to all subprocess steps. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix: restore the files f105136 reverted by mistake f105136 was built on a stale tree and deleted the #72 and #84 changes from this branch. Bring every file back to 6c4cc20; only the test_ci_local.py addition of f105136 stays. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(growth): a raised entry needs a new rationale; test loop/try else and match guards Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(ci_local, check_growth): propagate DOKIMI_GROWTH_BASE to isolated jobs and guard TryStar Two fixes for growth ratchet compatibility: 1. Isolated jobs (package-smoke) did not receive DOKIMI_GROWTH_BASE from base_env, causing the growth check to compare against the wrong baseline. Now included in env. 2. ast.TryStar is only available on Python 3.11+ (introduced with except*). Guarded with hasattr() so the script remains compatible with Python 3.10. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(ci_local, check_growth): cover isolated-job env and drop dead TryStar guard Add a regression test for DOKIMI_GROWTH_BASE reaching the isolated package-smoke job, and a try-star complexity case. ast.TryStar always exists on the supported Python (>=3.11), so the hasattr guard is removed. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * refactor: split functions over the complexity cap Extract helpers from cmd_answers_generate (runtime config, question loading, output path) and from the import-architecture test (import targets, owner module, Tarjan component pop, spawn-import aliases) so each is under the cap. cmd_answers_generate is back under its cap, so its baseline entry is removed. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(answers): cover the helpers answers generate was split into The refactor moved config loading, question loading and output-path choice into helpers that the existing tests never reach, leaving 12 changed lines uncovered (diff-cover gate: 61%, needs 80%). The new tests cover the missing-, empty-, invalid- and non-mapping config cases, the question load error, the output-path precedence, and the command end to end over fake Answerer and documentation-source clients. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): take the #101 growth into the baseline; changelog fragment Main's #101 grew dokimi/novelty/bundle.py to 1016 lines and evaluate_novelty to complexity 29 after the baseline was cut, so the ratchet failed on code this PR does not touch. Raise both entries with a rationale, as the #72 entry already does. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(growth): compare rationale against the fork point, not the base tip The rationale check read the baseline at the tip of origin/<base>. Once main lowers an entry, every branch behind it, and a re-run CI merge commit, was told it raised that entry. Read it at git merge-base HEAD <ref> instead. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(growth): pin the C901 count and guards that no test reached Eleven single-line mutants of check_growth.py passed the whole suite: defs in except/case bodies, async for, nested async defs, class bodies, `(_ as z)`, duplicate names, GITHUB_BASE_REF, cli.py, dot dirs, non-int entries and the rationale strip. Expected counts come from ruff C901 at max-complexity 0. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(growth,answers): pin the match-case counts, the subdirectory baseline read, and the top_k default Three review gaps with no failing test: a decision inside a case body and a refutable as-pattern last case in the complexity count; load_base_baseline when the project sits in a subdirectory of the git root (the ./ prefix); and the answers command's top_k default from retrieval.top_k when the flag is unset. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(answers): make the flag row differ from config and default; fix growth comments The flag row used top_k=3, which is also the built-in default, so a command ignoring --top-k still passed (mutant: if True:). Use 5 with a config of 7. Correct the test and function counts in check_growth.py comments and drop the unused _product_config assignment in answers.py. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): --seed mode, reseed baseline after merging main, document --update after rebase Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(growth): the --seed rationale does not explain a raise against a baselined base After this lands, `--seed` on a branch that grew llm.py wrote the entry at the new size with "pre-existing when the gate was introduced", and the gate passed. The rationale check now treats that text as missing, so only the one-time seeding (base with no baseline) can use it. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * ci(growth): reseed the baseline after the merge of main (main has no baseline) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): seed the baseline after merging main, with a rationale per raise Main grew four baselined or near-cap items and added three functions over complexity 10 before the gate landed: test_repo_environment.py 1369 -> 1371 (#147), manifest._check_task 21 -> 22 (#113), ci_local.run 28 -> 29 (#146), cmd_repo_validate 10 -> 11 (#150), cli_surface._value 17 (#88), research_register.check 19 and main 15 (#139). Each entry names its PR. --update after the seed lowers nothing. Now 15 files and 98 functions. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * ci(growth): re-seed the baseline after merging main again (#73, #115) #73 grew test_judge_validity.py 1423 -> 2235, test_judge_reliability.py 1505 -> 1695, model_compare.py 1296 -> 1297, cmd_eval_judge_validity 25 -> 29, and took degrade.build_pairs (17) and judge_sequential.run_sequential (12) over the cap. #115 grew l0 _build_names 14 -> 15 and shrank _cmake (now under the cap, entry dropped) and Scan.resolve_path (20 -> 19). Each raise names its PR. Now 15 files and 99 functions. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com>
Summary
Adds a register in the tree for Dokimi's own research: ideas, experiments (negative ones included), findings, decisions and implementation links. Each record is plain Markdown that agents and people can both read and write. It also produces the data for the plans-and-research map.
How the record types point at each other. The three states in brackets are separate fields:
exportwrites every record, the reference edges, and ideas grouped by map theme (P/E/V/K/G/O/C/D/W) with each idea's experiment outcomes.Why plain files instead of a server: I set up XWiki, OpenProject CE, eLabFTW, GitLab CE and git+Markdown locally and ran the same 8 tests on each. Only git+Markdown passed all 8, with no server and real history. None of the server tools rejected a reference to a missing record or a wrong enum value on an API write, so a validator would be needed anyway.
Evidence
After:
tests/test_research_register.py(69 tests) runscheckon the real tree. It also covers broken records that must fail: a reference to a missing record, a reference to a record of the wrong type, a bad enum value, an unknown field, a superseded decision withoutsuperseded_by, a wrong id prefix, a non-public link, and a duplicate id.make ci-local --offline: everything passes (network and docker jobs are skipped). The license header check passes on the new files.Merge Danger
Door: two-way
Data and a standalone script only. Nothing is added to the package, so no changelog fragment is needed. Revert is a clean delete.
Blast Radius: docs
AGENTS.mdcurrently says "the tree has no ADRs or status reports". This PR adds one named exception. If maintainers would rather keep that rule strict, the register can move to a separate repo in the same format.github.com/intel|napetrovURLs (the schema enforces this).🤖 Generated with Claude Code
Map UI and implementation tracking (64bb550, c874618)
render --html: one self-contained page (no network, no JS library): collapsible theme > idea > experiment > finding > decision > implref tree, search, filters, sortable table, record panel with back-references. Not committed and not on the docs site; generated on demand.gaps [--live]: accepted decisions with nothing merged, pending implrefs; with--live, implref status drift against GitHub and open issues no record links to.checkstays offline.intel/dokimiissue is now either linked from a record (R031-R038) or listed with a reason inresearch/untracked.yaml.gaps --liveat c874618: 0 drift, 0 unlinked issues; D005 is the one accepted decision with no implementation.