Repository navigation
test, docs: leftovers from the reviews of #43, #44, #59, #69 and #75 - #87
Merged
Merged
Conversation
intel#69 and intel#75 Points from those reviews that were neither changed nor answered, and that change no package code. Tests: - intel#43: questions refine reads its input without validating it; a set whose duplicate record has `"topics": "perf"` is repaired and written. Reading it strictly instead passed the suite. - intel#44: shorten turns a bundle block it cannot slice into an exit that names the atoms file, not a traceback; an entry-point record without a stored outside_share reports the share its counts give (83%). - intel#69: the cross-judge gate refuses one damaged question among eight with int ids (a gate that read str ids only passed); from_artifact skips a non-dict arm next to a served one; unreproducible_arms skips records, arms, evaluations and scores of the wrong type. Docs: - intel#59: README says to add --all-atoms to the tier command and to run --estimate-only first, and names the credential variables repo reach passes on. - intel#69, intel#75: re-flow the long lines in evaluating-treatments.md, the runner comment in ci.yml and the coverage bullet in CONTRIBUTING.md. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
requested review from
jharlow-intel,
napetrov,
vchamarthi and
xaleryb
as code owners
October 17, 2026 12:57
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #87 +/- ##
==========================================
+ Coverage 90.06% 90.14% +0.07%
==========================================
Files 205 205
Lines 24193 24193
Branches 3892 3892
==========================================
+ Hits 21790 21808 +18
+ Misses 1724 1713 -11
+ Partials 679 672 -7 🚀 New features to boost your workflow:
|
napetrov
approved these changes
Oct 17, 2026
napetrov
added a commit
to napetrov/dokimi
that referenced
this pull request
Oct 18, 2026
Resolve conflicts with intel#81 (empty-question rule): the judge paths keep the record objects and call require_question_texts / require_question_text on the parsed records before the first judge call; the duplicate RecordError question check in Judge.evaluate_answers is replaced by intel#81's EmptyQuestionError (exit 2). checklist_jobs, pairwise_jobs and scored_answers read records. A record without question_id stays a RecordError (intel#80) in the degradation pairs; unreproducible_arms keeps intel#87's skip of malformed records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RW8mAicb3ibXdzHMV83wDn Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Merged
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Points from the reviews of #43, #44, #59, #69 and #75 that were neither changed nor answered before those PRs merged. Only tests and text change here, no package code.
Changes
Tests
questions refinereads its input without validating it. A set whose duplicate record has"topics": "perf", which strict reading refuses, is repaired and written. Reading it strictly instead passed the suite.shortenturns a bundle block it cannot slice (a file row withoutend) into an exit that names the atoms file, not a traceback.outside_sharereports the share its counts give (83%, wheremainbefore refactor(novelty): the bundle record owns its validity #44 printed 0%). That figure was printed but pinned nowhere. The gate figures for the other stored shapes are already pinned intest_novelty_bundle_guards.py. The parity golden file is a capture ofmainbefore refactor(novelty): the bundle record owns its validity #44, so these cases stay out of it.cross-judge --judgesgate refuses one damaged question among eight when the ids are ints. A gate that read onlystrids passed before.from_artifactskips a non-dict arm next to one it serves.unreproducible_armsskips records, arms, evaluations and scores of the wrong type.Docs
--all-atomsto the tier command and to run--estimate-onlyfirst, since--all-atomsalone drops the tier.repo reachpasses on:ANTHROPIC_API_KEY, orAWS_BEARER_TOKEN_BEDROCKwithAWS_REGIONandCLAUDE_CODE_USE_BEDROCK.evaluating-treatments.md, the runner comment inci.yml(111 characters) and the coverage bullet inCONTRIBUTING.md.Risk
None to behaviour. No changelog fragment: nothing under
dokimi/dokimi/changes.Testing
On a Linux review host:
make ci-local: 19 passed, 0 failed, 4 skipped (the Docker catalog smoke,test-latest, docsdeploy, semgrep). 5,118 tests passed, 1 skipped.load_questions(args.questions)withoutvalidate=Falseinrefine;except KeyErrorinstead ofValueErroraroundentry_point_onlyinshorten;isinstance(question_id(r), str)in the gate;isinstance(arm, dict)infrom_artifact;isinstance(s, dict)inunreproducible_arms.Checklist
pytestpassesruff,mypy, and schema checks pass.github/.licenserc.yaml)