Skip to content
rig-cPublic

About

An open-book exam for AI agents: one painting to a rig with the published rig-c tools, journal first

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Profile results automation

The refresh workflow lives in rig-c/.github, where it runs daily and on workflow_dispatch. This repository supplies the two table generators and tools/profile_results.py; it does not publish to the profile on results pushes. The profile workflow checks out public exam main, records the actual exam checkout SHA, and updates its own profile/README.md through one fixed-branch PR. Review and merge remain a human sign.

The workflow uses only its repository's default GITHUB_TOKEN with contents: write and pull-requests: write; no additional secrets are needed. Polling means a results change can wait up to a day for the next scheduled run, plus any GitHub scheduler delay. A manual dispatch can refresh sooner. Checks on GITHUB_TOKEN-created PRs do not run automatically; the signer must run or approve the appropriate checks before merging. The destination workflow's README explains dispatch setup and the first-run default-branch requirement.

Generated regions and migration

tools/profile_results.py writes the exact stdout bytes, including the final newline, of:

python3 results.py --profile
python3 advanced/results.py --profile

These replace the bodies of exam:start / exam:end and exam-advanced:start / exam-advanced:end. The existing marker comments are preserved. A third pair, exam-summary:start / exam-summary:end, owns just two derived count paragraphs. All markers must be complete HTML comments on their own lines, with exactly one ordered, non-overlapping pair per region. A missing table marker, a partial or duplicate summary pair, or overlapping regions fails before writing the file.

The first run supports the two-pair profile before and after rig-c/.github#5: the Basic sentence beginning “Every line passed, and the three models named…” or “Eleven models have sat it, from each vendor's lighter tier…”, and the Advanced sentence beginning “Five/Six models have sat all three rounds and one cell failed…” through “(shown here as a what-if).” These exact sentence templates allow changed numeric counts and line wrapping. Only those obsolete sentences are removed. The invitation to sit the exam, Advanced's introduction, links and unrelated prose remain. The summary pair is inserted immediately after the Basic table. A second run is a byte no-op and does not rewrite the file.

If an editor has changed those sentences into an unrecognised form, the workflow refuses to guess. In a reviewed profile change, remove only the stale count or result-comparison sentences, preserve the other prose, and insert this empty pair outside the two table regions. Then rerun the workflow to populate it:

<!-- exam-summary:start — generated by rig-c/exam; do not edit by hand -->
<!-- exam-summary:end -->

The summary reads the existing generators' data and selection functions; it does not change tables or grade any run:

  • Basic: group by the same directory/model key, include only counted() sittings (counts starts with independent), and count a model as passing every graded line only if it has graded tasks and every task in every counted sitting has a positive denominator and passed == of. This includes tasks on Round 5's separately tabled painting. Missing tasks do not establish a pass or completion of the whole exam. Repeated sittings do not add models.
  • Advanced: use by_model(sittings()), including its last-sitting-per-round selection and without adding an independence filter. A model passes all three only with R1, R2 and R3 present and every selected grade having a positive denominator, passed == of, and all row statuses PASS. FAIL and SKIP prevent that count. The non-passing cell count covers only selected, present rounds. Shin-scale what-ifs never change these counts; a native edition's shin row is already part of its recorded grade.

Local targeted verification

No Node install, grader, build or full exam suite is needed. Keep dependencies and temporary files in the ignored work directory:

python3 -m pip install --no-cache-dir --target work/profile-pydeps -r .github/requirements-profile-tests.txt
PYTHONPATH=work/profile-pydeps python3 -B -m unittest discover -s tools -p 'test_profile_results.py' -v

profile-results-test.yml runs this targeted suite in CI. The fixture is the two-pair public profile read for this change. Tests cover the real CLI outputs, both legacy migrations, preservation, malformed/missing/duplicate markers, counting edge cases, the local CLI, and its second-run no-op. Tests use work/profile-tests/ and never invoke Git or publish. The profile repository owns the refresh workflow's contracts and remote checkout/commit/PR verification.

About

An open-book exam for AI agents: one painting to a rig with the published rig-c tools, journal first

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages