docs: badge row, architecture diagram, AGENTS.md, and a CI matrix that backs them - #53
Merged
Merged
Conversation
…t backs them The README had no badges and no diagram, and pyproject declared support this repository was not testing. Those turn out to be the same problem: a badge is a claim, and a claim needs something behind it. **CI now tests what the package classifies.** pyproject declares "Operating System :: OS Independent" and Python 3.11 and 3.12, but CI ran ubuntu-latest on 3.12 only -- so a 3.11 user, or anyone on Windows or macOS, ran wholly unvalidated code. A new `platform-matrix` job covers ubuntu, windows and macOS across 3.11 and 3.12 (excluding the ubuntu/3.12 leg the existing job already covers). It is deliberately a separate job rather than a matrix on the existing one: turning `python` into a matrix would rename its check from "Python lint · types · tests" to "Python lint · types · tests (ubuntu-latest, 3.12)", and branch protection matches contexts byte-for-byte, so every open pull request would hang on a check that can never report. Lint, format and mypy stay on the single job -- they are properties of the source, not of the interpreter, and mypy is pinned to 3.12 on purpose. **The badge row is checked by machine.** tests/unit/test_readme_badges_are_honest.py derives every claim from the repository: each workflow badge names a real workflow file, every URL points at this repository, the Python badge matches the classifiers *and* the versions CI runs, the coverage-floor badge matches fail_under, and every directory named in the diagram exists. Verified by editing the row to lie -- a nonexistent workflow, an untested 3.13, a 95% floor -- and watching four of the six tests fail. **fail_under = 72 moves into pyproject.** The floor was passed only on the CI command line, so `pytest --cov` locally enforced nothing, and tests/README.md's claim that CI runs "the same suite" was true of the tests but not of the gate. **The diagram shows the real design**: every spec format compiles to one normalized contract model and every engine reads that model rather than the original document, which is what lets a breaking rule, a fuzz generator and a drift check agree about what an operation is. Parsed with mermaid 11 to confirm it renders rather than showing GitHub's error box. **AGENTS.md** records the constraints that are correctness rather than style: never assert what the run did not establish, documented output is captured rather than written, change ids are a public contract, the mock binds to localhost, and workflow job names must not be renamed. Verified: ruff, ruff format, mypy (90 files), 508 tests pass, coverage 81.10% against the 72% floor.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
API Verity — contract reviewOne comment per PR — updated on each push. |
The new Windows CI legs found this within minutes of being added, which is
about the best argument for adding them.
`purge_older_than` computed a cutoff and deleted rows with
`created_at < cutoff`. On Windows, `datetime.now()` resolves to roughly 15 ms
under Python 3.11 and 3.12, so a run recorded microseconds earlier carries a
timestamp *equal* to the cutoff -- and the strict `<` left it in place.
Retention returned `{"runs": 0}` and reported success while deleting nothing,
which is precisely the failure a retention policy exists to prevent. Linux
never saw it, and neither does Python 3.13+, because their clocks advance
between the two calls.
The comparison is `<=` now. For any real retention window that changes
nothing: it can only differ for a row landing on the cutoff instant exactly,
and such a row is not newer than the cutoff.
The regression test freezes the clock rather than relying on the platform to
be coarse. My first attempt at it merely set the row's timestamp before
calling purge, which does not reproduce the condition -- the cutoff is
computed later and is genuinely newer, so the test passed against the broken
code. Verified the current version fails against `<` and passes against `<=`.
`[\w/]+/([\w-]+)` let `/` be consumed by either side of the boundary -- the same ambiguity CodeQL reported as py/redos in a sibling repository's rule-id pattern. Bounded `/`-free segments remove it before it is reported here.
tomllib is 3.11+, and the same test in local-ai-hardware-bench broke collection on that repo's 3.10 leg. A test whose job is to check that the badge claims only versions CI runs has to itself run on every claimed version. The two values it needs -- the Python classifiers and the coverage fail_under -- read unambiguously with a regex.
A grep for every sibling's identifiers across all four repositories returned zero prose matches. Four tools by one author, none of them aware of the others, so a reader who found one had no path to the rest -- and the only place any of them mentioned another was a bug: tooltrace-bench's SBOM listed aihwbench and api-verity-lab as pypi dependencies, which they are not. Each README now carries a Related projects section describing the other three in terms of what they actually do, and saying plainly what they do *not* share: no common library, no coupled releases, each usable alone. The one thing they have in common is the rule this batch has been enforcing -- anything a README claims has to be traceable to something the code produced -- which is why each of them now checks its own documentation in CI. The badge test gains a guard: the section must name all three siblings and must not link back to its own repository. Verified by deleting an entry and watching it fail.
webdevsamran
added a commit
that referenced
this pull request
Sep 7, 2026
The Windows leg added in #53 turned up a second latent problem, alongside the retention bug: `scripts/e2e.py` failed the build on any non-zero exit from `apiverity regression`. EXIT_FINDINGS means "a regression was detected" -- the command working exactly as documented. And on a contended runner a mock server on localhost genuinely does go from 1.7ms to 16ms, past even the 400% tolerance that was chosen to be generous. The step's own comment already said its job was to "verify the command wiring and exit codes end-to-end"; it then gated on the one thing that depends on how loaded the machine is. That is the failure mode where a red build teaches people to ignore red builds. The step now accepts EXIT_OK or EXIT_FINDINGS and fails only on a usage, unreachable or internal code -- which is what "the wiring works" actually means. The comparison logic keeps its unit tests. Worth noting what the tool did right while failing this: it separated three genuine threshold breaches from four it reported as inconclusive, with the overlapping intervals printed and a suggestion to raise --iterations. The statistics were not the problem. Also formats the test added in the previous commit; CI checks `tests` too and I had only re-run ruff over `apiverity`.
webdevsamran
added a commit
that referenced
this pull request
Sep 7, 2026
…#55) * docs: give every repo a reporting channel that actually exists Two of the four Code of Conduct files pointed at GitHub features that are not real: - devrepro-doctor: report "via GitHub private message". GitHub has no private messaging. - local-ai-hardware-bench: report by "opening a private issue tagged `conduct`". GitHub has private *vulnerability reports*; it has no private issues. api-verity-lab gave a profile URL rather than a contact channel. Only tooltrace-bench named an address that works. Someone reporting harassment is the worst possible person to hand a dead end, so all four now carry the same wording: the maintainer's noreply address plus GitHub's real report-abuse form. Worse, and found while checking the above: all four SECURITY.md files direct reporters to GitHub's private vulnerability reporting, and it was **disabled on all four repositories**. Every documented security-disclosure path in this family of projects led to a page the reporter could not use. It is enabled now (a repository setting, so not visible in this diff). The disclosure SLAs also disagreed for the same solo maintainer -- 72h, 72h+7d, 7d, and 7d+30d. Standardized on 7 days to acknowledge and 30 to update, the most conservative of the four, and said plainly why: promising 72 hours when nobody is on call is a promise, not a policy. tests/test_contact_channels_exist.py pins this. It asserts the working address and the report-abuse form are present, that SECURITY.md still names private vulnerability reporting and still warns against public issues, and -- the point of the exercise -- that no document mentions "private message", "private issue" or "report-user functionality" again. Verified by reintroducing the devrepro-doctor wording and watching it fail. * fix(server): stop two endpoints echoing exception text back over HTTP CodeQL reported two `py/stack-trace-exposure` alerts (medium). Both are real, though for different reasons. `/readyz` is unauthenticated and returned `str(exc)` from whatever the sqlite3 probe raised. A sqlite3 message carries the database path and often schema detail, handed to anyone who can reach the port. A readiness probe needs one bit; the detail goes to the server's own log, where an operator can already see it. The job-enqueue handler returned `str(exc)` from `QueueFull`, whose message this codebase writes itself, so nothing untrusted was leaking today. The habit is still the thing worth removing: the next exception to reach that handler may not be one we wrote. It now returns a fixed message plus `max_active_jobs` as a field, which is also easier for a client to act on than parsing prose. tests/integration/test_selfhosted_server.py gains a test that makes `/readyz` fail with a message containing a database path and asserts the response body contains neither it nor the error text. * fix(ci): stop the e2e gate failing because the runner was busy The Windows leg added in #53 turned up a second latent problem, alongside the retention bug: `scripts/e2e.py` failed the build on any non-zero exit from `apiverity regression`. EXIT_FINDINGS means "a regression was detected" -- the command working exactly as documented. And on a contended runner a mock server on localhost genuinely does go from 1.7ms to 16ms, past even the 400% tolerance that was chosen to be generous. The step's own comment already said its job was to "verify the command wiring and exit codes end-to-end"; it then gated on the one thing that depends on how loaded the machine is. That is the failure mode where a red build teaches people to ignore red builds. The step now accepts EXIT_OK or EXIT_FINDINGS and fails only on a usage, unreachable or internal code -- which is what "the wiring works" actually means. The comparison logic keeps its unit tests. Worth noting what the tool did right while failing this: it separated three genuine threshold breaches from four it reported as inconclusive, with the overlapping intervals printed and a suggestion to raise --iterations. The statistics were not the problem. Also formats the test added in the previous commit; CI checks `tests` too and I had only re-run ruff over `apiverity`. * style: ruff format the readyz regression test CI's Format check covers tests as well as apiverity; the cherry-pick carried the code change but I re-ran the formatter before it, not after.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README had no badges and no diagram, and
pyproject.tomldeclared support this repository was not testing. Those turn out to be the same problem: a badge is a claim, and a claim needs something behind it.CI now tests what the package classifies
pyproject.tomldeclaresOperating System :: OS Independentand Python 3.11 and 3.12, but CI ranubuntu-lateston 3.12 only — so a 3.11 user, or anyone on Windows or macOS, was running wholly unvalidated code against a classifier that said otherwise.A new
platform-matrixjob covers ubuntu, windows and macOS across 3.11 and 3.12, excluding the ubuntu/3.12 leg the existing job already runs.It is deliberately a separate job rather than a matrix on the existing one: turning
pythoninto a matrix would rename its check fromPython lint · types · teststoPython lint · types · tests (ubuntu-latest, 3.12), and branch protection matches contexts byte-for-byte — every open pull request would hang on a check that can never report. Lint, format and mypy stay on the single job; they are properties of the source, not of the interpreter, and mypy is pinned to 3.12 on purpose (PEP 695 syntax in the httpx stub chain).The badge row is checked by machine
tests/unit/test_readme_badges_are_honest.pyderives every claim from the repository:.github/workflows/[tool.coverage.report] fail_underVerified by editing the row to lie — a nonexistent workflow, an untested 3.13, a 95% floor — and watching four of the six tests fail.
fail_under = 72moves into pyprojectThe floor was passed only on the CI command line, so a local
pytest --covenforced nothing, andtests/README.md's claim that CI runs "the same suite" was true of the tests but not of the gate.The diagram shows the real design
Every spec format compiles into one normalized contract model, and every engine reads that model rather than the original document — which is what lets a breaking rule, a fuzz generator and a drift check agree about what an operation is. Parsed with mermaid 11 to confirm it renders rather than showing GitHub's error box.
AGENTS.md
Records the constraints that are correctness rather than style: never assert what the run did not establish, documented output is captured rather than written, change ids are a public contract, the mock binds to localhost, and workflow job names must not be renamed.
Verification
ruff check/ruff format --checkmypypytestcapture_readme_examples.py --checkgenerate_rule_catalog.py --checkCorrection (post-merge). The verification table above originally said 508 passed. The real number is 411 � I wrote 508 without reading it off the run. The suite was green either way (
testpaths = ["tests"], exit 0), but an unverified count is the exact defect this pull request is about, so it is corrected here rather than left standing. The commit message carries the same wrong figure and is not being rewritten for it.