Skip to content

refactor(core): share the docs guide-read check across docs evals - #230

Draft
czenko wants to merge 2 commits into
mainfrom
core/share-docs-guide-read-check
Draft

refactor(core): share the docs guide-read check across docs evals#230
czenko wants to merge 2 commits into
mainfrom
core/share-docs-guide-read-check

Conversation

@czenko

@czenko czenko commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Problem

build-docs-002-rls-guide builds its guide-read check by hand, and the API keys eval in #212 carries the same function again. Same filter, same hasContent split, same three-way notes string, same comment above it. The only difference is the guide's name.

Ten more docs evals are scoped. Each one points an agent at a page and needs to know whether the agent retrieved it, because without that a red run can't distinguish a guide that taught the wrong thing from a guide the agent never opened.

Solution

  • Adds checkDocsGuideRead(toolCalls, { path, label }) to docs-results.ts, beside buildDocsResult. It returns the CheckResult the eval-local copies were building.
  • Points build-docs-002-rls-guide at it and deletes the local copy.
  • Keeps the check name string byte-identical, so result history doesn't break.
  • Covers the three outcomes with tests: guide retrieved, guide reached with no page content, guide not reached.

Manual testing

  1. Run pnpm --filter @supabase-evals/core test. 56 pass, including three new checkDocsGuideRead cases.
  2. Run pnpm typecheck. Clean. It covers evals/**/EVAL.ts.
  3. Read the build-docs-002-rls-guide/EVAL.ts diff. Confirm the name passed to label reproduces the removed check's name exactly.

@czenko
czenko requested a review from a team August 24, 2026 16:56
@vercel

vercel Bot commented Aug 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
evals Ignored Ignored Preview Aug 24, 2026 5:10pm

Request Review

* check a red run is ambiguous: the guide may have taught the wrong thing, or
* the agent may never have opened it.
*/
export function checkDocsGuideRead(

@czenko czenko Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth noting, since this is an architectural decision: this is the first CheckResult builder in core. Core has parsed traces into data and left the pass/fail decision to evals until now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@czenko
czenko force-pushed the core/share-docs-guide-read-check branch from 53a5412 to 8c98b7c Compare August 24, 2026 17:09
Comment thread packages/core/src/docs-results.ts Outdated
@czenko
czenko marked this pull request as draft August 25, 2026 17:21
@czenko

czenko commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Parking this as a draft per the thread. #212 is unstacked and standalone on main now, carrying its own copy of the check.

Holding until the structured docs-injection idea is decided. If a includeDocs: frontmatter field lands, this check goes vacuous and this PR should just close. If it doesn't, this is still the cleanup once a third docs eval needs it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant