feat(bench): answer bench harness and the 5-filing pilot

[hallelx2]

The answer-accuracy harness that sits between the head-skim work (#71) and pincites (#72) in this stack. It grades answers with PageIndex/Mafin's own grading prompt; the pilot over 5 filings and 30 questions graded 27 correct.

Verified on Lorx CI at the stack's tip (vectorless-engine run #3: fmt, vet, build, golangci-lint, test all green). GitHub Actions cannot start jobs on this account (HAL-1712).

Closes HAL-2705

Summary by Sourcery

Add answer generation and richer performance instrumentation to the navigation benchmark harness.

New Features:

• Add an optional answer-generation stage to navbench that produces grounded responses from retrieved evidence and records answer quality and token/latency metrics.
• Expose answer-generation controls for model configuration, reasoning, evidence-page limits, and page head-length benchmarking.
• Record PDF parsing duration in tocdump output.

Enhancements:

• Extend benchmark results with generated answers, gold answers, and answer-generation performance data.
• Allow navigation benchmarks to configure the number of characters included in coarse page skims.

[sourcery-ai]

Sorry @hallelx2, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 10 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 08b4e8e4-ed07-40b0-bedd-7d09c6633c3a

---

📥 Commits

Reviewing files that changed from the base of the PR and between a7259fa and ca9321c.

---

📒 Files selected for processing (4)

• cmd/navbench/answer.go
• cmd/navbench/main.go
• cmd/tocdump/main.go
• navbench

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Extends navbench with an optional, configurable grounded-answer generation step over retrieved pages and records answer metrics alongside retrieval results; also adds head-skim configuration and PDF parse-time telemetry.

Sequence diagram for the navbench grounded-answer evaluation

sequenceDiagram
    participant CLI as navbench
    participant Nav as JudgeNavigator
    participant API as OpenAICompatibleAPI
    participant Output as ResultsJSONL

    CLI->>Nav: retrieval.JudgeNavigator.Search
    Nav-->>CLI: res.Evidence
    CLI->>API: POST /chat/completions
    API-->>CLI: answer text and usage
    CLI->>Output: write answer, gold answer, and answer metrics

Loading

Flow diagram for navbench answer and telemetry options

flowchart TD
    Start[Run navbench] --> Configure[Configure head skim and page limits]
    Configure --> Retrieve[Retrieve and rank evidence pages]
    Retrieve --> AnswerEnabled{answer flag enabled?}
    AnswerEnabled -- No --> Record[Record retrieval result]
    AnswerEnabled -- Yes --> Prompt[Build grounded prompt from evidence pages]
    Prompt --> Generate[Generate one answer with configured model]
    Generate --> RecordAnswer[Record answer, gold answer, tokens, and latency]
    RecordAnswer --> Record
    Record --> End[Write JSONL output]

    Parse[Parse PDF pages] --> ParseTime[Record parse_seconds]

Loading

File-Level Changes

Change	Details	Files
Add an optional answer-generation stage to the benchmark pipeline.	Call an OpenAI-compatible chat-completions endpoint using the selected evidence pages. Support configurable endpoint, model, API key, page budget, and reasoning mode. Record generated answers, gold answers, latency, and token usage in benchmark output. Surface answer failures through the existing per-question error field.	cmd/navbench/answer.go cmd/navbench/main.go
Expose coarse head-skim sizing as a benchmark control.	Add a command-line flag for page head character limits. Pass the configured value into JudgeNavigator.	cmd/navbench/main.go
Record PDF parsing time in TOC ingestion results.	Measure readPages duration separately from subsequent document building. Persist the duration as parse_seconds in dump output.	cmd/tocdump/main.go

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.