Skip to content

Fix section hits across mismatched sources - #1

Open
Abduhu1 wants to merge 1 commit into
VibeCodingScientist:feat/amfv-bench-v0from
Abduhu1:abdullah/section-hit-source-match
Open

Fix section hits across mismatched sources#1
Abduhu1 wants to merge 1 commit into
VibeCodingScientist:feat/amfv-bench-v0from
Abduhu1:abdullah/section-hit-source-match

Conversation

@Abduhu1

@Abduhu1 Abduhu1 commented Aug 21, 2026

Copy link
Copy Markdown

Summary

  • Require a section hit to match both the evidence source_id and section heading.
  • Prevent identical section headings from different guidelines from being counted as section hits.
  • Add a regression test covering the mismatched-document case.

Why

Previously, _section_hit compared only section text. This meant evidence from nice-ng238 could receive a section hit against gold evidence from nice-ng136 when both happened to use the same heading, even though the document hit correctly failed.

Section-level retrieval should only count when the section belongs to the correct source document.

Tests

  • uv run --frozen pytest -q — 56 passed
  • uv run --frozen ruff check . — passed
  • uv run --frozen ruff format --check . — passed
  • git diff --check — passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant