Add GOV.UK mirror + task verifiers (site by @lamawmouk, verifiers by reviewer)#67
Open
Django-Jiang wants to merge 5 commits into
Open
Add GOV.UK mirror + task verifiers (site by @lamawmouk, verifiers by reviewer)#67Django-Jiang wants to merge 5 commits into
Django-Jiang wants to merge 5 commits into
Conversation
Adds a Flask mirror of https://www.gov.uk/ as the 16th WebHarbor site, running on port 40015. ## What's mirrored - 16 top-level topics (Money and tax, Visas and immigration, Driving, ...) - 44 subtopics - 15 government departments (HMRC, DfE, Home Office, DVLA, NHS England, ...) with real ministers / permanent secretaries / employee counts - 73 guidance articles (Self Assessment, Income Tax, Universal Credit, Skilled Worker visa, passport applications, vehicle tax, ...) - 20 announcements (press releases, news stories, speeches) - Search across articles / announcements / departments ## Visual fidelity Uses the official MIT-licensed govuk-frontend v6.1.0 CSS + JS + GDS Transport font + crown SVG. Templates use the canonical Design System component DOM (govuk-header, govuk-breadcrumbs, govuk-summary-list, govuk-pagination, govuk-grid-row, etc.) so an agent's selectors match the real GOV.UK. Content licensed under the Open Government Licence v3.0 (synthesized in the spirit of GOV.UK guidance; no upstream copy embedded). ## Folder layout Matches the canonical site layout (compare wolfram_alpha, google_search): sites/gov_uk/ |-- _health.py |-- app.py |-- seed_data.py |-- tasks.jsonl |-- instance_seed/ (HF-managed) |-- static/{css,js,fonts,icons,images,external_cache}/ \`-- templates/ ## Wiring - websyn_start.sh: gov_uk appended to SITES, 15->16 counts - control_server.py: gov_uk added to SITES - Dockerfile: EXPOSE 40000-40015 ## Pre-PR verification (passed) - docker build webharbor:dev clean (5.92 GB) - 16/16 sites bind in 2s - All gov_uk routes (/, /browse, /browse/<topic>, /browse/<t>/<s>, /guidance/<slug>, /government/organisations[/<dept>], /government/announcements, /search, /_health) return 200 - /reset/gov_uk -> {ready: true}, md5 byte-identical pre/post - Byte-identical after docker restart ## Asset PR Seed DB (gov_uk.tar.gz, 32 KB) uploaded as HF PR: https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/22 .assets-revision will be bumped to the HF merge SHA once that PR lands.
…ta from tarballs - .assets-revision: point revision at refs/pr/22 so fetch_assets.sh pulls gov_uk.tar.gz (it isn't on main yet); TODO bump back to main after merge - extract_assets.sh: set COPYFILE_DISABLE=1 and use --no-mac-metadata / --no-xattrs so repacks on macOS no longer embed ._* AppleDouble files or com.apple.* xattrs (feature-detected; safe on GNU tar)
Replace the empty tasks.jsonl ([] placeholder) with 20 grounded,
answerable tasks in the standard JSONL format
({web_name,id,ques,web,upstream_url}, port 40015). Coverage spans
guidance fact-finding (Income Tax, VAT, State Pension, Self Assessment),
department lookups (minister, headcount, established year),
announcements, search, and topic browsing. Each task's answer was
verified to exist in instance_seed/gov_uk.db.
# Conflicts: # .assets-revision # control_server.py # websyn_start.sh
Add the reviewer grading contract for the GOV.UK mirror (site + 20 tasks by @lamawmouk, PR aiming-lab#32): - sites/gov_uk/verify/verify_lib.py: shared deterministic + anchored-LLM utilities (trajectory navigation, answer/token match). This site is read-only, so there is no DB after-state check. - verify_0.py .. verify_19.py: one deterministic verifier per task. - tasks.jsonl: backfill verifier_path + judge_rubric for all 20 tasks. Grading is deterministic-first: navigation evidence (anti knowledge-shortcut, which is what makes real-world-fact questions like the Personal Allowance or VAT rate valid navigation tasks) plus answer/token match against frozen ground truth. The anchored LLM check is secondary — it can only reject on a genuine contradiction and is skipped when no LLM is configured, so it never fails an otherwise-correct answer. Validated: a no-op run FAILs all 20; a correct solve PASSes all 20; wrong-answer and no-navigation (shortcut) cases FAIL as expected. Also confirmed through the primary eval_judge.py --verifier True entry point.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reviewer grading contract for the GOV.UK mirror. Based on @lamawmouk's original PR #32 (site + 20 tasks); this branch adds the deterministic verifiers +
judge_rubricthat the reviewer role is responsible for. Ground truth lives only inside the verifiers —tasks.jsonlgetsverifier_path+judge_rubriconly (no answer key).Review summary
Mechanical: PASS
gov_uk(md5 ofinstance/vsinstance_seed/match)reset-all~1s.assets-revisionpins a real HF commit SHA (d02361e…);gov_uk.tar.gzresolvesVisual + functional: PASS
Task quality: PASS — no task changes required
/government/announcements/ homepage) without a further click. These two are weaker anti-shortcut tasks than the other 18. The verifier still requires navigating to the announcements area and matching the figure, but it can't require a deeper page that doesn't exist. Not fixed in this PR because the titles live in the contributor's seed DB, which is a pinned HF-managed asset (instance_seed/gov_uk.db) — regenerating and re-uploading it is out of scope for a reviewer verifier PR (and real GOV.UK headlines do carry the figure). Suggested follow-up for @lamawmouk: add a per-announcement detail route (mirroring/guidance/<slug>) and keep the figure in the story body rather than the headline, so these two tasks force one level deeper of navigation.Grading contract added
sites/gov_uk/verify/verify_lib.py+verify_0.py…verify_19.py(one per task)tasks.jsonl:verifier_path+judge_rubricbackfilled for all 20 tasksGrading validated three ways
--no_llm True)eval_judge.py --verifier Trueentry point (correct → PASS, no-op/shortcut/wrong → FAIL)