fix: close the defect ledger, make workspace paths install-safe, correct the docs - #21
Merged
Merged
Conversation
…ect the docs Clears every defect in the review ledger except the provider-deadline item, fixes the documentation claims that had drifted from the code, and removes development-only files from the repository. Correctness - Unified terminal outcomes. `succeeded` / `needs_clarification` / `blocked` / `partial` / `failed` / `cancelled` are derived in `queryforge/core/outcomes.py`, so the persisted run status, the delivery report and the event stream can no longer disagree. A quality gate that blocks a run is no longer reported as completed. - Reflection that asks for clarification used to raise, discarding the answer and the run's artifacts. It now terminates as a structured `needs_clarification`. - The `reasoning` audit payload was silently dropped whenever the model returned type-compatible but schema-incompatible shapes (string lists, the string "None", word confidences such as "high"). Shapes are normalised and a discarded payload is now reported instead of ignored. - Preview execution evaluated a rewritten statement, so a `require_limit` policy was satisfied by the injected LIMIT rather than by the caller's SQL. - Time-filter validation ignored `BETWEEN` and per-call boundaries, so a partially bounded time filter could validate as complete. - Truncated result sets overwrote the row count, hiding that the bound had been hit; `truncated` and `fetched_row_count` now preserve both numbers. - Chinese follow-up questions were not recognised, so multi-turn context was lost. Route matching for report/explain intents is no longer triggered by a bare substring, so a table named `sales_report` is not routed as a report request. Governance and audit - `DatabaseTool.last_policy_decision` is read-only. It can no longer be overwritten by a caller, which had let the tool report an audit record for a call that never happened. - Execution fingerprints bind data, semantic and policy versions, persisted in the run journal, so a resumed run cannot silently reuse a step computed against a different model or policy. - Budget state is inherited across a resume instead of restarting at zero. - Cross-instance journal leases are exclusive (`fcntl.flock` plus a reload before acquiring), so two workers can no longer hold the same lease. - CSV knowledge is trusted only when a reviewer is recorded; the governed knowledge build path is reachable from the CLI (`--kb-knowledge`). Measurement - The evaluator compares result projections tolerantly: a correct answer is no longer scored wrong for returning extra columns or an equivalent shape. - Account-level failures (HTTP 402 and friends) are classified as environment errors and excluded from accuracy denominators instead of being counted as model failures. - Token usage is reported as measured when the provider returns it, with `token_source` distinguishing measured from estimated figures. - `--skill-mode auto|off` and `--parallel-candidates` make the skill-selection and candidate-count questions testable; the `skills=[]` default that silently disabled the skill catalogue in every measured run is gone. - The evaluator's isolation guard is recursive and covers plain `import` statements. Repository and structure - Workspace-relative paths (`.queryforge/` state, sample data, evaluation sets) are resolved by `queryforge/core/paths.py` from `QUERYFORGE_ROOT`, an enclosing source checkout, or the working directory. Deriving them from `Path(__file__).parents[N]` pointed into `site-packages` after an install, so run state moved into the installed package. Packaged resources such as `bundled_skills/` deliberately still resolve relative to the package. - `make` targets prefer the repository virtualenv, so `make check` uses the same interpreter as `./init.sh` instead of whatever `python` is first on `PATH`. - Development-only files (process plans, defect ledgers, session notes) are removed; `./init.sh` is now a single offline verification entry point. - Studio uploads accept only csv/parquet, matching what publication can accept; a rejected database file explains why. `docs/` is reorganised and its index no longer points two entries at the same page. Documentation - Both READMEs state measured behaviour with its sample size and its caveats, and the stale figures are corrected: 928 tests (was 691/806), 23/23 tier-1 tasks over 32 gold tasks across 3 schemas (was 32/32), and a real-model NL2SQL result (was "not run here"). Claims about automatic skill selection and the PostgreSQL backend are marked as unestablished and unverified rather than implied working. - `docs/nl2sql_evaluation.md` records the frozen baselines, and its token and cost section no longer describes a heuristic the code no longer uses. - CHANGELOG.md lists the above. Verification - `./init.sh` — 928 tests, 25 skipped, 0 failures - `make check` — 13/13 offline acceptance checks - `make web-check` — eslint, tsc, production build, 24 TypeScript tests - `python scripts/check_repository.py` — no broken links, no secrets, hygiene clean Not verified: no tier-3 real-model run was repeated after these changes, so accuracy remains 0.875 on 40 anime cases from a single run, where +/-0.03 is noise. The PostgreSQL backend is still unverified against a live server.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Closes every defect in the review ledger except the provider-deadline item, fixes documentation claims that had drifted from the code, and removes development-only files from the repository.
Correctness
succeeded/needs_clarification/blocked/partial/failed/cancelledare derived inqueryforge/core/outcomes.py, so persisted run status, the delivery report and the event stream can no longer disagree. A quality gate that blocks a run is no longer reported as completed.needs_clarification.reasoningaudit payload was silently dropped whenever the model returned type-compatible but schema-incompatible shapes (string lists, the string"None", word confidences like"high"). Shapes are normalised, and a discarded payload is reported instead of ignored.require_limitpolicy was satisfied by the injected LIMIT rather than by the caller's SQL.BETWEENand per-call boundaries, so a partially bounded filter could validate as complete.truncatedandfetched_row_countnow preserve both numbers.sales_reportis not routed as a report request.Governance and audit
DatabaseTool.last_policy_decisionis read-only — it could previously be overwritten by a caller, letting the tool report an audit record for a call that never happened.fcntl.flock+ reload before acquiring).--kb-knowledge).Measurement
token_sourcedistinguishing measured from estimated.--skill-mode auto|offand--parallel-candidatesmake the skill-selection and candidate-count questions testable; theskills=[]default that silently disabled the skill catalogue in every measured run is gone.importstatements.Repository and structure
.queryforge/state, sample data and evaluation sets resolve throughqueryforge/core/paths.pyfromQUERYFORGE_ROOT, an enclosing source checkout, or the working directory. Deriving them fromPath(__file__).parents[N]pointed intosite-packagesafter an install, so run state moved into the installed package. Packaged resources such asbundled_skills/deliberately still resolve relative to the package.maketargets prefer the repository virtualenv, somake checkuses the same interpreter as./init.shinstead of whateverpythonis first onPATH../init.shis now a single offline verification entry point.docs/reorganised; its index no longer points two entries at the same page.Documentation
docs/nl2sql_evaluation.mdrecords the frozen baselines, and its token/cost section no longer describes a heuristic the code no longer uses.Verification
./init.shmake checkmake web-checkpython scripts/check_repository.pyNot verified