Skip to content

fix: close the defect ledger, make workspace paths install-safe, correct the docs - #21

Merged
lanerchenbuna merged 1 commit into
mainfrom
fix/defect-ledger-and-docs
Sep 22, 2026
Merged

lanerchenbuna merged 1 commit into
mainfrom
fix/defect-ledger-and-docs

Conversation

@lanerchenbuna

Copy link
Copy Markdown
Owner

What this does

Closes every defect in the review ledger except the provider-deadline item, fixes documentation claims that had drifted from the code, and removes development-only files from the repository.

Correctness

  • Unified terminal outcomes — succeeded / needs_clarification / blocked / partial / failed / cancelled are derived in queryforge/core/outcomes.py, so persisted run status, the delivery report and the event stream can no longer disagree. A quality gate that blocks a run is no longer reported as completed.
  • Reflection that asks for clarification used to raise, discarding the answer and the run's artifacts; it now terminates as a structured needs_clarification.
  • The reasoning audit payload was silently dropped whenever the model returned type-compatible but schema-incompatible shapes (string lists, the string "None", word confidences like "high"). Shapes are normalised, and a discarded payload is reported instead of ignored.
  • Preview execution evaluated a rewritten statement, so a require_limit policy was satisfied by the injected LIMIT rather than by the caller's SQL.
  • Time-filter validation ignored BETWEEN and per-call boundaries, so a partially bounded filter could validate as complete.
  • Truncated result sets overwrote the row count, hiding that the bound had been hit; truncated and fetched_row_count now preserve both numbers.
  • Chinese follow-up questions were not recognised, so multi-turn context was lost. Route matching no longer fires on a bare substring, so a table named sales_report is not routed as a report request.

Governance and audit

  • DatabaseTool.last_policy_decision is read-only — it could previously be overwritten by a caller, letting the tool report an audit record for a call that never happened.
  • Execution fingerprints bind data, semantic and policy versions, so a resumed run cannot silently reuse a step computed against a different model or policy.
  • Budget state is inherited across a resume instead of restarting at zero.
  • Cross-instance journal leases are exclusive (fcntl.flock + reload before acquiring).
  • CSV knowledge is trusted only when a reviewer is recorded; the governed knowledge build path is reachable from the CLI (--kb-knowledge).

Measurement

  • The evaluator compares projections tolerantly: a correct answer is no longer scored wrong for returning extra columns or an equivalent shape.
  • Account-level failures (HTTP 402 and friends) are environment errors, excluded from accuracy denominators, not model failures.
  • Token usage is reported as measured when the provider returns it, with token_source distinguishing measured from estimated.
  • --skill-mode auto|off and --parallel-candidates make the skill-selection and candidate-count questions testable; the skills=[] default that silently disabled the skill catalogue in every measured run is gone.
  • The evaluator isolation guard is recursive and covers plain import statements.

Repository and structure

  • Workspace paths are install-safe. .queryforge/ state, sample data and evaluation sets resolve through queryforge/core/paths.py from QUERYFORGE_ROOT, an enclosing source checkout, or the working directory. Deriving them from Path(__file__).parents[N] pointed into site-packages after an install, so run state moved into the installed package. Packaged resources such as bundled_skills/ deliberately still resolve relative to the package.
  • make targets prefer the repository virtualenv, so make check uses the same interpreter as ./init.sh instead of whatever python is first on PATH.
  • Development-only files (process plans, defect ledgers, session notes) removed; ./init.sh is now a single offline verification entry point.
  • Studio uploads accept only csv/parquet, matching what publication accepts; a rejected database file explains why.
  • docs/ reorganised; its index no longer points two entries at the same page.

Documentation

  • Both READMEs state measured behaviour with its sample size and its caveats, and stale figures are corrected: 928 tests (was 691/806), 23/23 tier-1 tasks over 32 gold tasks across 3 schemas (was 32/32), and a real-model NL2SQL result (was "not run here"). Claims about automatic skill selection and the PostgreSQL backend are marked unestablished and unverified rather than implied working.
  • docs/nl2sql_evaluation.md records the frozen baselines, and its token/cost section no longer describes a heuristic the code no longer uses.

Verification

Check Result
./init.sh 928 tests, 25 skipped, 0 failures
make check 13/13 offline acceptance checks
make web-check eslint + tsc + production build + 24 TypeScript tests
python scripts/check_repository.py no broken links, no secrets, hygiene clean

Not verified

  • No tier-3 real-model run was repeated after these changes. Accuracy remains 0.875 semantic correctness / 1.0 execution success on 40 anime cases from a single run, where ±0.03 is noise across identical code. No accuracy improvement is claimed.
  • The PostgreSQL backend is still unverified against a live server and is not exported from the package API.

…ect the docs

Clears every defect in the review ledger except the provider-deadline item, fixes the
documentation claims that had drifted from the code, and removes development-only
files from the repository.

Correctness

- Unified terminal outcomes. `succeeded` / `needs_clarification` / `blocked` /
  `partial` / `failed` / `cancelled` are derived in `queryforge/core/outcomes.py`,
  so the persisted run status, the delivery report and the event stream can no
  longer disagree. A quality gate that blocks a run is no longer reported as
  completed.
- Reflection that asks for clarification used to raise, discarding the answer and
  the run's artifacts. It now terminates as a structured `needs_clarification`.
- The `reasoning` audit payload was silently dropped whenever the model returned
  type-compatible but schema-incompatible shapes (string lists, the string
  "None", word confidences such as "high"). Shapes are normalised and a discarded
  payload is now reported instead of ignored.
- Preview execution evaluated a rewritten statement, so a `require_limit` policy
  was satisfied by the injected LIMIT rather than by the caller's SQL.
- Time-filter validation ignored `BETWEEN` and per-call boundaries, so a partially
  bounded time filter could validate as complete.
- Truncated result sets overwrote the row count, hiding that the bound had been
  hit; `truncated` and `fetched_row_count` now preserve both numbers.
- Chinese follow-up questions were not recognised, so multi-turn context was lost.
  Route matching for report/explain intents is no longer triggered by a bare
  substring, so a table named `sales_report` is not routed as a report request.

Governance and audit

- `DatabaseTool.last_policy_decision` is read-only. It can no longer be overwritten
  by a caller, which had let the tool report an audit record for a call that never
  happened.
- Execution fingerprints bind data, semantic and policy versions, persisted in the
  run journal, so a resumed run cannot silently reuse a step computed against a
  different model or policy.
- Budget state is inherited across a resume instead of restarting at zero.
- Cross-instance journal leases are exclusive (`fcntl.flock` plus a reload before
  acquiring), so two workers can no longer hold the same lease.
- CSV knowledge is trusted only when a reviewer is recorded; the governed
  knowledge build path is reachable from the CLI (`--kb-knowledge`).

Measurement

- The evaluator compares result projections tolerantly: a correct answer is no
  longer scored wrong for returning extra columns or an equivalent shape.
- Account-level failures (HTTP 402 and friends) are classified as environment
  errors and excluded from accuracy denominators instead of being counted as model
  failures.
- Token usage is reported as measured when the provider returns it, with
  `token_source` distinguishing measured from estimated figures.
- `--skill-mode auto|off` and `--parallel-candidates` make the skill-selection and
  candidate-count questions testable; the `skills=[]` default that silently
  disabled the skill catalogue in every measured run is gone.
- The evaluator's isolation guard is recursive and covers plain `import`
  statements.

Repository and structure

- Workspace-relative paths (`.queryforge/` state, sample data, evaluation sets) are
  resolved by `queryforge/core/paths.py` from `QUERYFORGE_ROOT`, an enclosing
  source checkout, or the working directory. Deriving them from
  `Path(__file__).parents[N]` pointed into `site-packages` after an install, so run
  state moved into the installed package. Packaged resources such as
  `bundled_skills/` deliberately still resolve relative to the package.
- `make` targets prefer the repository virtualenv, so `make check` uses the same
  interpreter as `./init.sh` instead of whatever `python` is first on `PATH`.
- Development-only files (process plans, defect ledgers, session notes) are removed;
  `./init.sh` is now a single offline verification entry point.
- Studio uploads accept only csv/parquet, matching what publication can accept; a
  rejected database file explains why. `docs/` is reorganised and its index no
  longer points two entries at the same page.

Documentation

- Both READMEs state measured behaviour with its sample size and its caveats, and
  the stale figures are corrected: 928 tests (was 691/806), 23/23 tier-1 tasks over
  32 gold tasks across 3 schemas (was 32/32), and a real-model NL2SQL result (was
  "not run here"). Claims about automatic skill selection and the PostgreSQL
  backend are marked as unestablished and unverified rather than implied working.
- `docs/nl2sql_evaluation.md` records the frozen baselines, and its token and cost
  section no longer describes a heuristic the code no longer uses.
- CHANGELOG.md lists the above.

Verification

- `./init.sh` — 928 tests, 25 skipped, 0 failures
- `make check` — 13/13 offline acceptance checks
- `make web-check` — eslint, tsc, production build, 24 TypeScript tests
- `python scripts/check_repository.py` — no broken links, no secrets, hygiene clean

Not verified: no tier-3 real-model run was repeated after these changes, so accuracy
remains 0.875 on 40 anime cases from a single run, where +/-0.03 is noise. The
PostgreSQL backend is still unverified against a live server.
@lanerchenbuna
lanerchenbuna merged commit d1e1e4f into main Sep 22, 2026
10 checks passed
@lanerchenbuna
lanerchenbuna deleted the fix/defect-ledger-and-docs branch September 22, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant