Skip to content

Harden secret handling, output paths, and HTTP timeouts - #1

Open
devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/1785356339-security-hardening
Open

Harden secret handling, output paths, and HTTP timeouts#1
devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/1785356339-security-hardening

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Jul 29, 2026

Copy link
Copy Markdown

Summary

Security pass over the pipeline. No committed secrets were found (worktree and full git history are clean); the fixes below address the real issues that were present. main's shared-utils refactor (#2) is merged in, so the classifier now uses require_env from src/env_utils.py rather than a bespoke key loader.

  • Placeholder API key masked missing credentials. openrouter_classifier.__main__ did os.environ.get("OPENROUTER_API_KEY", "your-api-key-here"), so an unset key silently sent a bogus Authorization: Bearer your-api-key-here header to OpenRouter instead of failing. Now (API_KEY,) = require_env("OPENROUTER_API_KEY"), matching the other entrypoints.

  • Untrusted filenames flowed into output paths. _save_image / _save_json interpolated document_name — derived from input filenames and dataset directory names — directly into a path:

    # before: document_name="../../evil" writes outside output_dir
    image_path = self.images_dir / f"{document_name}_page_{page_num:04d}.png"
    # after
    image_path = self.images_dir / f"{sanitize_document_name(document_name)}_page_{page_num:04d}.png"

    sanitize_document_name collapses anything outside [A-Za-z0-9._-] to _, strips leading/trailing ./_, caps length at 200, and raises on names with no usable characters. Normal dataset names (letter_0000123, scientific_report_...) are unchanged.

  • Two unbounded HTTP calls. braintrust_metrics_visual.fetch_experiment_results called requests.get for /project and /experiment with no timeout (the paginated POST already had one) — a stalled connection hangs the script forever. Added timeout=60.

  • .gitignore secret coverage. .env.local only; now .env.* (keeping !.env.example) plus *.pem, *.key, credentials.json, kaggle.json — the Kaggle download path makes a stray kaggle.json plausible.

Not applicable / not found: no SQL, database, or web/API server exists in the repo, so there is no SQL injection, CORS, debug endpoint, or authentication surface. pip-audit on requirements.txt reports no known vulnerabilities. Remaining non-critical note, left alone deliberately: most requirements use unbounded >= ranges (Pillow>=10.4.0, requests>=2.32.0, ...), which makes builds non-reproducible and exposes the project to a bad upstream release — worth pinning, but that is a dependency-policy decision rather than a fix.

Link to Devin session: https://app.devin.ai/sessions/a3c51c3c047e4e20b46efb208ca46daa
Requested by: @Exios66

@Exios66 Exios66 self-assigned this Jul 29, 2026
@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Exios66 added a commit that referenced this pull request Aug 4, 2026
- v16 = v11.9 (chart fix) + 2 worked examples (budget/invoice, handwritten/letter)
- v1 (test_images source): 154/160 (96.2%) — regressed from v11.8's 99.4%
- v2 (HF mirror): 134/160 (83.8%) — slightly below v14's 85.0%
- v3 (HF mirror): 127/160 (79.4%) — slightly below v15's 81.2%

Infra improvements:
- Added 300s HTTP timeout to prevent hung eval runs
- Failed rows now return ERROR_PREFIX sentinel + tracked  score metric
- All runs use --manifest for resumability

Key pain points:
- 16 failed rows (13 finish_reason=length, 3 provider errors)
- v2/v3 HF-mirror source ~15pp harder than v1 test_images source
- Worked examples did not help (handwritten→letter #1 error despite example)
- v11.8 remains the gold-standard prompt at 99.4%
Exios66 added a commit that referenced this pull request Aug 6, 2026
* Add unit test suite and coverage tooling

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* New Experiments & v16

qwen3.7-flash_v11_8_reasoning_160_t0_3: 157/159 (98.7%), temp 0.3, reasoning high. Failure: wat19d00 (no content). Misses: jed71e00 form→presentation, tqi16e00 budget→invoice (regressed vs temp 0.1's 99.4%).
qwen3.5-35b-a3b_v11_8_reasoning_160: 155/157 (98.7%), temp 0.1, 8192 max_tokens. Failures: mvr50f00, iby31c00, umv76d00. Misses: jed71e00, noz90d00 form→advertisement. Actual cost $0.8280.
qwen3.5-35b-a3b_v11_8_reasoning_v12retro (dataset qwen_v12_retroactive_eval, id dcd90cb4-38eb-4a37-870f-3c5443c6d648, 52 rows): 16/52 (30.8%). 5 failures all finish_reason=length: rvl_cdip__form__0005.png, rvl_cdip__invoice__0006.png, rvl_cdip__presentation__0011.png, rvl_cdip__questionnaire__0005.png, rvl_cdip__scientific_report__0016.png.
gemini-2.5-flash-lite_v11_8_reasoning_160-d558f2bc: 139/160 (86.9%), temp 0.2, effort max, zero failed rows. Top misses: memo→specification (3), scientific_publication→scientific_report (2), form→specification (2), budget→invoice (2); notably jed71e00 NOT missed. Actual cost $0.1134.
Reports generated: reports/report_qwen3.7-flash_v11_8_reasoning_160_t0_3.md, reports/report_qwen3.5-35b-a3b_v11_8_reasoning_160.md, reports/report_qwen3.5-35b-a3b_v11_8_reasoning_v12retro.md, reports/report_gemini-2.5-flash-lite_v11_8_reasoning_160-d558f2bc.md, each with confusion_matrix_*_v11_8_*.{md,png}, misclassification_reasoning_*_v11_8_*.md, per_class_accuracy_*_v11_8_*.png.
Manifests: reports/manifests/eval_v11_8_t0_3.jsonl, eval_v11_8_qwen35.jsonl, eval_v11_8_qwen35_v12retro.jsonl, eval_v11_8_gemini_lite.jsonl (all 160/160 or 52/52); eval_v11_8_kimi.jsonl (109/160 cached, resumable).
Docs updated: docs/experiments/experiment_log.md (new "Cross-model v11.8 runs" section + gemini/kimi-abort entries), CHANGELOG.md (Added section with temperature/reasoning flags and results), docs/CHANGELOG.md (v11.8 cross-model table with findings).
kimi killed (PIDs 10089, 10029, 10090 terminated; final gemini resumed as PID 10520 and completed).

* v16 multispect evaluation: 96.2%/83.8%/79.4% on v1/v2/v3 slices

- v16 = v11.9 (chart fix) + 2 worked examples (budget/invoice, handwritten/letter)
- v1 (test_images source): 154/160 (96.2%) — regressed from v11.8's 99.4%
- v2 (HF mirror): 134/160 (83.8%) — slightly below v14's 85.0%
- v3 (HF mirror): 127/160 (79.4%) — slightly below v15's 81.2%

Infra improvements:
- Added 300s HTTP timeout to prevent hung eval runs
- Failed rows now return ERROR_PREFIX sentinel + tracked  score metric
- All runs use --manifest for resumability

Key pain points:
- 16 failed rows (13 finish_reason=length, 3 provider errors)
- v2/v3 HF-mirror source ~15pp harder than v1 test_images source
- Worked examples did not help (handwritten→letter #1 error despite example)
- v11.8 remains the gold-standard prompt at 99.4%

* v17 prompt: simplified financial rules + handwritten-letter override

Data-driven response to three root causes from v16 multispect evaluation:

1. Provider failures (16 rows, 13 finish_reason=length):
   - Trimmed check-7 from 6284 to ~1100 chars (removed agency-estimate sub-protocol)
   - Prompt is 5476 chars lighter than v16 (46277 vs 51753)
   - Reasoning effort reduced to "medium" for qwen (was "high")
   - MAX_TOKENS_CAP raised to 32768 (was 16384)
   - 300s HTTP timeout on OpenAI client

2. Slice hardness gap (~15pp v1 vs v2/v3):
   - Stronger rules for robustness across image sources
   - Failed rows now tracked as "failed" metric in Braintrust

3. Prompt regression from v11.8 (99.4%):
   - Removed entire agency-estimate sub-protocol that caused budget-invoice confusion
   - Simple rule: estimate = budget (planning), only explicit payment demand = invoice
   - Added LETTER/MEMO OVERRIDE in check-2: handwritten wins over letter/memo formatting

v17 = v11.9 with surgical string replacements (finds anchors via .find() at import time)

* Create create_smoke_v11_8_16_dataset.py

v17 Additions in progress - Return to structure of v11.8 prompting.

* v17 evaluation: 153/160 (95.6%) on v1 + v17.1 fix for AT&T MONTHLY INVOICE

v17 results (v1 slice):
- 153/160 (95.6%), 1 failed row, 6 misclassifications
- handwritten→letter: 0 errors (v16 had 3) — LETTER/MEMO OVERRIDE works
- budget→invoice: 2 errors (AT&T MONTHLY INVOICE wrongly classified as invoice;
  model matched "INVOICE" header in invoice bullet before reaching budget carveout)
- invoice→budget: 2 new errors (regression from v16)
- form→presentation: 1 (persistent)
- memo→scientific_report: 1 (new)

v17.1 fix:
- Moved periodic-statement carveout from budget bullet into invoice bullet
- Model now reads: "BUT a provider periodic customer statement that says MONTHLY
  INVOICE at the top... is budget, not invoice" BEFORE stopping at the invoice match

Config:
- Qwen reasoning restored to "high" (was "medium" in initial v17 draft — user-requested
  after seeing AT&T invoice misclassification with medium reasoning)

* Create AGENTS.md

* Update AGENTS.md

* v17.1 enhancements

* v17.2 Updates

* Add v17.2 document classification calibrations

Refine v17.2 prompt enhancements with five new clarifications to reduce common misclassifications:

- Invoice: emphasize payment demand signals over form-like layout
- Budget: clarify as internal spending documents, not payment demands
- Publication: require journal/publication metadata on the same page
- Specification: distinguish requirement definitions from data-recording forms
- Scientific_report: require narrative prose interpretation, not standalone data tables

These calibrations target recurring false positives across slices where document layout or data content can blur class boundaries.

* Move scripts to subpackages; add cache & retry

Restructure scripts/ into subpackages (braintrust, datasets, eda, openrouter) and update tests to new import paths. Enhance scripts/braintrust/create_v11_v17_eval_dataset.py: add JSON cache load/save for deduped records, a --cache flag, and upload_rows_with_retry to retry dataset uploads on transient failures. Tweak prompt assembly in src/prompts.py (v17 build and calibration addendum). Update tests to reflect renamed helpers (encode_image_base64, use image_utils._pad_color_for_mode) and the new default prompt version.

* Add dataset copy script; improve parquet download

Add scripts/braintrust/copy_datasets_to_new_env.py to copy Braintrust image datasets between orgs/projects. Uploads attachments synchronously with retries, inserts rows only after confirmed upload, and offers verification, delete-existing, and CLI-specified source/destination keys and projects. Improve scripts/braintrust/create_braintrust_800_dataset.py: stream source parquet to a .part file and atomically rename (os.replace) to avoid partial-cache corruption, add --cache-dir, and guard cleanup so partial state isn't mistaken for a valid cache. Add PROMPT_V0 to src/prompts.py and register "v0" in PROMPTS (short function-focused prompt).

* Benchmark Evaluations v0 and v11.8

* Organizational Updates

* Add manifest tools, retry logic, and completion sounds

Introduce local tooling and robustness for Braintrust evals: add src/notify.py (macOS success/failure jingles), scripts/braintrust/score_manifest.py (compute final metrics from local manifest, write JSON/MD), and scripts/braintrust/resume_until_complete.py (loop runner until expected rows complete). Enhance braintrust_openrouter_input.py with adaptive throttling, OpenRouter key failover, quota/rate/content-filter handling, optional fallback model, safe span logging, and completion-sound support (--no-sound / --fallback-model). Update CHANGELOG and add final report artifacts for a qwen3.7-flash run.

* Add near-miss (runner-up) scoring

Record and score ‘near-miss’ cases where the model's runner-up (second-choice) label matches the ground truth. Adds extract_runner_up and near_miss_score helpers, tracks runner_up_by_index during eval, logs runner_up in row metadata, and registers a near_miss code scorer in braintrust_openrouter_input.py. Updates score_manifest.py to compute near_miss counts, accuracy and filenames. Purpose: surface cases where the model nearly chose the correct class (zero-cost regex-based scorer for better error analysis).

* Add cost/failure scoring, backfill & reporting

Capture per-row OpenRouter billed cost and unify failure naming; compute near-miss (runner-up) and cost locally with backfill from Braintrust traces. Changes: capture usage.cost and runner_up in scripts/braintrust/braintrust_openrouter_input.py and expose as Braintrust metadata; add scripts/braintrust/braintrust_report_manifest.py to generate full reports from a local manifest + merged Braintrust traces; extend scripts/braintrust/score_manifest.py to backfill runner_up/cost (honoring braintrust.env), report failure rate and cost stats, and produce final JSON/MD; increase experiment fetch page size to 1000 in src/braintrust_utils.py. Docs/CHANGELOG/AGENTS updated and example report artifacts amended.

* Agents.md Enhancements

* Update AGENTS.md

* Monte Carlo Additions

* Update prompt_ablation.md

* Visualization Enhancements

* Posit Cloud Site Updates

* Website Enhancements v2

* Fix markdown table formatting in confusion pairs

The confusion pairs summary table rows were being joined without newlines, causing all rows to render on a single line. Changed `"".join()` to `"\n".join()` to properly format the markdown table with each row on its own line.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Add walkthrough notebooks and builder script

Add three capstone walkthrough notebooks (notebooks/ and website/notebooks/): environment & single-image, balanced sampling & Braintrust upload, and watchers/evaluators/full experiment. Add scripts/site/build_notebooks.py to programmatically generate and mirror those notebooks as nbformat v4 JSON for the site. Minor code tidy: add from __future__ import annotations to scripts/braintrust/braintrust_openrouter_input.py, src/braintrust_config.py, and src/braintrust_utils.py. These changes introduce documentation artifacts and a generator to keep them reproducible and in-sync with the repo.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Add GitHub issue/PR templates & publish config

Add repository contribution templates and a site publish config. New files: .github/ISSUE_TEMPLATE/{bug_report.md,feature_request.md,eval_report.md,config.yml}, .github/PULL_REQUEST_TEMPLATE.md, and website/_publish.yml. CHANGELOG.md updated to document the templates. Templates encode repo conventions (preflight checks for evals, prompt changelog links, no secrets or generated output committed, testing checklist) and surface links to the experiment log and changelogs. website/_publish.yml configures Posit Connect publishing for the project site.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Add research memos and update website navigation

Add a set of research memos (accuracy-arc, confidence-routing, cost-per-image, ensemble-voting, exemplar-mining, failure-pipeline, generalization-falloff, hardest-classes, hasty-stop-words, model-comparison) and wire them into the site. Update website/_quarto.yml to expose a Research Memos section, enable enhanced search/TOC/sidebar behavior and page/navigation features. Add landing page metadata and GitHub action buttons in index.qmd. Extend assets/css/custom.scss with styles for memo pages, floating sidebar, margin TOC, and page chrome polish.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Add sources citations to website memos

Append a {.memo-sources} provenance block to several website memos to improve traceability and references. Affected files: accuracy-arc.qmd, confidence-routing.qmd, cost-per-image.qmd, ensemble-voting.qmd, exemplar-mining.qmd, failure-pipeline.qmd, generalization-falloff.qmd, hardest-classes.qmd, hasty-stop-words.qmd, and model-comparison.qmd. Each block references the experiment log, relevant monte_carlo/report artifacts, and the repository so readers can follow the data and analysis behind the conclusions.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Add script to build website experiment data

Adds scripts/site/build_experiment_data.py which parses committed markdown reports (docs/experiments/experiment_log.md, reports/experiment_reports/*.md, docs/experiments/1pic_cost_estimation.md, reports/confusion_matrices/*.md) to produce the website's interactive data layer offline. Emits four JSON assets into website/data/: experiments.json, cost-models.json, per-class-accuracy.json, and confusion-matrices.json. Designed to be deterministic and require no API access; used to drive Observable/ipywidgets pages and notebooks.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Site Updates

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Posit Cloud Site Polishing

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Add manuscript lit review, acknowledgements, executed notebooks; fix confusion matrices

- Rewrite Related Work into five verified threads (LayoutLM family, non-neural
  classifiers, hybrid OCR-LLM, small-model specialization, unified small-model
  wave + cheap serving) with 27 new verified bibliography entries
- Add Acknowledgements section (UW-Madison affiliation, Siddharth Suresh, DSHB,
  CHTC, Dr. Timothy Rogers, Dr. Caitlin Roa)
- Execute and mirror walkthrough notebooks 01-04 with real outputs
- Fix truncated/duplicated confusion-matrix headers in build_site.py and
  braintrust_report.py; patch 16 committed matrix md sources; regenerate charts
- Refine editorial typography in custom.scss (system fonts, prose measure,
  balanced headings, styled captions) keeping the navy palette

* Wiki rebuild: full site mirror with APA formatting + charts

* Apply APA 7 formatting to CHANGELOG statistical expressions

* Create bug_report.yml

* Update CHANGELOG: v1.0.0 release & site additions

Add comprehensive release notes and website/content updates: manuscript literature review, acknowledgements, executed notebooks (01–04), website experiment data layer, research memos, PR/issue summaries, and GitHub templates. Fix confusion-matrix headers and regenerate SVGs; Posit Cloud site rebuild and editorial typography tweaks. Record v1.0.0 release (2026-08-05) and document evaluation/runtime improvements (truncated-response salvage, finish_reason logging, reduced reasoning, raised MAX_TOKENS_CAP, 300s HTTP timeout, manifest support, resilient retry/fallback model, and completion alerts).

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Polish website docs: callouts, formatting, findings

Refine website QMD pages for clarity and emphasis. Added subtitles, callout-note/warning/important blocks, typographic fixes, inline code formatting for class names, and table/value styling across cost, methods, manuscript, montecarlo, and results pages. Clarified cost methodology (measured vs list-price), pixel-hash dedup rules, prompt-engineering claim and caveats, and noted measured negative results for routing/exemplar. Document-processor page now lists system deps and output specs; CLI guide highlights env/key safety. These edits improve readability and surface key reproducibility and risk notes.

* Add prompt-sensitivity literature and refs

Weave LLM prompt-sensitivity literature into the manuscript: add a sixth related-work thread and insert citations in the Introduction, Methods (role-framing), Discussion (validation discipline), and Limitations (prompt drift). Add 10 verified bibliography entries to website/references.bib and update CHANGELOG.md to document the addition. Purpose: acknowledge prompt-engineering research and justify the paper's role-framing and versioned-prompt validation approach.

* Add appendix materials & sources index

Link every artifact behind the misclassification analysis in the
appendix: full 1,120-image trace report, all 13 failed_reasoning_traces
reports, early-run traces, confusion matrices (MD+PNG+directory), per-class
chart, experiment reports, raw manifests, eval logs, Monte Carlo corpus,
interactive website/data layer, and source code (prompts, evaluation,
monte_carlo, runner). All 21 file paths verified to exist; page renders
warning-free. Update CHANGELOG.

* VIZUALIZATION OVERHAUL

* Fix chat traces page formatting, explorer widgets, and chart label clipping

- chats.qmd: escape illustrative thread scratchpad/label tags (previously
  parsed as HTML), real arrow glyph in inline code, add title frontmatter,
  fetch the 8 missing source document images from Braintrust attachments
- experiment-explorer: per-class + confusion charts use report-style 6-char
  abbreviations at -40/-45 degrees with proper margins so tick labels never
  clip; model bars + cost calculator gain x-domain headroom so value labels
  stay in frame; filter/table styling
- CHANGELOG updated

* Add theme switcher, favicon, and site chrome

Add site UI assets: a pre-parse theme-head (FOUC guard + skip-link), a theme-switcher (navbar dropdown, localStorage persistence, keyboard support, site-theme-change event), and site-chrome (reading progress bar + minor polish). Adds favicon.svg and standalone JS files under website/assets for modular use. Also ensures generated tables get proper scope attributes for accessibility. These changes enable light/dark plus two custom themes and improve keyboard/screen-reader support.

* Trace-language viz: fix node aggregation, scope-suffixed outputs, regenerate site charts

- _node_stats aggregates per-word token stats (was silently keying on the
  literal 'a'/'b'), restoring node size/color encoding in the phrase net
- Scoped runs (--confusion-pair/--prompt-version/--class) now suffix every
  output (logodds, scatter, loop inventory, report) so they never clobber
  the default all-scope artifacts; adds letter->memo example outputs
- Fix report links to point at the suffixed loop inventory
- Regenerate trace-language SVGs (logodds_dirichlet, scattertext_style
  rasterized to shrink SVG, phrase_net_differential) and the site page

* Fix chart legibility, corpus confusion pairs, light-default theme, and pluggable env

- f3_model_comparison: 5 best-per-model bars with two-line labels, data
  labels above Wilson CI caps, baseline annotation inside canvas
- accuracy_progress: shortened labels, larger canvas, legend moved below
- cost charts (f6 + cost_projection_models): 25-degree right-aligned ticks,
  9pt fonts
- monte_carlo_corpus: confusion_pair only set for genuine misses; 4,641-row
  committed corpus patched in place (3,724 rows), top-20 pairs now real
  confusions (letter->memo 53, budget->invoice 52)
- theme-head.html FOUC guard defaults to Light; OS prefers-color-scheme
  ignored (site ships inlined copy)
- requirements.txt: add pyarrow + networkx, document Tesseract/Poppler
- .venv.example: pluggable OpenRouter/Ollama/vLLM + Braintrust template
- openrouter_utils: OPENROUTER_BASE_URL env override for OpenAI-compatible
  local backends

* Continued Site Polishing

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Visualization Recalibration

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Posit Site Most Recent Version

08-06-2026

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Update CHANGELOG.md

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Fix forest-plot data labels: place delta values outside 95% CI caps

- prompt_ablation.svg: value labels moved from on-bar (d+0.004, overlapping
  the bar fill and CI line) to 0.008 past the outer CI cap, right/left-aligned
  by sign, 9pt bold, with an explicit xlim so nothing clips
- phrase-net static fallback: widen label collision box (px_per_unit 0.008)
  so adjacent word labels cannot touch
- CHANGELOG: interactive phrase-net widget + trace-language chart legibility
  overhaul entries

* Archive deprecated scripts and remove dead code

Move several one-off/obsolete scripts and raw chat caches into archive/ (history preserved) and add archive/README. Update docs and site references (AGENTS.md, README.md, docs/CLI_COMMANDS.md, scripts READMEs, website QMD) to point at archived paths. Remove unused symbols: classify_row_status (src/evaluation.py), top_labels (src/monte_carlo.py), and VISION_MODELS (src/openrouter_classifier.py). Add ipython to requirements-dev.txt and adjust build_chat_examples cache to archive/chat_data. CHANGELOG updated to note the cleanup.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Repo declutter: archive deprecated scripts and caches, remove dead code

- Move 7 superseded/orphaned scripts to archive/ (git-mv, history preserved):
  create_smoke_v11_8_16_dataset, create_v115_eval_dataset,
  create_v115_v12_eval_dataset, copy_braintrust_dataset,
  run_v11_8_800_after_480 (braintrust); build_wiki (site);
  download_dataset (datasets)
- Move raw chat-thread caches (~26 MB) from reports/chat_data/ to
  archive/chat_data/; point build_chat_examples.py CACHE_DIR at it
  (committed site output website/data/chat-examples.json untouched)
- Remove dead code: classify_row_status (src/evaluation.py),
  top_labels (src/monte_carlo.py), VISION_MODELS (src/openrouter_classifier.py)
- Add ipython to requirements-dev.txt (build_notebooks.py imports it)
- Repoint all doc references (AGENTS.md, README.md, docs/CLI_COMMANDS.md,
  scripts READMEs, website/methods/cli-commands.qmd); add archive/README.md;
  fix stale README claims (prompt range v1->v17.2, committed reports)
- Update CHANGELOG.md

* Stop tracking Quarto render artifacts (website/.quarto, website/site_libs)

Render outputs accidentally swept in by git add -A; now gitignored alongside
website/_site/.

* Fix stale tests and add Posit-site/Quarto/SVG-legibility test suite

- Fix 7 stale failing tests instead of deleting them:
  * TestFetchExperimentResults rewritten against the config-based API
    (load_braintrust_config + mocked requests); old tests referenced a
    removed PROJECT_NAME constant and hit the real Braintrust API with
    401-retry backoffs (~80s of network retries per run)
  * TestCollectClassImages extensions corrected from globs to dotted
    extensions matching the _collect_class_images contract
  * test_document_processor / test_run_tiff_processing now
    pytest.importorskip pdf2image (2 clean skips vs collection errors)
- Add tests/test_site_and_charts.py: SVG legibility (anchor-aware
  text-collision scan across all 56 committed charts/figures, phantom
  geometry excluded — zero genuine collisions), Quarto assets
  (_quarto.yml pages, front-matter resources, vis-network widget,
  theme assets, data JSON structure, chat-example records), and site
  integrity (internal links/images resolve, bib citations exist,
  SCSS theme tokens balanced)
- Add tests/test_module_gaps.py: constants, prompts registry,
  env_utils.require_env, image_utils, hermetic braintrust_config
  (defaults/overrides/empty fallback)
- Full suite: 380 passed, 2 skipped, ~4s (was 185 passed, 7 failed,
  2 collection errors, ~90s)
- Update CHANGELOG.md

* Fix phrase-net tooltip rendering and restyle trace-language charts (mint palette + legend)

- Tooltips: vis-network 9.1.9 renders string titles via innerText, so <b>/<br>
  markup showed literally; titles now pass through tooltipHtml() as DOM
  elements, with a regression test asserting both node and edge titles route
  through it
- Palette: trace-language charts (phrase net widget + SVG, log-odds bars,
  scattertext) swap saturated green/red for a relaxed mint->coral gradient
  (66c4a8 -> e08580); softened node borders, muted gold leak diamonds,
  coral structural-loop edges
- Legend: static phrase_net_differential.svg gains a 'Phrase net' legend;
  widget footer legend restyled to match

* Complete design-token theming; dark themes lose white chart-frame/navbar highlights

- Tokenize remaining hardcoded colors into per-scheme variables: navbar
  text/hover/border/shadow, hero text/badges, and the dark-mode Pandoc
  syntax palette (--syn-*), with jade/synthwave accent flavors
- Dark themes: --chart-frame is now a dark surface per theme with a dark
  --chart-shadow (figures sit in the page instead of glowing white boxes);
  navbar white hairline border replaced by dark border + deeper shadow
- Light theme rendering unchanged; SCSS token-balance test + quarto render pass

* Trace Chart Widget Addition

'language.qmd` now embeds a d3 widget (library vendored at `website/assets/js/d3.v7.min.js`, word-grid data at `website/charts/scattertext_style.json`, both declared as Quarto `resources`) as the primary artifact under **Scattertext-style Frequency Scatter**, matching the phrase-net widget pattern. Features: log-log grid (0.3 → 10⁴) with the same bias colors (`#e08580` failure-biased / `#66c4a8` success-biased / `#b9c4d6` neutral) and dot sizing by trace frequency, gold-bordered diamonds for prompt-leaked words, hover tooltips with raw counts / log-odds z / per-million frequencies, wheel-zoom + drag-pan, click-to-pin word labels, a search box (live dim + ring highlight, Enter jumps to the first match), legend chips that toggle bias/leak visibility, and a Reset view button. Words off the static grid are pinned to the grid edge instead of dropped. Data is emitted by the same pipeline as the SVG (`chart_trace_scatter_data` in `build_site_charts.py`), so interactive and static cannot drift; the static SVG remains beneath as a no-JS/print fallback. Widget smoke-tested headlessly (render, pin, tooltip, search, filter, reset) with the real page markup.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Theme Bootstrap components + search + widget chrome for all four schemes

- Quarto compiles Bootstrap 5.3 component vars to literal light colors, so
  dropdowns/modals/cards/offcanvas/popovers stayed #f6f8fb in dark themes;
  re-point .card/.dropdown-menu/.modal/.offcanvas*/.popover --bs-* vars at
  theme tokens (canvas/card/border + accent link states), tokenize mark
  (--highlight-bg) and kbd (--kbd-bg/--kbd-fg) with light-exact values
- Re-theme the Algolia/Quarto search surface for all schemes: form, input,
  clear/copy/cancel controls, results panel, detached mobile container +
  overlay, result rows, source headers, selected state, search-match marks
- Theme interactive widget outer chrome (phrase-net + scattertext frames,
  legend/toolbar bars, toolbar controls) via !important overrides while
  keeping the canvases white for ink legibility; table row tints and the
  Quarto sidebar-navigation border follow tokens
- Sticky-layout audit: nav-fixed 64px offset, headroom pin/unpin, z-index
  chain 3000->1000 all verified sound; no changes required

* Phrase-net widget shows both sides of the classification coin

- Add differential_bigrams_two_sided: returns failure-biased (z>=min_z)
  AND success-biased (z<=-min_z) differential bi-grams, each tagged with a
  bias field, dominant-class min-count, per-side cap, sorted by |z|
- build_site_charts: extract shared _graph_from_edges (tags every edge with
  bias; structural loops detected over the failure subgraph only so loops
  stay a failure-only diagnostic); widget JSON now uses _trace_graph_widget
  while the static SVG keeps the one-sided graph
- Widget JS: nodes size by total trace count (success-dominant words now
  render mint), edge width by |z|, edges colored by bias (mint=success,
  coral=failure, dark coral=loop), tooltips tag each connection type,
  legend gains success/failure/loop edge swatches
- Regenerated phrase_net_differential.json (707 words / 600 edges, 300 per
  side) and trace-language.qmd; data validated (0 schema violations, 0
  z/bias inconsistencies, both dominance classes)
- Tests: TestDifferentialBigramsTwoSided (4 cases) + widget JSON asserts
  both biases and both node dominance classes; 386 passed

* Site polish: self-hosted fonts, widget font tokens, chat page formatting

- Add Google-fonts woff2 files (Source Sans 3 / Source Serif 4 / Source
  Code Pro) under website/assets/fonts/
- Widgets use the --font-sans token (canvas + SVG faces -> Source Sans 3)
  and drop leading indentation from generated HTML strings
- Regenerate scattertext_style.json and chats.qmd from current sources
- gitignore the transient session-ses_02a1.md artifact

* Complete theme tokens; fix dark-mode highlights

Tokenize remaining hardcoded colors and remove white highlights in dark themes. Added/normalized design tokens (navbar, hero, chart-frame/chart-shadow, kbd/mark, syntax --syn-*) and re-pointed Bootstrap component custom properties (.card, .dropdown-menu, .modal, .offcanvas, .popover, tables) at those tokens so dark themes render with dark frames/shadows instead of white boxes. The Algolia/Quarto search UI now has dark-mode overrides. Minor scattertext widget and chat QMD/HTML formatting edits. Ran site render and tests; SCSS token-balance and test suite pass.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Normalize notebook cell sources for site build

Add _src_lines helper and update site notebook generator to emit nbformat-compliant list-of-line sources so Quarto renders headings, lists and code correctly. notebook() gained optional description/frontmatter and a static-render callout to avoid duplicated H1s. Imported re and switched code/md/raw to use _src_lines. Regenerated notebooks under notebooks/ and website/notebooks/ to the new format.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Notebook pages: list-form sources & site sync

Write notebook cell sources as nbformat list-of-lines and emit subtitle frontmatter so Quarto renders notebooks correctly. Update scripts/site/build_notebooks.py to produce list-form sources and subtitle, adjust the four notebooks and their website/ mirrors, and add TestNotebookPages in tests/test_site_and_charts.py to guard the contract (sources are lists, frontmatter has title+subtitle, opening cell framed, mirrors match). Update CHANGELOG with the rationale: Quarto flattens single-string sources and strips in-cell newlines, breaking rendered notebooks.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Delete _publish.yml

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Create documentation.yml

Addition of new issue & PR templates

* Add Model & Prompt A/B comparator

Introduce an A/B comparator for model/prompt reasoning: new build script (scripts/site/build_model_ab.py) that emits website/data/model-ab.json, a vendored widget (website/assets/html/model-ab-widget.html), and thumbnails (website/chat_images) plus integration into the chat page via scripts/site/build_chat_page.py and website/chats.qmd. Add tests (tests/test_site_and_charts.py) to validate data shape, thumbnails, and widget column-0 injection. Also add archive/model_ab/ to .gitignore and fix nested Quarto sidebar groups in website/_quarto.yml. CHANGELOG updated with feature notes.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

* Add agent Braintrust account & scorer control

Introduce support for a separate "agent" Braintrust account and runtime control over Braintrust scorers. Adds load_agent_config() and a common env resolver in src/braintrust_config.py, new --agent/--no-scorers/--scorers CLI options in scripts/braintrust/braintrust_openrouter_input.py, and logic to register only selected scorers (agent defaults to exact_match). Update .env.example and braintrust.env.example and document the feature in AGENTS.md. Wire --no-scorers through resume_until_complete.py. Add tests for agent key handling and scorer parsing, and include a new website/_publish.yml asset. These changes let evaluations run under a minimal agent profile (reducing Braintrust scorer work) or run with no remote scorers and score locally.

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>

---------

Signed-off-by: Lucius Morningstar <148591095+Exios66@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant