[Aaron LaBeau (biozal)]

Prepares the v0.2.0 release. Draft until review is done; tagging v0.2.0 after merge publishes to npm, GitHub, and Homebrew.

What's in it

Ditto SDK 5.1.0 → 5.2.0

• @dittolive/ditto bumped to ^5.2.0. Same native platform matrix (macOS arm64, Linux x64/arm64, Windows x64; still no Intel Mac), so the doctor/README/AGENTS platform text only changes the version.
• Notes about bugs found on 5.1.0 (retail-joins hang, NO_COLOR abort, system:collections first-read-only) are left as 5.1.0 — not yet re-checked on 5.2.0.

--profile shows the full 5.2.0 profile (parity with the VS Code extension)

• Per-operator recv / send / stor times, yield counts by reason (budget, sender, receiver, heartbeat), skipped documents, and the operator's remaining attributes.
• nlJoin.inner subplans render as a labeled nested tree (previously a raw JSON attribute) and count toward hotspot detection.
• Summary strip gains Stor and Yields totals; header shows db / request type / feature flags / directives.
• Profile fixture regenerated from SDK 5.2.0 (scripts/spike-c.mjs now includes a join).

New: .metrics in the dql shell

• The session now opens with the startup-gated system:metrics exporter enabled (DittoConfig systemParameters, SDK 5.2+), as the extension does.
• .metrics renders store + query (DQL) metrics as boxed tables; .metrics store|query|all|<section> selects sections.
• Counters: total since session open + change since last read. Timings: a two-level header separates running Count/Avg/Peak from since-last-read New/p50/p90/p99/Max (the SDK reports them over different windows).
• Namespace rows, └ db=main breakdowns, unit-aware values, shared column widths, terminal-width fitting. dql.* aliases of ditto.query.* are hidden.
• Side effect: SELECT * FROM system:metrics now returns real data in any dittosh dql command instead of the disabled placeholder.

New: dittosh tutorial (written walkthrough + interactive DQL lessons)

• dittosh tutorial writes dittosh-tutorial.md, a hands-on walkthrough tailored to the install (CLI/SDK version, Homebrew vs npm upgrade commands, platform). Covers import/query/export, the shell and .metrics, datasets, --explain/--profile/--advise, joins, diagnostics and advanced DQL, and agent skills. --out, --force, --stdout.
• The content is structured blocks; tests/e2e/tutorial.test.ts runs every runnable block in order against a fresh store and checks expected output (30 blocks), so the tutorial can't silently go stale.
• dittosh tutorial --interactive teaches DQL in 27 lessons across beginner (8), intermediate (10), and advanced (9). Each lesson explains the concept (projection, predicates, scalar vs aggregate functions, MISSING vs NULL, joins, indexes, query plans…), adds an "if you know SQL" note, shows syntax, sets a task, and checks the answer by its result. Separate practice store (<config dir>/tutorial-store), saved progress, --level.

Interactive lessons: full-screen UI (Ink)

• The lessons open in a full-screen view (alternate screen, like less) built with Ink 8 + React 19: header with progress bar, lesson panel, results panel, a status line with the latest ✓/✗ verdict, the learn> prompt, and a key bar.
• Left: lesson outline with per-level progress (✓ done, ▸ current), auto-shown at 90+ columns, Ctrl+B toggles. Right: slide-in panel for help (?) or the data view (Ctrl+T / .schema): the fields of the lesson's collections read from the live store, with types, n of m for partial fields, examples, dot paths (details.origin) and lines[].sku for arrays of objects. Each lesson card says which collections it uses and that Ctrl+T shows them.
• Enter runs (; optional); a | line only when input looks unfinished. Each action starts with an empty results panel; long output opens at its top (PgUp/PgDn). Ctrl+L lesson picker, Ctrl+N/P, Ctrl+O for the whole lesson in a short window, ↑/↓ history, Tab hint. Over-wide tables are cut off with … rather than wrapped; the task line always stays visible.
• Architecture: a TutorialProvider (React context) supplies the LessonRunner — which owns the Ditto session — and a small output store; the SDK never enters React. Editor, layout, outline, help, and schema logic are pure modules. Tests drive the real screen through a fake TTY.
• Cost: Ink/React are lazy-loaded (await import) into their own chunk, bundled in production mode, so other commands are unaffected (cli.js size unchanged). Measured: ~0.39 s vs 0.34 s to the first prompt, ~138 MB vs ~115 MB RSS (the Ditto SDK dominates both). --plain, or a window under 60×20, keeps the node:repl shell.

Lesson content review

• Syntax examples and hints no longer contain the answer: examples show the pattern on other data (suppliers, stock, fair-trade…). A unit test fails if any example or hint contains its lesson's answer, and an integration test runs every complete example against a real store.
• Tasks reworded in plain English and say what to show when the check needs it (e.g. "Find the products whose category is coffee, and show their names."). Full-statement examples for CASE, BETWEEN, IS MISSING; lesson 13 shows the details object's shape.
• INSERT uses VALUES (docs: equivalent to DOCUMENTS; verified on 5.2 with literals, multiple documents, and ON ID CONFLICT) in the lessons and the Markdown tutorial.
• Case sensitivity: verified on 5.2 — collection and field names are case-sensitive (FROM Products silently returns nothing), keywords aren't. Lesson 1 explains it, and a wrong-case collection or field name gets a specific "names are case-sensitive" verdict instead of a generic failure.
• SDK 5.2 DQL changes were researched by comparing 5.1.0 and 5.2.0 directly; the advanced lessons and the tutorial's diagnostics section use them (without version call-outs — the tutorial just teaches the bundled SDK): ANY … SATISFIES with AND, DELETE/EVICT … USE IDS, mixed ASC/DESC composite indexes, constant-first index ranges, system:sync_degradations, support bundles, new system:system_info keys, ditto.query.* metrics. No 5.1 → 5.2 regressions found.
• Every lesson's sample answer is verified against the real SDK (tests/integration/lessons.test.ts), both in sequence and with each lesson started directly.

New: dittosh dql bench (benchmarking)

Modeled on the dql-metrics-benchmark runners so results are comparable.

• dql bench "<statement>" or -f file.json: --runs <n> timed runs (default 50), optional --warmup (bare = adaptive ~250 ms, clamped 1–1000, like the runners; or --warmup <n>). Reports mean, p50, p95, p99, min, max, stddev (true median, floor-indexed p95/p99, population stddev — same formulas as stats.rs/Stats.kt). One run shows just its time.
• --pre runs once before warmup/timing; --before-each (alias --reset) runs untimed before every warmup and timed run; --post runs once at the end, even after a failure. All repeatable.
• Benchmark files use the benchmark repo's benchmarks.json format (query, preQueries, resetQueries, postQueries), so suite files run unchanged; a list or single object also works. Its oracle keys (category, qb, expected_count, …) are ignored; other unknown keys are a usage error with typo suggestions (prequeries → preQueries). --only a,b selects entries.
• --example <dataset> writes a sample file from the dataset's catalog (plain read, indexed read, aggregate/join, a write that restores its own data), keys in run order, each entry with a description. Integration test: every sample runs cleanly and leaves doc counts and indexes unchanged.
• dataset run <query> --runs <n> --warmup benchmarks a catalog query with the same runner (existing --setup/--yes rules).
• A write that runs more than once with no reset statements gets a warning. Found while building this: the movies/retail catalog writes (e.g. evict_single) insert their fixture in preQueries, so repeated runs time no-ops; pos does it right with resetQueries. The samples avoid those entries; fixing the catalogs is left for follow-up.
• Results go to stdout (table / JSON when piped / -o, --format), progress to stderr; exit 1 if any benchmark or cleanup fails, 2 for usage errors.
• Docs: docs/benchmarking.md (options, order of steps, file format, how to build a file), a "Benchmark a query" section in the Markdown tutorial (run by e2e), README, release notes, AGENTS.md.

New: dittosh doctor --report (diagnostics for bug reports)

• Writes one .tar.gz to attach to a GitHub issue: report.md + report.json (versions, build, install method, update status, OS/CPU/memory, terminal and locale, doctor checks, paths, store size and free disk space, DITTOSH_* and related settings, collection names with document counts and indexes, the last failed command and its error output) and ditto-support-bundle.tar.
• SDK 5.2 support bundle: requested with EXECUTE FUNCTION system_request_support_bundle(), which returns an attachment token; fetched and copied out. Holds the SDK's debug logs (ditto-logs.log.gz, kept up to 15 MB / 15 days) and config_snapshot.json (secrets removed). Works offline. The SDK admits 3 requests per 10 minutes per store; past that, the newest bundle in system:support_bundles is saved instead, with the SDK's reason.
• Privacy: the license token is never included (checked in every file, including inside the SDK bundle); credential env vars show only as set/not set; home shows as ~; no document contents. --no-store / --no-sdk-bundle opt out.
• Last error: failed commands (exit 1/3/4, not usage errors) save the command, exit code, last 60 stderr lines, and any stack to <config dir>/last-error.json, so a report can be made after the fact. doctor and support-bundle don't overwrite it.
• dittosh dql support-bundle [-o file.tar] [--list] saves only the SDK bundle. dittosh doctor now works without dql.
• --debug / DITTOSH_DEBUG=1|verbose turns on the SDK's logs (the native logger writes to stderr, so stdout stays results-only) and prints stack traces.
• Bug found and fixed: the attachment fetch completes through a native callback that doesn't keep Node's event loop alive, so Node could exit mid-fetch ("unsettled top-level await", then exit 134). A timeout timer now holds the loop open and turns a stuck fetch into an error; the flaky e2e went from 3/5 failures to 6/6 passes.
• Bug-report issue template (.github/ISSUE_TEMPLATE/bug_report.yml), docs/troubleshooting.md, a "Report a problem" tutorial section (run by e2e), README, release notes, AGENTS.md, manual checklist in docs/testing-dql-doctor.md.

Security (dev dependencies only — not in the installed CLI)

• Resolves both open Dependabot alerts: vite → 8.3.6, source-map-js → 1.2.2 (they close when this merges to main).
• vitest / @vitest/coverage-v8 → 4.1.12 (moderate npm audit advisories). npm audit: 0 vulnerabilities.

Adversarial review

Five independent reviewers went over the whole PR; the 11 issues at least three of them agreed on are fixed (a35054d):

• Practice store guard: a blank -d now means the lessons' own store (never the default dql store), and the lessons refuse a directory they didn't create — every lesson resets products/suppliers/orders.
• Clean shutdown: quitting the full-screen view waits for in-flight work before the session closes; plain-shell dot-commands queue behind a running statement; submit() credits the lesson it started on.
• Lesson isolation: every lesson starts from fresh practice data, so skipped lessons or stray writes can't fail a correct answer; input stops once a statement passes; the Indexes lesson checks for the named index.
• Screen: input scrolls sideways around the cursor, long multi-line input collapses, a too-small window shows a placeholder, multi-line chunks keep every line, PgUp stops at the top.
• "Progress saved" goes to stderr.

Raised by fewer than three reviewers and left for follow-up: .break on a | line, -- comments in multi-line pastes, .metrics window labels after a direct system:metrics read, the composite-index lesson's setup pre-creating its index, a blank screen with CI set, and Update/Evict lessons accepting no WHERE.

Second adversarial review (bench + diagnostics)

Five more independent reviewers went over the commits after a35054d; everything at least three agreed on is fixed (4344fde), plus one credential leak two confirmed:

• Credential leak (2 reviewers, confirmed): doctor --report copied every DITTOSH_* value, including DITTOSH_GITHUB_TOKEN. That and any credential-like name (TOKEN, SECRET, PASSWORD, API_KEY, …) now shows only as set/not set, and those values are scrubbed from the whole report and the saved last error.
• Windows home path in report.json (5): redaction ran after JSON encoding doubled the backslashes; each string is now redacted first.
• SDK bundle privacy wording (5): help, docs, issue template, release notes, and tutorial now say the SDK bundle is included as is and holds the database ID, full file paths, and peer keys.
• --report report.md failed (5); --report overwrote files (4): the archive is staged in its own folder; an existing file is refused (exit 2, before any checks); a directory gets the default name inside it.
• Tests wrote the developer's real last-error.json (3): tests now get a throwaway DITTOSH_CONFIG_DIR.
• --example samples triggered the reset warning (3): the warning skips UPDATE, which does the same work every run.
• Failed multi-run bench switched columns (3): columns follow the requested --runs.

Left for follow-up (fewer than three): Ctrl-C during bench skips postQueries; the lessons' data drawer can read mid-reset; bench -o prints "Wrote…" on stdout; --debug can't override DITTOSH_DEBUG=0; auto warmup ignores reset time; support-bundle opens the store before checking -o.

Dependencies added

• Runtime (bundled into dist): ink 8, react 19, wrap-ansi 10 (already present via Ink). Dev: @types/react. tsconfig gains "jsx": "react-jsx"; tsup defines process.env.NODE_ENV=production and keeps react-devtools-core (Ink dev-only) external.

Release prep

• Version 0.2.0 in package.json / package-lock.json; releases/v0.2.0.md added.

Testing

• npm test with dev credentials: 972 passing (907 unit + integration, 65 e2e).
• npm run coverage: ~94% statements / 89% branches (gate 85%); src/tutorial/ui ~97%.
• npm run typecheck, npm run build, npm run release:check pass.
• Biome is clean on src, tests, scripts. Note: npm run lint fails on main too because biome.json uses the 2.2.0 schema with Biome 2.5.11 installed (npx biome migrate fixes it) — not addressed here.
• Manually ran --profile, .metrics, and the interactive lessons in a real terminal; the full-screen UI was also driven in a pseudo-terminal at 80×24, 100×30, 110×26, and 120×30 and checked frame by frame. Manual checklist: docs/testing-tutorial.md.

Before tagging

• Review and merge
• Optionally re-check the 5.1.0 known issues on 5.2.0
• Tag v0.2.0 from main per docs/publishing.md

🤖 Generated with Claude Code