Release 0.2.0: SDK 5.2.0, full-screen DQL lessons, tutorial, dql bench, shell .metrics, full profiles

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,44 @@
+name: Bug report
+description: Something in dittosh doesn't work as expected
+labels: ["bug"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for reporting a problem. The fastest way to help us is a diagnostic report:
+
+        ```bash
+        dittosh doctor --report
+        ```
+
+        It writes `dittosh-report-<time>.tar.gz` in the current folder, with versions, OS details,
+        health checks, the last failed command, a store summary (no document contents), and Ditto's
+        SDK support bundle (SDK debug logs). The license token is never included. The SDK bundle is
+        Ditto's own file and holds the database ID, full file paths, and peer keys. Extract it and
+        read it before attaching it. Leave parts out with `--no-store` or `--no-sdk-bundle`.
+  - type: textarea
+    id: what
+    attributes:
+      label: What happened?
+      description: What did you run, what did you expect, and what happened instead?
+    validations:
+      required: true
+  - type: textarea
+    id: command
+    attributes:
+      label: Command and output
+      description: The exact command and what it printed. Add `--debug` to see SDK logs.
+      render: shell
+  - type: textarea
+    id: report
+    attributes:
+      label: Diagnostic report
+      description: Drag the `dittosh-report-….tar.gz` file here. If you can't attach files, paste the contents of `report.md` instead.
+    validations:
+      required: true
+  - type: input
+    id: version
+    attributes:
+      label: dittosh version
+      description: Output of `dittosh --version`
+      placeholder: "0.2.0"

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: true

AGENTS.md

@@ -15,10 +15,14 @@ The Ditto CLI: an npm/Homebrew-installable TypeScript CLI (binary `dittosh` —
 
 ## Layout
 
-- `src/cli/` — commander entry (`index.ts`), injected version (`version.ts`, tsup `define`), `groups/` per command group (`dql`, `skills`, `system`)
+- `src/cli/` — commander entry (`index.ts`), injected version (`version.ts`, tsup `define`), `groups/` per command group (`dql`, `skills`, `system`, `tutorial`)
 - `src/config/` — data-dir resolution (`--data-dir` > `DITTOSH_DATA_DIR` > OS default), config dir (`DITTOSH_CONFIG_DIR` > OS default; env-paths caches homedir at module load, so tests must use this override, not `$HOME`), persisted state (one-time warnings, update cache)
 - `src/identity/` — token loading (dev env / release reassembly), expiry
-- `src/ditto/session.ts` — the only SDK touchpoint: init/open/close, log taming, lock mapping
+- `src/ditto/session.ts` — the only SDK touchpoint: init/open/close, log taming, lock mapping; enables the startup-gated `system:metrics` exporter (`systemParameters`)
+- `src/tutorial/` — `dittosh tutorial`: Markdown walkthrough as structured blocks (`content.ts`, runnable blocks executed by `tests/e2e/tutorial.test.ts`) and interactive DQL lessons (`lessons.ts` by level, `runner.ts`, `lesson-eval.ts`; every sample answer verified in `tests/integration/lessons.test.ts`). The full-screen lesson view is Ink + React in `src/tutorial/ui/` (`app.tsx` screen, `components.tsx`, `context.tsx` provider for the runner/store, pure `editor.ts`/`layout.ts`/`store.ts`/`help.ts` (right drawer: help, or the data view from `src/tutorial/schema.ts`)/`outline.ts` (left lesson sidebar); tests in `tests/unit/tutorial-ui.test.ts` with a fake TTY). It is lazy-loaded (`await import`) so Ink/React never load for other commands; `--plain` or a <60×20 window uses the node:repl shell. Wiring: `src/cli/groups/tutorial.ts`, `tutorial-repl.ts`
+- `src/bench/` — `dittosh dql bench` (wiring: `src/cli/groups/dql/bench.ts`; also `dataset run --runs/--warmup`): benchmark file parsing (`spec.ts`, the dql-metrics-benchmark `benchmarks.json` format), the timed runner (`runner.ts`: pre → warmup → reset+timed runs → post), stats matching the benchmark runners (`stats.ts`: floor-indexed p95/p99, adaptive ~250 ms warmup), and `--example` sample files from the dataset catalogs (`example.ts`). User docs: `docs/benchmarking.md`
+- `src/support/` — diagnostics for bug reports: `dittosh doctor --report` (wiring: `src/cli/groups/doctor.ts`, shared by `dittosh doctor` and `dittosh dql doctor`) gathers a redacted report (`report.ts`, `redact.ts`) and archives it as .tar.gz (`archive.ts`); `sdk-bundle.ts` requests the SDK 5.2 support bundle (`EXECUTE FUNCTION system_request_support_bundle()` → attachment token → `DittoSession.saveAttachment`; rate limit 3/10 min per store → falls back to the newest in `system:support_bundles`; also `dittosh dql support-bundle`); `last-error.ts` tails stderr and saves the last failed command to `<config dir>/last-error.json` (written from `src/cli/index.ts`). `--debug`/`DITTOSH_DEBUG` enables SDK logs (native logger writes to stderr). Never put the token or document contents in a report. User docs: `docs/troubleshooting.md`
+- `src/metrics/` — `system:metrics` row parsing and section grouping (REPL `.metrics`, rendered by `src/render/metrics.ts`)
 - `src/query/` — statement classifier, splitter, param binding, result extraction, row cap
 - `src/render/` — table (terminal-width fitting)/JSON/CSV/markdown/HTML/vertical, pager (`$PAGER`/`less`, `--no-pager`/`DITTOSH_NO_PAGER`), `-o/--out`, (M4: explain/profile/advise renderers)
 - `datasets/` — vendored benchmark suite definitions (movies, retail, retail-joins, pos); **no generated data ever committed**
@@ -66,7 +70,7 @@ scripts/install-release.sh # stamp token → RELEASE=true build → npm i -g . (
 
 ## Known platform facts
 
-- SDK `@dittolive/ditto@5.1.0` native matrix: macOS arm64, Linux x64/arm64, Windows x64 (no darwin-x64).
+- SDK `@dittolive/ditto@5.2.0` native matrix: macOS arm64, Linux x64/arm64, Windows x64 (no darwin-x64).
 - `DittoConfig(appId, { mode: "smallPeersOnly" }, dir)` + `Ditto.open` + `setOfflineOnlyLicenseToken` — verified (Spike A). EXPLAIN → first item `plan`; PROFILE → trailing `~request_profile` item.
 - The native tracing bootstrap writes ~7 WARN/INFO lines to **stderr** at `sdk.init()` (fd-level, not suppressible from JS). Cosmetic only; stdout is clean.
 - Two processes on one data dir → "File already locked" → mapped to `LockError` (exit 4).

README.md

@@ -21,7 +21,7 @@ npm i -g @dittolive/cli          # npm (primary)
 brew install getditto/tap/dittosh # Homebrew (macOS/Linux)
 ```
 
-The binary is `dittosh`. Requires Node.js ≥ 22 for npm installs; Homebrew manages Node.js 24. Supported platforms (matching the Ditto Node SDK): **macOS arm64, Linux x64/arm64, Windows x64**. Intel Macs (darwin-x64) are not supported by SDK 5.1.0.
+The binary is `dittosh`. Requires Node.js ≥ 22 for npm installs; Homebrew manages Node.js 24. Supported platforms (matching the Ditto Node SDK): **macOS arm64, Linux x64/arm64, Windows x64**. Intel Macs (darwin-x64) are not supported by SDK 5.2.0.
 
 Update using the package manager you installed with:
 
@@ -41,7 +41,7 @@ The CLI ships with a built-in offline license and runs entirely locally — no a
 
 ```bash
 # check your install
-dittosh dql doctor
+dittosh doctor
 
 # load a sample dataset
 dittosh dql dataset load movies
@@ -84,7 +84,7 @@ dittosh dql                                            # interactive REPL
 | `--continue-on-error` | keep running after a failure (batch mode) |
 | `--time` | timing footer (host wall-clock + server parse/plan/elapsed when profiling) |
 | `--explain` | print the query plan (EXPLAIN side-trip, SELECTs only) |
-| `--profile` | print the execution profile with per-operator timings and hotspot flags (SELECTs only) |
+| `--profile` | print the execution profile with per-operator timings (exec/recv/send/stor), document counts, scheduler yields, join subplans, and hotspot flags (SELECTs only) |
 | `--advise` | print index advice (ADVISE, SELECTs only) |
 | `--apply` | apply ADVISE's suggested `CREATE INDEX` statements (prompts; `-y` skips) |
 | `-y, --yes` | skip confirmation prompts |
@@ -98,10 +98,17 @@ dittosh dql "SELECT _id FROM movies WHERE _id.year = '2001' LIMIT 1" \
   | dittosh dql "SELECT * FROM movies WHERE _id = :id" --args -
 ```
 
-### `dittosh dql doctor`
+### `dittosh doctor` (also `dittosh dql doctor`)
 
 Platform/arch, Node version, data-directory writability, token validity + expiry, SDK load, and store-lock probe — with an exit code that says what's wrong.
 
+**Reporting a bug?** `dittosh doctor --report` writes `dittosh-report-<time>.tar.gz` to attach to a [GitHub issue](https://github.com/getditto/ditto-cli/issues/new?template=bug_report.yml): versions, OS and terminal details, the checks, paths and free space, dittosh settings (credentials only as set/not set), a store summary (collection names, counts, indexes; no document contents), the last failed command and its error output, and the Ditto SDK support bundle with the SDK's debug logs. `--no-store` and `--no-sdk-bundle` leave parts out.
+
+- `dittosh dql support-bundle [-o file.tar]` saves only the Ditto SDK support bundle (`--list` shows the ones kept in the store).
+- `--debug` (or `DITTOSH_DEBUG=1`) on any command shows the SDK's debug logs and stack traces on stderr; stdout is unchanged.
+
+See [Troubleshooting](docs/troubleshooting.md).
+
 ### `dittosh dql collections` / `dittosh dql indexes [collection]`
 
 List collections (`system:collections`) and indexes (`system:indexes`).
@@ -146,6 +153,27 @@ dittosh dql dataset reset retail --yes            # evict the dataset's collecti
 
 `dataset run` prints the resolved statement (on stderr, so stdout stays clean), then executes it. Query names resolve across datasets; ambiguous names list the matches. `--setup` applies the entry's index DDL first; write-category catalog queries require `--yes` and clean up after themselves. `--seed <n>` reproduces a dataset exactly; changing seeds adds new documents (reset first for a clean slate).
 
+### `dittosh dql bench` — benchmark statements
+
+Time a statement over many runs and get mean, p50, p95, p99, min, max, and standard deviation:
+
+```bash
+dittosh dql bench "SELECT * FROM movies WHERE rated = 'PG'" --runs 100 --warmup
+dittosh dql bench "SELECT * FROM movies WHERE rated = 'PG'" \
+  --pre "CREATE INDEX IF NOT EXISTS movies_rated ON movies (rated)" \
+  --post "DROP INDEX IF EXISTS movies_rated ON movies"
+dittosh dql bench --example movies -o movies-bench.json   # sample benchmark file
+dittosh dql bench -f movies-bench.json --runs 50 --warmup
+dittosh dql dataset run filtered_query_with_index --runs 50 --warmup   # a catalog query
+```
+
+- `--runs <n>`: timed runs (default 50). `--warmup`: untimed runs first, enough to fill about 250 ms; `--warmup <n>` for an exact count.
+- `--pre` runs once before timing, `--before-each` (or `--reset`) before every run (untimed, e.g. re-insert the document an `EVICT` removes), and `--post` once at the end, even after a failure. Each is repeatable.
+- `-f <file>`: benchmarks from a JSON file, in the same format as Ditto's DQL benchmark suites: `{"name": {"query": "…", "preQueries": […], "resetQueries": […], "postQueries": […]}}`. `--only a,b` picks entries. `--example <dataset>` writes a sample file for `movies`, `retail`, `retail-joins`, or `pos`.
+- Results go to stdout (a table in a terminal, JSON when piped, or `-o`/`--format`); progress goes to stderr. Exit code 1 if any benchmark fails.
+
+The full file format, the order of steps, and how to build a benchmark file are in [docs/benchmarking.md](docs/benchmarking.md).
+
 ### Global flags
 
 `--no-color`, `--quiet` (suppress informational notes), `--no-update-check` (planned; update flow lands in a later milestone).
@@ -171,6 +199,21 @@ Semantics: `add` never upgrades an existing install (it skips; `--force` overwri
 
 Agent targets: Claude Code (`~/.claude/skills` or `.claude/skills`), OpenCode (`~/.agents/skills` or `.agents/skills`), Codex (`~/.codex/skills`), Gemini (`~/.gemini/skills`), Antigravity (`~/.gemini/config/skills` or `.agents/skills`), Cursor (`.cursor/rules`, project-only), Copilot + Windsurf (project instruction files). While `getditto/agent-skills` is private, set `GITHUB_TOKEN` (e.g. `GITHUB_TOKEN=$(gh auth token)`).
 
+### `dittosh tutorial` — learn the CLI and DQL
+
+```bash
+dittosh tutorial                       # write ./dittosh-tutorial.md, a hands-on walkthrough of the CLI
+dittosh tutorial -o ~/t.md --force     # choose the file / overwrite
+dittosh tutorial --stdout              # print the Markdown instead
+dittosh tutorial --interactive         # guided DQL lessons, full screen
+dittosh tutorial --interactive --plain # the same lessons in a line-by-line shell
+dittosh tutorial --interactive --level advanced   # start at beginner | intermediate | advanced
+```
+
+The Markdown tutorial is tailored to this install (CLI and SDK version, Homebrew or npm upgrade commands, platform) and covers importing, querying, exporting, the interactive shell and `.metrics`, sample datasets, `--explain`/`--profile`/`--advise`, benchmarking with `dql bench` (including benchmark files), joins, diagnostics with system collections and more advanced DQL, and agent skills. Every runnable command in it is exercised by the e2e suite.
+
+The interactive lessons teach DQL concepts — documents and collections, projection, predicates, boolean logic, sorting, writes, expressions and functions, `CASE`, MISSING vs NULL, aggregation and grouping, joins, `ANY … SATISFIES`, indexes, `EXPLAIN`, `PROFILE`, and system collections — in 27 lessons across beginner, intermediate, and advanced levels. Each lesson explains the idea (with an "if you know SQL" note), shows the syntax, sets a task, and checks your answer. They run in a separate practice store (`<config dir>/tutorial-store`) and save your progress. The lessons open full screen: the lesson stays in a panel at the top, results scroll below it, the latest ✓/✗ verdict sits above the prompt, a sidebar on the left outlines the lessons and your progress (Ctrl+B toggles it), Ctrl+T (or `.schema`) shows the fields of the collections each lesson uses, and `?` slides a help panel in from the right. Press Enter to run a statement (`;` is optional); Tab hints, Ctrl+N/Ctrl+P move between lessons, Ctrl+L picks a lesson from a list, PgUp/PgDn scroll, and Ctrl+D quits. The dot-commands (`.hint`, `.answer`, `.next`/`.back`, `.lessons`, `.level <name>`, `.goto <n>`, `.reset`, `.exit`) work too. `--plain`, or a window smaller than 60×20, uses a simple line-by-line shell instead.
+
 ### Planned for later milestones
 
 `dittosh version`, `dittosh update` (self-update banner + channel-aware upgrade).
@@ -208,7 +251,9 @@ dittosh dql --advise --apply -y "SELECT …"  # apply them
 
 ## REPL
 
-Bare `dittosh dql` starts an interactive session: multi-line statements terminated with `;`, history, per-statement timing, dot-commands (`.help`, `.collections`, `.indexes [name]`, `.break`, `.exit`).
+Bare `dittosh dql` starts an interactive session: multi-line statements terminated with `;`, history, per-statement timing, dot-commands (`.help`, `.collections`, `.indexes [name]`, `.metrics [section]`, `.break`, `.exit`).
+
+`.metrics` shows the SDK's live system metrics for the session as formatted tables — store (SQLite transactions, fsync, WAL size, commit/lock timings) and query/DQL (requests, selects, failures, parse/plan/execution time) by default. Totals run since the session opened; the Change, p50/p90/p99 and Max columns cover the time since the previous `.metrics`. `.metrics query` or `.metrics store` narrows the view; `.metrics all` adds network, sync, transport, and the other sections.
 
 ## Development