From 650e4e31ea054977978696b7083b2e82b6777587 Mon Sep 17 00:00:00 2001 From: leemour Date: Thu, 15 Oct 2026 02:33:33 +0200 Subject: [PATCH] test(agents): output size of --json fields, --brief and --jsonl on tg and memo Measured on a synthetic 5,000 message store, 3 rounds, deterministic. Trimmed --json= cuts 31% to 64%; --jsonl does not shrink; --brief only affects memo search messages (7%). Co-Authored-By: Claude Opus 5.5 (1M context) --- performance/README.md | 1 + performance/agents/README.md | 6 ++ performance/agents/output-size/README.md | 83 ++++++++++++++++++++++++ performance/agents/output-size/run.sh | 48 ++++++++++++++ performance/agents/output-size/seed.ts | 75 +++++++++++++++++++++ 5 files changed, 213 insertions(+) create mode 100644 performance/agents/README.md create mode 100644 performance/agents/output-size/README.md create mode 100644 performance/agents/output-size/run.sh create mode 100644 performance/agents/output-size/seed.ts diff --git a/performance/README.md b/performance/README.md index ac2a4d4..545fcd0 100644 --- a/performance/README.md +++ b/performance/README.md @@ -6,4 +6,5 @@ - [mail](mail/README.md): reading a mailbox into the store — Gmail import speed and limits, and checking the sender score against Gmail's Spam. - [store](store/README.md): the shared SQLite store — filters, migrations. +- [agents](agents/README.md): what the CLIs cost the AI agent that reads their answers — output size. - [tasks](tasks/README.md): tasks found in messages — duplicate detection. diff --git a/performance/agents/README.md b/performance/agents/README.md new file mode 100644 index 0000000..3f26dff --- /dev/null +++ b/performance/agents/README.md @@ -0,0 +1,6 @@ +# Agents + +Measurements of what the WireCat CLIs cost the AI agent that reads their answers. + +- [output-size](output-size/README.md) — bytes and tokens of one answer under `--json`, `--json=`, + `--jsonl`, `--brief` and the default text (2026-10-15). diff --git a/performance/agents/output-size/README.md b/performance/agents/output-size/README.md new file mode 100644 index 0000000..9138733 --- /dev/null +++ b/performance/agents/output-size/README.md @@ -0,0 +1,83 @@ +# Output size: what `--json=`, `--brief` and `--jsonl` save an agent + +Measured 2026-10-15 with tg 0.56.0 and memo 0.32.0, for the "Search efficiently" page of the CLI docs. + +## Why we ran it + +The owner wants agents to spend fewer tokens reading the CLIs' answers. The docs page "Search efficiently" +recommends trimming output (`--json=`, `--brief`, `--jsonl`) but gave no numbers, and nobody had checked +that each option does what the page says on every command. The owner decided (NEED-1099 A, NEED-1122 A) to +measure first and let the page and a blog post quote the result. + +The question: how much does each option cut from the answer, compared with a full `--json`? + +## Setup + +- AMD Ryzen AI 9 HX 470, Linux 7.0, Node v24.19.0. tg 0.56.0 and memo 0.32.0, installed from npm into a scratch + folder (`npm i --prefix @wirecat/tg-cli@0.56.0 @wirecat/cli-memo`). +- Synthetic store, made by [`seed.ts`](seed.ts): 5,000 invented messages in five chats (one private, four + groups), 1,000 each, about 110 characters per message. Every 25th message has an invented phone number and + address (`example.test`), so `mentions` has something to list. No real account, name or message is used. + HOME, XDG_*, `MESSAGING_STORE`, `TG_STATE_DIR` and `WIRECAT_STATE_DIR` point at a throwaway folder. +- Read only, from the store, `--offline` for tg. No network. +- Each answer is measured in bytes of stdout. Tokens are bytes / 4. No tokenizer is installed here, so these are + estimates, not counts. +- "Human" is what a terminal shows, captured through a pty with colour off and 120 columns. A pipe gives JSON from + tg, so the human text needs the pty. +- 3 rounds per command and option. Every round gave the same bytes. +- Run: `node seed.ts `, then `bash run.sh 3` + ([`seed.ts`](seed.ts), [`run.sh`](run.sh)). The header of `run.sh` lists the variables. + +## Results + +Bytes of one answer, tokens in brackets. "Trimmed" is `--json=`: `ref,timestamp,excerpt` for +searches, `ref,timestamp,text` for `messages list`, `ref,title` for `chats list`, `at,locator` for +`contacts timeline`, `kind,value,locator` for `mentions list`. + +| Command | `--json` | trimmed | saved | `--jsonl` | human text | +|---|---|---|---|---|---| +| `tg search messages heron --limit 20` | 6,669 (1,667) | 4,628 (1,157) | 31% | 6,529 | 17,500 | +| `tg messages list 1 --limit 50` | 21,122 (5,280) | 10,271 (2,567) | 51% | 22,024 | 8,171 | +| `tg chats list` (5 chats) | 976 (244) | 414 (103) | 58% | 1,026 | 260 | +| `tg contacts timeline Alice --limit 20` | 4,638 (1,159) | 1,778 (444) | 62% | 4,825 | 2,730 | +| `tg mentions list --limit 20` | 5,774 (1,443) | 2,065 (516) | 64% | 5,999 | 1,988 | +| `memo search all heron --limit 20` | 7,194 (1,798) | 4,313 (1,078) | 40% | not accepted | 2,934 | +| `memo search messages heron --limit 20` | 6,149 (1,537) | 4,108 (1,027) | 33% | not accepted | 3,835 | + +`--brief` and `--full`: + +| Command | `--json --brief` | `--json --full` | human `--brief` | +|---|---|---|---| +| every tg command above | same as `--json` | same as `--json` | same as human | +| `memo search all` | same as `--json` | same as `--json` | same as human (2,934) | +| `memo search messages` | 5,749 (7% less) | same as `--json` | 3,555 (7% less) | + +## What it showed + +- `--json=` is the option that works everywhere. It cut 31% to 64% on these commands. The saving is + bigger where the full row carries many fields an agent rarely needs (`mentions`, `contacts timeline`, + `chats list`) and smaller where the one useful field is long text (searches: the excerpt is most of the bytes). +- `messages list` is its best use: `ref,timestamp,text` drops sender ids, provider, account, chat id, + `edited_at`, `attachments`, `reply_to`, `forwarded_from` and `reactions` from every row, and halves the answer. +- `--jsonl` does not shrink anything. It is 2% smaller on a search and 4% to 5% larger on lists, because each row + repeats its own envelope. It is for streaming and `jq`, not for saving tokens. memo does not accept it. +- `--brief` did nothing to tg's answers, in JSON or in text. On `memo search messages` it saved 7% of the text + and of the JSON; on `memo search all` it changed nothing, because the default text is already the brief shape. + The tg help says `--brief` is for found messages and mail, so a docs page should not offer it as a general trim. +- Against the human text, JSON is bigger on most lists (`messages list` 2.6 times, `mentions` 2.9 times), and + smaller on tg searches, where the terminal shows surrounding messages next to each hit (17,500 bytes for 20 + hits). An agent that reads a terminal pays for that context whether it wants it or not. +- On a 20-hit search the trimmed JSON is about 500 tokens fewer than the full JSON. The bigger lever is `--limit`, + which was not varied here. + +## Not measured + +- Real tokenizer counts. Tokens are bytes / 4; JSON with many short keys and ids probably tokenizes worse than + that, so the saving in tokens may differ from the saving in bytes. +- Long messages. Synthetic messages are about 110 characters, so a search excerpt is nearly the full text and + `--full` could not differ from `--json`. With long mail or messages, `--brief` and the excerpt length will matter + more than here. +- Other commands: `tg chats list` on a large account, `memo context`, mail, notes, `--limit` sweeps. +- Time. Output options change bytes, not the work of the query; speed was not timed. +- MAX (`max`) and other CLIs. They share cli-core, but their answers were not run. +- Whether an agent does the task as well with the trimmed fields. That needs an agent run, not a byte count. diff --git a/performance/agents/output-size/run.sh b/performance/agents/output-size/run.sh new file mode 100644 index 0000000..2f6230b --- /dev/null +++ b/performance/agents/output-size/run.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Bytes of one answer under each output option, read-only and offline, on the store made by seed.ts. +# +# TG_PROFILE=bench MESSAGING_STORE= TG_STATE_DIR= bash run.sh [rounds] +# +# Run it with HOME, XDG_* and WIRECAT_STATE_DIR pointed at a throwaway folder: nothing here may touch a real +# account. tg and memo come from PATH (TG_BIN / MEMO_BIN override). Prints one TSV line per command and variant: +# command, variant, bytes of every round (or "error" when the command refuses the combination). +# "human" is what a terminal shows, captured through a pty; stdout of a pipe is JSON for tg, plain text for memo. +set -u +rounds=${1:-3} +tg=${TG_BIN:-tg} +memo=${MEMO_BIN:-memo} +export NO_COLOR=1 COLUMNS=120 + +measure() { # label variant bin args... + local label=$1 variant=$2 bin=$3; shift 3 + local sizes=() out + for _ in $(seq "$rounds"); do + if [[ "$variant" == human* ]]; then + out=$(script -qec "$bin $* 2>/dev/null" /dev/null | sed 's/\x1b\[[0-9;?]*[a-zA-Z]//g; s/\r//g' | wc -c) + else + out=$("$bin" "$@" 2>/dev/null | wc -c) + fi + [[ "$out" == 0 ]] && out=error + sizes+=("$out") + done + printf '%s\t%s\t%s\n' "$label" "$variant" "${sizes[*]}" +} + +bench() { # label bin fields args... + local label=$1 bin=$2 fields=$3; shift 3 + measure "$label" json "$bin" "$@" --json + measure "$label" "json=$fields" "$bin" "$@" "--json=$fields" + measure "$label" "json --brief" "$bin" "$@" --json --brief + measure "$label" "json --full" "$bin" "$@" --json --full + measure "$label" jsonl "$bin" "$@" --jsonl + measure "$label" human "$bin" "$@" + measure "$label" "human --brief" "$bin" "$@" --brief +} + +bench "tg search messages heron --limit 20" "$tg" ref,timestamp,excerpt --offline search messages heron --limit 20 +bench "tg messages list 1 --limit 50" "$tg" ref,timestamp,text --offline messages list 1 --limit 50 +bench "tg chats list" "$tg" ref,title --offline chats list +bench "tg contacts timeline --limit 20" "$tg" at,locator --offline contacts timeline Alice --limit 20 +bench "tg mentions list --limit 20" "$tg" kind,value,locator --offline mentions list --limit 20 +bench "memo search all heron --limit 20" "$memo" ref,timestamp,excerpt search all heron --limit 20 +bench "memo search messages heron --limit 20" "$memo" ref,timestamp,excerpt search messages heron --limit 20 diff --git a/performance/agents/output-size/seed.ts b/performance/agents/output-size/seed.ts new file mode 100644 index 0000000..65efd8d --- /dev/null +++ b/performance/agents/output-size/seed.ts @@ -0,0 +1,75 @@ +// Synthetic store for the output-size test: 5,000 invented messages in five chats, for the profile "bench". +// +// node seed.ts +// +// Nothing here reads a real account, store or message. Names, phones and addresses are invented +// (phones in the 555-01xx range, addresses at example.test). +import { mkdirSync, writeFileSync } from "node:fs" +import { join } from "node:path" + +const [root, path, stateDir] = process.argv.slice(2) +if (!root || !path || !stateDir) + throw new Error("usage: node seed.ts ") +const { openStore } = await import(join(root, "dist/store/index.js")) + +const WORDS = ["heron", "ledger", "invoice", "river", "budget", "census", "meeting", "report", "garden", "schedule"] +const SENDERS = ["Alice Example", "Bob Sample", "Carol Test", "Dan Invented", "Eve Placeholder"] +const account = { provider: "telegram", account: "100" } +const chats = [ + { id: "1", title: "Alice Example", kind: "private" }, + { id: "2", title: "Garden club", kind: "group" }, + { id: "3", title: "Project Heron", kind: "group" }, + { id: "4", title: "Reading circle", kind: "group" }, + { id: "5", title: "Budget team", kind: "group" }, +] +const PER_CHAT = 1000 +const word = (at: number) => WORDS[at % WORDS.length] as string + +const store = await openStore({ path }) +await store.saveChats( + account, + chats.map((chat) => ({ + ...chat, + unreadCount: 0, + lastMessageAt: "2026-02-01T00:00:00.000Z", + participantsCount: null, + })), +) +await store.savePeople( + account, + SENDERS.map((name, at) => ({ id: String(200 + at), name })), +) +for (const chat of chats) { + const batch = [] + for (let at = 0; at < PER_CHAT; at++) { + const extra = + at % 25 === 0 + ? ` call +1 555 010 ${String(at % 100).padStart(2, "0")} or write to person${at}@example.test` + : at % 40 === 0 + ? ` see https://example.test/docs/${at}` + : "" + const sender = chat.kind === "private" ? 0 : at % SENDERS.length + batch.push({ + id: String(at + 1), + chatId: chat.id, + senderId: String(200 + sender), + senderName: SENDERS[sender] as string, + timestamp: new Date(Date.UTC(2026, 0, 1) + (Number(chat.id) * PER_CHAT + at) * 60_000).toISOString(), + editedAt: null, + text: `Update on the ${word(at)} and the ${word(at * 7 + 3)}: we agreed on item ${at}, the owner will confirm it before the end of the week.${extra}`, + outgoing: false, + attachments: [], + replyTo: null, + forwardedFrom: null, + reactions: null, + }) + } + for (let at = 0; at < batch.length; at += 500) + await store.saveMessages(account, chat.id, batch.slice(at, at + 500), { via: "history" }) +} +await store.fillSearchIndex?.() +await store.close() + +mkdirSync(join(stateDir, "accounts"), { recursive: true }) +writeFileSync(join(stateDir, "accounts", "bench.json"), `${JSON.stringify({ account: "100" })}\n`) +process.stdout.write(`seeded ${chats.length * PER_CHAT} messages in ${chats.length} chats at ${path}\n`)