NumberNames

Number name ConLang grammars for testing frontier models

Requirements

pip3 install -r requirements.txt

pynini is the only dependency that builds against a system library. Install OpenFst first with your platform's package manager — libfst-dev on Debian/Ubuntu, openfst-devel on Fedora/RHEL, openfst on Homebrew — or use conda install -c conda-forge pynini, which ships prebuilt binaries.

To run only tasks/reasoning_api (below), pip install anthropic is enough; pynini is needed for the conlang grammars, the coding task, and scoring.

Build number names

python3 conlangs/build_all_numbers.py --top TOP_VALUE

Where TOP_VALUE is any number up to 10,000,000.

All the training examples one should need to derive a complete understanding of the system are in the "train" division.

Running the tasks

There are two tasks, each in two variants. Every variant reads the same inputs — train_data/LANG.train.jsonl for the examples, dev_in_data/LANG.dev_in.jsonl for the 500 numbers to read — and produces the same output: dev_out_data/LANG.dev_out.jsonl, with only the reading field filled in.

Task	Variant	What the model does
tasks/reasoning	harness	Reads the numbers directly, no code or tools
tasks/coding	harness	Writes a pynini grammar, runs it to produce the readings
tasks/reasoning_api	API-only	Same as reasoning, one Messages API call per language
tasks/coding_api	API-only	Same as coding, one call returns the grammar module

The _api variants are an ablation: they remove the agentic harness, so the model gets one shot with no file access, no tool use, and no iteration. The pipeline owns all file IO, prompt assembly, and grammar execution.

Harness variants

Point an agent (Claude Code or similar) at the task directory and let it follow instructions.txt. It writes dev_out_data/LANG.dev_out.jsonl itself.

Published results for previous runs are under task_out/MODEL-TASK/dev_out/.

API-only variants

Set a key, then run the pipeline:

export ANTHROPIC_API_KEY=sk-ant-...

python3 tasks/reasoning_api/pipeline.py -v
python3 tasks/coding_api/pipeline.py -v

Both accept --model (default claude-opus-4-8), --langs for a subset, and --effort. Useful before spending anything:

# Assemble the prompts and count tokens; makes no generation calls.
python3 tasks/reasoning_api/pipeline.py --dry-run --langs ferulian

# Smoke test on 10 numbers. Writes to dev_out_data_smoke/ so the scorer
# cannot mistake a truncated run for a full one.
python3 tasks/reasoning_api/pipeline.py -v --limit 10 --langs ferulian

reasoning_api sends 250 numbers per call (--batch-size, 0 for the whole language at once). Large batches avoid re-deriving the numeration system on every call, but a reply that hits max_tokens loses every number in its batch, so the batch size is the blast radius of a truncated reply. On Opus 4.8 at effort=high a 500-number call measured ~62k output tokens, nearly all of it thinking, so give --max-tokens plenty of room. coding_api needs pynini, since it executes the grammar it gets back — it exits before spending anything if pynini is missing.

To sweep several models across both tasks, score them, and file the results under task_out/ in the same layout as the harness runs:

bash run_experiments.sh --dry-run    # check prompts and token counts first
bash run_experiments.sh              # the real thing

It resumes by default — languages that already have output are skipped, so an interrupted sweep can be restarted without paying twice. MODELS, TASKS, and LANGS override what it covers.

Each run writes everything it did to tasks/TASK/raw/: the exact prompt sent (*.prompt.txt, written before the call, so a run that dies mid-flight still records what it asked), the full model replies (*.md), and run_manifest.json with per-call request IDs, token usage, stop reasons, and an estimated cost. Check the manifest before comparing scores: score.py counts a blank reading exactly like a wrong one, so the manifest is what distinguishes an API or execution failure from a genuine model error.

Scoring LLM outputs

Score every language of one task at once:

bash score.sh reasoning_api

Or a single file:

python3 score.py --test=task_out/claude-opus-4-8-reasoning/dev_out/archiferulian.dev_out.jsonl --verbose

Output will be in task_out/claude-opus-4-8-reasoning/dev_out/archiferulian.dev_out.txt.

Plot the results

python3 plot_error_rates.py task_out/claude-opus-4-8-coding/dev_out

See the results for claude-sonnet-4-5-20250929 in coding, reasoning.