AI build tests

Source code for the SideButton build tests: one brief, handed to several AI coding setups, with everything they produced published side by side.

Write-ups and playable builds: https://sidebutton.com/ai-build-tests

These are not benchmarks. A benchmark scores a model against a standard task set and reports a number. These take one real brief, run it several ways, and hand you the output to judge yourself. The interesting result is usually not which build scored highest. It is which features quietly went missing, and why.

Layout

One directory per experiment, one subdirectory per arm.

<experiment>/
  BRIEF.md        what every arm was asked to build
  README.md       the arms, the results, a link to the write-up
  <arm>/          that arm's complete source, exactly as measured
    ARM.md        how this arm was run

Experiment	Arms	Write-up
asteroids	sonnet, opus, sidebutton	Asteroids, Three Ways
wisp	opus, sidebutton (running)	not published yet

Directory names are stable identifiers. The site derives every source link from them, so <experiment>/<arm> is also how an arm is named in the write-up and in the comparison table.

The rules these runs follow

Every arm gets the same brief. Not a paraphrase of it. BRIEF.md is what they received, so you can check rather than take our word for it.

Every arm starts from the same scaffold. Contracts, fixtures, budget ceilings and lint rules are the starting line, not part of the contest. In asteroids you can verify this directly: src/types.js, src/state.js and fixtures/mid-game.json are byte-identical across all three arms, while every implementation file differs.

Published performance numbers come from one machine. A frame-time comparison across different graphics cards measures the graphics cards.

What is deliberately left free is how each arm gets there: how many agents, which model, what review process. That is the variable under test, so fixing it would defeat the exercise.

These directories are frozen

Each arm holds the code as it was when measured. That is what makes the numbers in the write-ups checkable, so arm directories are not maintained, refactored or kept current. If something in one is wrong, the fix is a commit that says what changed and why, never a silent edit.

Expect the code to look its age, and expect the one-shot arms in particular to contain bugs. Publishing them unretouched is the point.

Checking a number

Every figure in a write-up should be reproducible from this repository. For example, the Asteroids comparison table's code, test and documentation line counts:

# lines of implementation code, per arm
find asteroids/sonnet/src -name '*.js' | xargs wc -l | tail -1

# lines of tests
find asteroids/sidebutton/test -name '*.js' | xargs wc -l | tail -1

If a number here does not match a number there, the write-up is wrong and we would like to know. Open an issue.

Licence

MIT. See LICENSE.

The arms were written by AI coding agents against the briefs in this repository. They are published as experimental artifacts rather than as maintained software.