JSON Lexer + Parser from Scratch

A JSON lexer and recursive-descent parser built character-by-character in pure Python — no json, no regex-driven shortcuts on the hot path — with a CLI, a typed error taxonomy, a differential conformance harness against Python's stdlib, and a benchmark/stress suite that produces real numbers.

▶ Live pipeline playground — this parser running in your browser via Pyodide. Watch the token stream come out of the real lexer, and every run is a differential test against the RFC-strict stdlib. The page fetches lexer.py and parser.py from this repo unmodified — check the network tab.

Superset solution to the Build-Your-Own-JSON-Parser coding challenge.

The pipeline

• JSONLexer (src/lexer.py) — a single-pass character state machine: quote state, escape-pair handling (\", \\, \uXXXX), and a bracket stack that stamps every token with its nesting level — which is what lets the parser enforce bounded depth without recursing to find out.
• JSONRDParser (src/parser.py) — textbook recursive descent, one function per grammar rule (parse_value → parse_object | parse_array | parse_string | parse_number), a strict peek/consume(expected) discipline, and RFC-8259 number validation.
• Every failure is typed and positioned — [UNEXPECTED_TOKEN_ERROR], [NUMBER_MISMATCH_ERROR], [STRING_MISMATCH_ERROR], [DUPLICATE_KEY_ERROR], [MAX_DEPTH_EXCEEDED_ERROR], [END_OF_INPUT_ERROR], [TRAILING_TOKENS_ERROR], [EXTRA_CLOSING_TOKENS_ERROR] — with the token index where parsing stopped.

CLI

python src/cli.py validate file.json        # exit 0 valid / 1 invalid (typed error + index)
python src/cli.py tokens file.json --stats  # token counts by type + max nesting level
python src/cli.py conformance               # differential run vs stdlib json
python src/cli.py bench --mb 8              # throughput + stress drills
python src/cli.py test                      # the bundled 50-file suite

Conformance — measured, not asserted

src/conformance.py runs 98 cases (50 bundled files + 48 edge cases: escapes, unicode, exponent forms, leading zeros, truncations, trailing commas, structural garbage) through this parser AND through Python's stdlib json in RFC-strict mode (parse_constant rejecting Infinity/NaN, which bare stdlib accepts as a documented Python extension):

conformance vs python stdlib json — 98 cases
  agree                : 90
  documented deviations: 8
  DISAGREE             : 0

The 8 deviations are deliberate strict-mode policy, not accidents:

#	This parser	stdlib	Why
D1	Rejects duplicate object keys	keeps the last silently	silent data loss is a bug factory
D2	Requires object/array at top level	accepts bare 42, "str"	document-shaped inputs only
D3	Bounded nesting: depth ≤ 20	~1000 (then RecursionError)	bounded input handling by policy

Measured performance

Apple M5, pure Python (best of 3), deterministic seeded corpus:

Metric	Value
Lexing	572K tokens/sec — 5,136,643 tokens over a 12 MB corpus in 8.97 s
Full lex+parse	460K tokens/sec (11.16 s for the same corpus)
vs stdlib json (C extension)	stdlib is ~163× faster — the honest pure-Python-vs-C ratio
Depth-25 nesting	rejected (MAX_DEPTH_EXCEEDED, bounded at 20)
50,000-key object	parsed OK in 0.78 s
2 MB single string token	OK but quadratic (~30 s) — known limit: per-char append on one giant token

Repo layout

src/lexer.py         the state machine
src/parser.py        recursive descent + error taxonomy
src/cli.py           jsonlp CLI (validate / tokens / conformance / bench / test)
src/conformance.py   differential harness vs stdlib
src/bench.py         corpus generator + throughput + stress drills
tests/               50 pass_*/fail_* files

Honest scope

Pure-Python by intent — the project is about owning the algorithm (state machines, recursive descent, grammar-to-code mapping), and the differential harness + typed errors are the production discipline around it. The quadratic single-giant-token case is documented above rather than hidden; fixing it (chunked scanning instead of per-char append) is the obvious next milestone.