fix!: v4 correctness pass — truncated 300 rows, missing-vs-zero readings, typed errors, dead sql removal

[cohenrobinson]

Draft. Breaking major (4.0.0). Each finding below has a test that fails on main and passes here.

Baseline on main: 163 tests, mypy --strict clean, ruff check clean. Here: 175 tests pass (3 removed with tests/test_sql.py, 15 added/changed), mypy/ruff/format all clean.

1. Silent truncation of over-long 300 rows

Before: _emit_intervals and parse_to_columns only checked len(row) >= 2 + n. A 15-minute file under a stale 30-minute 200 header emitted 48 readings, silently dropped the other 48, and read quality_method out of a dropped reading value. validate_file did not check 300 field counts at all — it returned clean for a 30-minute NMI carrying 288 values.

After: a 300 row may carry at most INTERVAL_TRAILER_FIELDS = 5 trailing fields after the values (QualityMethod, ReasonCode, ReasonDescription, UpdateDateTime, MSATSLoadDateTime — MDFF v2.6 §4.4). Anything longer raises NEM12ParseError naming the NMI, the interval date, the actual value count and the expected one. validate_file reports both too-short and too-long 300 rows (skipped when IntervalLength does not divide a day, which _parse_nmi still tolerates).

2. Blanks must not become 0.0 (breaking)

Before: _parse_float("") returned 0.0 — indistinguishable from a genuine zero reading in a billing input — for NEM12 300 values and NEM13 previous_register_read / current_register_read / quantity.

After: reading values are float | None, propagated through IntervalReading, AccumulationReading, the columnar/pandas/parquet paths and the CSV writer (a missing value writes an empty cell). aggregate.daily_totals skips None rather than summing it as zero, excludes it from interval_count, and reports it in the new DailyTotal.missing_count — a day with missing_count > 0 is an incomplete total, not a lower one. mypy --strict still passes.

3. Raw ValueError escaping the typed-error boundary

Before: NEM12ParseError is the advertised contract, but _parse_int, _parse_float, the columnar float(cell), _parse_interval_event's int(row[1])/int(row[2]), and the fixed-width datetime(...) constructor all leaked bare ValueError — e.g. N/A in a value cell, 400,x,10,S, or 300,20049999 → "month must be in 1..12".

After: each is raise NEM12ParseError(...) from exc, with the offending field and value in the message. Tests assert NEM12ParseError specifically.

4. Swapped direction constants

spec.DIRECTION_INDICATORS had Import and Export the wrong way round. Import is grid → connection point; Export is connection point → grid.

5. Deleted the dead sql/ subpackage

sql/store.py called reader.data.VersionHeader / .subrecords / subsubrecord.__getattr__, none of which exist on NEMReader — every entry point raised AttributeError immediately. It was excluded from mypy and coverage yet still shipped, and QueryBuilder.insert_query interpolated unescaped file-derived values into SQL (injection surface via ReasonDescription). Removed the subpackage, tests/test_sql.py, the [mysql] extra, and the now-redundant mypy-override / coverage-omit / ruff per-file-ignore / bandit-exclude / dependabot entries, plus the README, SECURITY.md and requirements.txt references.

Also

400 StartInterval / EndInterval are bound-checked against the NMI's intervals per day — 400,1,999,S previously parsed happily against a 48-interval day.

Not done (out of scope, as agreed): applying 400-record events to interval quality — that's a design change, not a correctness fix.

Verification

175 passed in 0.30s
Success: no issues found in 7 source files   # mypy --strict
All checks passed!                           # ruff check
35 files already formatted                   # ruff format --check

Coverage 92.42% (gate: 90%).

https://claude.ai/code/session_01SdzSFrjvRTUyjpAPCvJpGC

[cohenrobinson]

Hot-path check (template asks for it — the columnar loop changed): `python benchmarks/bench_parser.py --nmis 2 --days 90 --interval-minutes 5`, same machine, back to back.

	main	this branch
parse(path) for-loop	32.6 ms	32.4 ms
NEMReader().read_from_file	36.1 ms	34.8 ms
to_columns(parse(path))	40.2 ms	38.1 ms
parse_to_columns(path) fast path	10.9 ms	10.3 ms
to_dataframe(path) fast path	47.0 ms	45.9 ms

No regression — the added try/except around float(cell) costs nothing on the non-raising path.