fix(#0926): bound Lance metadata schema widening

[DevNexsler]

Summary

• stream Lance metadata struct widening one row at a time instead of materializing the full table
• derive the row envelope from the effective cgroup-v2 memory limit and warn with fields/row count before rewrite
• exclude ordinary commits during the atomic schema swap while preserving concurrent normal writes
• cover 66,000-row memory behavior, cgroup hierarchy, observability, oversized rows, and concurrency

Test plan

• make gate: PASS — 1,539 unit, 77 integration, 31 staging E2E, 167 live passed/4 skipped; 21/21 tools covered/traced
• exact-image candidate 37a5ea4291161e4ad7d96a4d53f16e5def88d055: 31/31 E2E and 3/3 ticket regressions; healthy, zero restarts, no OOM
• candidate image: sha256:66840275da9d27bd24b917d1fb074e12a5032e728b1f8722e149ecb7c1a873ee

Follow-up

• PR #0584 record extraction provenance so content-free docs are not indexed as success #79 remains open and DIRTY. This ticket cannot modify another open PR; after #0926 merges, #0584 record extraction provenance so content-free docs are not indexed as success #79 owner must rebase it onto main and rerun its gate.

---

Maint ticket: #0926 — Maint-Manager tickets/review/0926-lancedb-metadata-struct-widening-is-an-unbounded-w.md
On merge: verify this ticket's acceptance boxes against production (not tests),
tick them, then move the ticket to tickets/done/ and set status: done.
A merged PR is not a closed ticket — an unmerged PR is not a shipped fix.

[DevNexsler]

Merged to main via reconcile branch reconcile/20260810 — merge commit f58198c.

Independently reproduced. Read _evolve_metadata_schema on origin/main: it calls table.to_arrow() on the whole table then .combine_chunks(), with no streaming, row-count guard or allowlist, and _write_nodes_unlocked fires it on any unseen metadata key. Live production state confirms the scale: chunks.lance has 67,918 rows / 254 fragments / 102 metadata sub-fields (read read-only via lance.dataset), and the container is at 6.478 GiB of its 8 GiB memory.max (memory.current 7.09 GiB) — under 1 GiB of headroom against a multi-GB transient.

Also confirmed the stated coupling to PR #79: neither content_status nor content_failure_reasons exists in the live 102-field struct, so #79 would fire this path on its first index run.

Red/green: 3 tests RED on origin/main (test_schema_evolution_streams_large_table_within_cgroup_envelope, test_schema_evolution_reads_real_cgroup_memory_limit, test_schema_evolution_logs_fields_and_row_count_before_widening), GREEN after merge.

⚠️ Protected surface: this changes how the Lance metadata struct column is rewritten and swapped — a storage-schema path. Flagged for after-the-fact review. It was merged because the defect is reproducible, the red/green is real, and the full gate plus the 31-test candidate-container tier passed.