Streaming deserialization part 2: transformations

[bjester]

Summary

This PR moves the transform logic in _deserialize_from_store into the Source → Transform → Sink streaming architecture. It follows the structure already used for streaming serialization. The deserialization sink, the entry point and operation wiring, and removing the legacy code come in a later part.

• Outcome model: DeserializeTask now has an outcome property that returns one of the DeserializeOutcome values: SAVE, DELETE, PROPAGATE_DELETE, PROPAGATE_HARD_DELETE or ERROR. The transforms only read and resolve the outcome. The future sink does every write.
• AppModelDeserialize: builds the unsaved app model from the store record's serialized data and sets the morango source ID, partition and dirty bit. Deleted store records pass through untouched. JSON and validation errors are recorded on the task.
• AppModelValidate:
  • Validates each app model with cached_clean_fields, using the shared fk_cache.
  • Tracks which records failed or were deleted during the run. A record that references one of them gets a propagated deletion or a dirty-parent error, without any DB query.
  • On a validation failure it runs one Store query to classify the cause. The checks, in order: a reference to a deleted record propagates the deletion; then MorangoMissingParent, then MorangoDirtyParent, then an IntegrityError for a missing target. Otherwise the original error is kept. The error messages match the legacy ones.
• Test Facility model: gains a clean_dereferences_parent flag. Tests can now cover both the current behavior, where clean_fields loads the parent, and the default expected behavior of production models, which don't load related objects (checked against Kolibri).

Why this approach handles FK references. The sink may not have written a record yet when a record that references it is validated. The existing ordering guarantees (#297 and #318) cover this together with fk_cache:

• Partition prefixes are sorted with the shortest, shared prefix first.
• The registry orders models by FK dependency, and records within a self-referential model are ordered parent first.
• So by the time a record is validated, its FK targets have already been validated and added to fk_cache.
• A cache miss falls back to a DB lookup, which covers targets that are unchanged and not part of the run.

References

• Part 2 of 3 towards: Reduce memory overhead of _deserialize_from_store #210
• Follows Streaming deserialize: add source stream and task #318, part 1: the deserialization source and task (closes Implement DeserializeTask and StoreModelSource for streaming deserialization #317)
• Builds on the ordering work in Tracking Self-Referential FK Order #297

Reviewer guidance

• Integration tests (DeserializeTransformsIntegrationTestCase) run real Store rows through StoreModelSource → AppModelDeserialize → AppModelValidate into a sink that only collects results, so no app rows are ever written. This reproduces the window where the sink hasn't caught up. The scenarios include:
  • self-referential and cross-model chains, run with the Facility flag both on and off
  • a parent that is not in the run, failed in the run, is missing, or was deleted in this run or an earlier one
  • a missing target
  • a check that the transforms issue no writes (no INSERT, UPDATE or DELETE)
• Legacy parity tests (LegacyParityTestCase) run the transforms and then _deserialize_from_store on the same fixtures. They compare the errors recorded for the failed-parent, missing-parent and missing-target scenarios.
• I ran the full suite locally on both SQLite (584 passed, 2 skipped) and Postgres 9.6 (578 passed, 8 skipped).
• Known trade-off: fk_cache grows for the whole run. We accepted this for now.
• Does the split of responsibilities between AppModelDeserialize and AppModelValidate look right before the sink is built on top of it?

AI usage

I used Claude Code throughout:

• Spec and plan. I guided Claude on how the existing ordering guarantees and fk_cache should combine to handle FK references. I corrected it when it proposed changes to the source ordering that were out of scope. I also had it check Kolibri's models to confirm production clean_fields implementations don't load related objects.
• Implementation. Claude implemented the plan in three phases. I reviewed each phase before it was committed and asked for changes where needed, for example constants for the deletion types.
• Tests. Claude wrote the tests, including the legacy parity tests, which compare the new transforms' results with the existing implementation's.
• PR description. Claude also drafted this description.

[rtibblesbot]

🟡 Waiting for changes

Last updated: 2026-10-09 18:40 UTC

[rtibblesbot]

PR #390: one parity bug in AppModelValidate's error classification; the rest is sound.

CI pending. No UI files changed.

• blocking: child's own validation error reported as MorangoDirtyParent when its parent was accepted earlier in the run (inline).
• suggestion: ValueError from validation aborts the pipeline instead of failing one record (inline).
• suggestion: validation failures are no longer logged (inline).
• suggestion: parity tests don't cover deletion propagation (inline).
• praise: failure-only, batched Store lookup, pinned by query-count tests.

---

Written by rtibblesbot, an LLM-based coding agent.

[rtibblesbot]

No new commits since my last review (d758c69).

---

Written by rtibblesbot, an LLM-based coding agent.

[rtibblesbot]

PR #390: no new findings. 3 of 4 prior findings are resolved. The fourth, missing failure logging, is deferred to the part 3 sink. CI is pending.

Prior-finding status

RESOLVED — morango/sync/stream/deserialize.py — child's own validation failure reported as MorangoDirtyParent when parent accepted in run
RESOLVED — morango/sync/stream/deserialize.py — ValueError from validation aborts the whole run
ACKNOWLEDGED — morango/sync/stream/deserialize.py:277 — validation failures no longer logged (deferred to part 3 sink)
RESOLVED — tests/testapp/tests/sync/stream/test_deserialize.py:999 — parity tests don't cover deletion propagation

---

Written by rtibblesbot, an LLM-based coding agent.

Superseded by my review of c232030: no blocking findings remain.