Skip to content

refactor(core)!: Destination stores one partition at a time; PartitionedDestination folds into it - #331

Open
aaaaahaaaaa wants to merge 1 commit into
mainfrom
refactor/destination-scopes
Open

refactor(core)!: Destination stores one partition at a time; PartitionedDestination folds into it#331
aaaaahaaaaa wants to merge 1 commit into
mainfrom
refactor/destination-scopes

Conversation

@aaaaahaaaaa

@aaaaahaaaaa aaaaahaaaaa commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Items 1 to 3 of the destination surface review.

  • PartitionedDestination is gone. Every shipped destination extended it (CSV, File, Memory,
    GCS, and DatabaseDestination under BigQuery), while the Destination docstring and the first
    guide example taught the plain read/write shape that overwrites every partition. The scope
    template is now Destination itself: a destination is partition-correct by construction.
  • The hooks are public. write_partition(context, partition, data) and read_partition(context, partition) are what a destination implements; a missing one raises naming itself. Overriding
    write/read wholesale stays possible for a backend that scopes differently, and is documented
    as that exception rather than as an alternative for everyone.
  • The three-way dispatch exists once. IOContext.scopes lists the scopes a context covers
    ([None], [partition], or the window's partitions) and IOContext.slices(data) pairs each with
    its slice of the data. Destination.write and .read are a loop over them; the three copies of the
    None / Partition / PartitionWindow branching and their asserts are gone, including from
    DatabaseDestination.write, which keeps its window batching (clear every scope, insert once) and
    gains a write_partition for single scopes.

Migration

Before After
class X(il.PartitionedDestination) class X(il.Destination)
def _write_partition(...) / def _read_partition(...) def write_partition(...) / def read_partition(...)
class X(il.Destination) overriding read/write for a partitioned asset implement the two partition hooks instead

The guide, README, the destination skill and the upgrade skill's migration table are updated.

Verification

  • tests/destination/test_partitioned.py folded into test_base.py on the public hooks; new
    test_context.py covers scopes, slices and the window-on-unpartitioned-asset error.
  • Whole repo: 2820 passed, ruff and ty clean.

By Digitl

@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.71795% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...rloper-core/src/interloper/destination/database.py 97.29% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…nedDestination folds into it

A destination stores data per scope, one partition or the unpartitioned whole. The scope
dispatch lives in one place, IOContext.scopes and IOContext.slices, and Destination.write and
.read are the template over it; the public hooks write_partition and read_partition are what a
destination implements. PartitionedDestination goes: every shipped destination extended it,
and the plain read/write shape the base class taught first was the one that overwrote
partitions. DatabaseDestination keeps its batch write for windows and gains a write_partition
for single scopes, with the three-way branching and the asserts gone from it.

By Digitl
@aaaaahaaaaa
aaaaahaaaaa force-pushed the refactor/destination-scopes branch from 0c09730 to 75be503 Compare September 9, 2026 17:50
@aaaaahaaaaa aaaaahaaaaa changed the title refactor(core)!: Destination stores one scope at a time; PartitionedDestination folds into it refactor(core)!: Destination stores one partition at a time; PartitionedDestination folds into it Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant