refactor(core)!: Destination stores one partition at a time; PartitionedDestination folds into it - #331
Open
aaaaahaaaaa wants to merge 1 commit into
Open
refactor(core)!: Destination stores one partition at a time; PartitionedDestination folds into it#331aaaaahaaaaa wants to merge 1 commit into
aaaaahaaaaa wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…nedDestination folds into it A destination stores data per scope, one partition or the unpartitioned whole. The scope dispatch lives in one place, IOContext.scopes and IOContext.slices, and Destination.write and .read are the template over it; the public hooks write_partition and read_partition are what a destination implements. PartitionedDestination goes: every shipped destination extended it, and the plain read/write shape the base class taught first was the one that overwrote partitions. DatabaseDestination keeps its batch write for windows and gains a write_partition for single scopes, with the three-way branching and the asserts gone from it. By Digitl
aaaaahaaaaa
force-pushed
the
refactor/destination-scopes
branch
from
September 9, 2026 17:50
0c09730 to
75be503
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Items 1 to 3 of the destination surface review.
PartitionedDestinationis gone. Every shipped destination extended it (CSV, File, Memory,GCS, and
DatabaseDestinationunder BigQuery), while theDestinationdocstring and the firstguide example taught the plain
read/writeshape that overwrites every partition. The scopetemplate is now
Destinationitself: a destination is partition-correct by construction.write_partition(context, partition, data)andread_partition(context, partition)are what a destination implements; a missing one raises naming itself. Overridingwrite/readwholesale stays possible for a backend that scopes differently, and is documentedas that exception rather than as an alternative for everyone.
IOContext.scopeslists the scopes a context covers(
[None],[partition], or the window's partitions) andIOContext.slices(data)pairs each withits slice of the data.
Destination.writeand.readare a loop over them; the three copies of theNone / Partition / PartitionWindowbranching and theirasserts are gone, including fromDatabaseDestination.write, which keeps its window batching (clear every scope, insert once) andgains a
write_partitionfor single scopes.Migration
class X(il.PartitionedDestination)class X(il.Destination)def _write_partition(...)/def _read_partition(...)def write_partition(...)/def read_partition(...)class X(il.Destination)overridingread/writefor a partitioned assetThe guide, README, the destination skill and the upgrade skill's migration table are updated.
Verification
tests/destination/test_partitioned.pyfolded intotest_base.pyon the public hooks; newtest_context.pycoversscopes,slicesand the window-on-unpartitioned-asset error.By Digitl