Repository navigation
Conversation
An operator declaring `UpToCapability` is woken by a frontier change only when the change removes an element less or equal to a capability it holds. Changes entirely above its capabilities, including all changes once the frontier has passed every capability it holds, do not schedule it. A subgraph's child zero now declares `UpToCapability`, so a region whose children are all `Never` or `UpToCapability` sleeps through frontier moves that cannot affect the capabilities it holds. A child declaring `IfCapability` or `Always` still raises the region to that interest.
The wake check compared a removed frontier element against the tracker's source pointstamps. Those sum the operator's capabilities over all workers as of the last progress exchange, so a capability held by this worker's instance can sit above every element they show. The frontier then passes it without waking the operator, and the other workers' later downgrades change only capabilities, never the input frontier, so nothing re-checks. Each child with an `UpToCapability` input now keeps a tally of the capabilities it holds on this worker, summed over its outputs and updated as `extract_progress` drains its internals. The check uses that tally. A region's tally is the capabilities it reports, as before.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
FrontierInterest::UpToCapability, and makes it the interest of a subgraph's child zero, so it becomes the floor for regions instead ofIfCapability.IfCapabilitywakes an operator on any frontier change while it holds any capability.UpToCapabilitywakes it only when a frontier change removes an element less or equal to a capability that this worker's instance of it holds. Only such a change can complete the time of a held capability, or a time greater than one. A region holding a capability the frontier has already passed is no longer scheduled every time the frontier moves.IfCapabilitykeeps its meaning. A child that declares it still raises its region toIfCapabilitythrough the existing max over children. Neither differential-dataflow nor Materialize declaresIfCapabilitytoday (both use onlyNeverand the defaultAlways), so in practice this changes behaviour for regions whose children are allNever.I'd like to discuss whether this should be its own variant, as here, or whether
IfCapabilityshould take on this meaning.Benchmark
One input fans out to R regions. The input frontier advances one epoch at a time, and each epoch steps the worker until a probe on the input passes. Each row is the median of 9 interleaved runs of master (
5d82cb0b) and this branch (335a6943), on an Apple Silicon laptop. The schedule counts are total operator schedules from a separate run with logging on.lag: each region holds a capability at time 0, which the frontier passes immediately.ahead: each region holds a capability past the last epoch, so the frontier stays below it.busy: each region maps, exchanges and maps one record per epoch. Its capabilities come from records in flight, so every frontier move is relevant.lagis where this helps: 0.27x to 0.42x per epoch.aheadandbusyare unchanged, so the extra check costs nothing measurable. The 4-worker rows are noisy: rerunningbusyat 4 workers with 15 reps gave 0.93x (100 regions) and 0.95x (10 regions), with interquartile ranges near 100µs on both builds.Benchmark source
Usage:
bench <lag|ahead|busy> <regions> <epochs> <workers> [count]Whose capabilities
The check uses the capabilities held by this worker's instance of the operator. A first version compared against the tracker's source pointstamps, and that was wrong with several workers. Those pointstamps sum the operator's capabilities over all workers as of the last progress exchange, so one worker's capability can sit above everything they show. The frontier then passes it without waking the operator. When the other workers catch up, only capabilities change, not the input frontier, so nothing re-checks, and the operator sleeps holding a capability that holds back everything downstream. This stalled min/max and top-k dataflows in a 16-worker Materialize test, and not with one worker.
Each child with an
UpToCapabilityinput now keeps a tally of the capabilities it holds on this worker, summed over its outputs and updated asextract_progressdrains its internals. Other children keep no tally. For a region, the tally is the capabilities it reports, which are its own view across workers, as before. If that view lags, the region has a progress exchange pending, and receiving it schedules the region, which then accepts the current frontier.Caveats
An operator declaring
UpToCapabilitymust not rely on a passed capability to stay awake. Every time it waits on must be at or above a held capability that the frontier has not yet passed. If it holds a capability at 5 that the frontier passed earlier, with pending work at 10, a frontier move from 9 to 11 does not wake it. This holds for partially ordered times as well: if some held capabilityc <= tis not yet passed, some frontier elementf <= cexists, and the change that completestmust removef. Operators that want the old behaviour keep declaringIfCapability.Regions rely on their capabilities reaching a scope output. The parent judges a region by the capabilities at its outputs. A region's children are all
NeverorUpToCapability, and path summaries do not decrease times, so a frontier change a nestedUpToCapabilitychild cares about removes a scope-input element at or below that capability's image at the scope outputs. The same argument covers lazy progress (ProgressMode::Demand): a withheld update at timetbecomes globally visible only after the frontier elements belowtare removed. The argument fails for pointstamps that reach no scope output, such as records in flight to a dead-end operator inside the region. Today those are only missed when the region holds no output capability at all (cap_counts == 0). With this change they are also missed when the region holds unrelated output capabilities. The likely effect is a delayed progress broadcast for that pointstamp, until peers' progress, arriving records or the shutdown check schedule the region. This is reasoning only; I have not hit it in a test.Capabilities ahead of the frontier still wake the region. A stricter test would wake only when a held capability goes from not passed to passed. Measured as an experiment, that brought
aheadto 0.36x to 0.45x as well. It is not sound for regions, though: with a summary that advances time between the scope input and a nested child, the child's frontier can pass its capability while the outer frontier has not passed the projected one.Cost. For each removed frontier element, the check scans the distinct capability times the operator holds on this worker. Each operator with an
UpToCapabilityinput also updates its tally as its capabilities change. A tally per output madeshape_scaling's region with 100,000 outputs quadratic at shutdown, so the tally is summed over outputs. Holding many distinct capability times, as partially ordered times can, makes the scan longer.The variant sits between
NeverandIfCapabilityso the max over children stays correct. That shifts the discriminants of the loggedFrontierInterestfrom Log each operator's frontier interest with its summary #836, which has not been released yet.Tests
capability_interest.rs: a region holding a passed capability sleeps (101 schedules over 100 rounds on master, under 10 here). A capability ahead of the frontier still wakes the region each round, and so does a child declaringIfCapability.capability_interest_workers.rs: with two workers, worker 0 holds a capability at 5 while worker 1 still holds one at 0, and the frontier moves from 3 to 10. Worker 0 must be woken and release its capability. It fails 3/3 against the first commit and passes 3/3 with the fix.frontier_interest_log.rsnow expects a quiet region to reportUpToCapability.cargo test --workspace --all-targetsandcargo test --docpass. Clippy reports no new warnings.Generated with Claude Code