Skip to content

Add FrontierInterest::UpToCapability and use it for regions - #837

Open
DAlperin wants to merge 2 commits into
TimelyDataflow:masterfrom
DAlperin:up-to-capability-interest
Open

DAlperin wants to merge 2 commits into
TimelyDataflow:masterfrom
DAlperin:up-to-capability-interest

Conversation

@DAlperin

@DAlperin DAlperin commented Nov 16, 2026 •

Copy link
Copy Markdown

Adds FrontierInterest::UpToCapability, and makes it the interest of a subgraph's child zero, so it becomes the floor for regions instead of IfCapability.

IfCapability wakes an operator on any frontier change while it holds any capability. UpToCapability wakes it only when a frontier change removes an element less or equal to a capability that this worker's instance of it holds. Only such a change can complete the time of a held capability, or a time greater than one. A region holding a capability the frontier has already passed is no longer scheduled every time the frontier moves.

IfCapability keeps its meaning. A child that declares it still raises its region to IfCapability through the existing max over children. Neither differential-dataflow nor Materialize declares IfCapability today (both use only Never and the default Always), so in practice this changes behaviour for regions whose children are all Never.

I'd like to discuss whether this should be its own variant, as here, or whether IfCapability should take on this meaning.

Benchmark

One input fans out to R regions. The input frontier advances one epoch at a time, and each epoch steps the worker until a probe on the input passes. Each row is the median of 9 interleaved runs of master (5d82cb0b) and this branch (335a6943), on an Apple Silicon laptop. The schedule counts are total operator schedules from a separate run with logging on.

  • lag: each region holds a capability at time 0, which the frontier passes immediately.
  • ahead: each region holds a capability past the last epoch, so the frontier stays below it.
  • busy: each region maps, exchanges and maps one record per epoch. Its capabilities come from records in flight, so every frontier move is relevant.
scenario regions workers master µs/epoch branch µs/epoch ratio master schedules branch schedules
lag 10 1 4.76 1.78 0.37x 280021 80031
lag 100 1 52.81 18.68 0.35x 520201 20301
lag 1000 1 619.67 260.86 0.42x 1006001 7001
ahead 10 1 7.67 7.58 0.99x 280031 280031
ahead 100 1 88.26 87.51 0.99x 520301 520301
ahead 1000 1 999.64 982.78 0.98x 1007001 1007001
busy 10 1 25.36 25.27 1.00x 1700000 1700000
busy 100 1 264.89 262.05 0.99x 4025000 4025000
lag 100 4 87.45 23.72 0.27x 2118234 112954
ahead 100 4 127.41 119.74 0.94x 2125510 2126135
busy 100 4 623.29 661.59 1.06x 7452467 7454635

lag is where this helps: 0.27x to 0.42x per epoch. ahead and busy are unchanged, so the extra check costs nothing measurable. The 4-worker rows are noisy: rerunning busy at 4 workers with 15 reps gave 0.93x (100 regions) and 0.95x (10 regions), with interquartile ranges near 100µs on both builds.

Benchmark source

Usage: bench <lag|ahead|busy> <regions> <epochs> <workers> [count]

use std::cell::Cell;
use std::rc::Rc;
use std::time::Instant;

use timely::container::CapacityContainerBuilder;
use timely::dataflow::channels::pact::{Exchange, Pipeline};
use timely::dataflow::operators::generic::operator::Operator;
use timely::dataflow::operators::{Enter, Input, Leave, Probe};
use timely::dataflow::operators::vec::Map;
use timely::logging::{TimelyEvent, TimelyEventBuilder};

fn main() {
    let args: Vec<String> = std::env::args().collect();
    let scenario = args[1].clone();
    let regions: usize = args[2].parse().unwrap();
    let epochs: u64 = args[3].parse().unwrap();
    let workers: usize = args[4].parse().unwrap();
    let count = args.get(5).map(|s| s == "count").unwrap_or(false);

    let results = timely::execute(timely::Config::process(workers), move |worker| {
        let index = worker.index() as u64;
        let schedules = Rc::new(Cell::new(0u64));
        if count {
            let schedules = Rc::clone(&schedules);
            worker.log_register().unwrap().insert::<TimelyEventBuilder, _>("timely", move |_time, data| {
                for (_, event) in data.iter().flat_map(|d| d.iter()) {
                    if let TimelyEvent::Schedule(s) = event {
                        if s.start_stop == timely::logging::StartStop::Start { schedules.set(schedules.get() + 1); }
                    }
                }
            });
        }

        let scenario = scenario.clone();
        let held: Rc<std::cell::RefCell<Vec<_>>> = Default::default();
        let held_outer = Rc::clone(&held);
        let (mut input, probe) = worker.dataflow::<u64, _, _>(|scope| {
            let (input, stream) = scope.new_input::<Vec<u64>>();
            for _ in 0..regions {
                let scenario = scenario.clone();
                let held = Rc::clone(&held);
                scope.region(|inner| {
                    let s = stream.clone().enter(inner);
                    let s = match scenario.as_str() {
                        "lag" | "ahead" => {
                            let hold_at = if scenario == "lag" { 0 } else { epochs + 1 };
                            s.unary::<CapacityContainerBuilder<Vec<u64>>, _, _, _>(Pipeline, "Hold", move |cap, info| {
                                held.borrow_mut().push((cap.delayed(&hold_at), inner.activator_for(info.address)));
                                move |input, output| {
                                    input.for_each(|cap, data| { output.session(&cap).give_container(data); });
                                }
                            })
                        }
                        "busy" => {
                            s.map(|x| x + 1)
                             .unary::<CapacityContainerBuilder<Vec<u64>>, _, _, _>(Exchange::new(|x: &u64| *x), "Xchg", |_cap, _info| {
                                 move |input, output| {
                                     input.for_each(|cap, data| { output.session(&cap).give_container(data); });
                                 }
                             })
                             .map(|x| x + 1)
                        }
                        other => panic!("unknown scenario {other}"),
                    };
                    s.leave(scope);
                });
            }
            let (probe, _) = stream.probe();
            (input, probe)
        });

        let start = Instant::now();
        for epoch in 0..epochs {
            if scenario == "busy" { input.send(epoch + index); }
            input.advance_to(epoch + 1);
            while probe.less_than(&(epoch + 1)) { worker.step(); }
        }
        let elapsed = start.elapsed();
        for (cap, activator) in held_outer.borrow_mut().drain(..) { drop(cap); activator.activate(); }
        if count { worker.log_register().unwrap().remove("timely"); }
        (elapsed, schedules.get())
    }).unwrap().join();

    let mut max = std::time::Duration::ZERO;
    let mut total_schedules = 0;
    for r in results {
        let (e, s) = r.unwrap();
        max = max.max(e);
        total_schedules += s;
    }
    println!("scenario={} regions={} epochs={} workers={} elapsed_ms={:.1} us_per_epoch={:.2} schedules={}",
        args[1], regions, epochs, workers, max.as_secs_f64() * 1e3, max.as_secs_f64() * 1e6 / epochs as f64,
        if count { total_schedules.to_string() } else { "-".into() });
}

Whose capabilities

The check uses the capabilities held by this worker's instance of the operator. A first version compared against the tracker's source pointstamps, and that was wrong with several workers. Those pointstamps sum the operator's capabilities over all workers as of the last progress exchange, so one worker's capability can sit above everything they show. The frontier then passes it without waking the operator. When the other workers catch up, only capabilities change, not the input frontier, so nothing re-checks, and the operator sleeps holding a capability that holds back everything downstream. This stalled min/max and top-k dataflows in a 16-worker Materialize test, and not with one worker.

Each child with an UpToCapability input now keeps a tally of the capabilities it holds on this worker, summed over its outputs and updated as extract_progress drains its internals. Other children keep no tally. For a region, the tally is the capabilities it reports, which are its own view across workers, as before. If that view lags, the region has a progress exchange pending, and receiving it schedules the region, which then accepts the current frontier.

Caveats

  1. An operator declaring UpToCapability must not rely on a passed capability to stay awake. Every time it waits on must be at or above a held capability that the frontier has not yet passed. If it holds a capability at 5 that the frontier passed earlier, with pending work at 10, a frontier move from 9 to 11 does not wake it. This holds for partially ordered times as well: if some held capability c <= t is not yet passed, some frontier element f <= c exists, and the change that completes t must remove f. Operators that want the old behaviour keep declaring IfCapability.

  2. Regions rely on their capabilities reaching a scope output. The parent judges a region by the capabilities at its outputs. A region's children are all Never or UpToCapability, and path summaries do not decrease times, so a frontier change a nested UpToCapability child cares about removes a scope-input element at or below that capability's image at the scope outputs. The same argument covers lazy progress (ProgressMode::Demand): a withheld update at time t becomes globally visible only after the frontier elements below t are removed. The argument fails for pointstamps that reach no scope output, such as records in flight to a dead-end operator inside the region. Today those are only missed when the region holds no output capability at all (cap_counts == 0). With this change they are also missed when the region holds unrelated output capabilities. The likely effect is a delayed progress broadcast for that pointstamp, until peers' progress, arriving records or the shutdown check schedule the region. This is reasoning only; I have not hit it in a test.

  3. Capabilities ahead of the frontier still wake the region. A stricter test would wake only when a held capability goes from not passed to passed. Measured as an experiment, that brought ahead to 0.36x to 0.45x as well. It is not sound for regions, though: with a summary that advances time between the scope input and a nested child, the child's frontier can pass its capability while the outer frontier has not passed the projected one.

  4. Cost. For each removed frontier element, the check scans the distinct capability times the operator holds on this worker. Each operator with an UpToCapability input also updates its tally as its capabilities change. A tally per output made shape_scaling's region with 100,000 outputs quadratic at shutdown, so the tally is summed over outputs. Holding many distinct capability times, as partially ordered times can, makes the scan longer.

  5. The variant sits between Never and IfCapability so the max over children stays correct. That shifts the discriminants of the logged FrontierInterest from Log each operator's frontier interest with its summary #836, which has not been released yet.

Tests

  • capability_interest.rs: a region holding a passed capability sleeps (101 schedules over 100 rounds on master, under 10 here). A capability ahead of the frontier still wakes the region each round, and so does a child declaring IfCapability.
  • capability_interest_workers.rs: with two workers, worker 0 holds a capability at 5 while worker 1 still holds one at 0, and the frontier moves from 3 to 10. Worker 0 must be woken and release its capability. It fails 3/3 against the first commit and passes 3/3 with the fix.
  • frontier_interest_log.rs now expects a quiet region to report UpToCapability.
  • cargo test --workspace --all-targets and cargo test --doc pass. Clippy reports no new warnings.

Generated with Claude Code

An operator declaring `UpToCapability` is woken by a frontier change only
when the change removes an element less or equal to a capability it holds.
Changes entirely above its capabilities, including all changes once the
frontier has passed every capability it holds, do not schedule it.

A subgraph's child zero now declares `UpToCapability`, so a region whose
children are all `Never` or `UpToCapability` sleeps through frontier moves
that cannot affect the capabilities it holds. A child declaring
`IfCapability` or `Always` still raises the region to that interest.
The wake check compared a removed frontier element against the tracker's
source pointstamps. Those sum the operator's capabilities over all workers
as of the last progress exchange, so a capability held by this worker's
instance can sit above every element they show. The frontier then passes
it without waking the operator, and the other workers' later downgrades
change only capabilities, never the input frontier, so nothing re-checks.

Each child with an `UpToCapability` input now keeps a tally of the
capabilities it holds on this worker, summed over its outputs and updated
as `extract_progress` drains its internals. The check uses that tally.
A region's tally is the capabilities it reports, as before.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant