Skip to content

Retire PolicyFingerprint as StreamingConfig's routing key #785

Description

@zzylol

PolicyFingerprint describes itself as legacy, in its own module doc:

//! Legacy routing wrapper for a deployed stored output.
//!
//! An explicit `PrecomputeMaterialization::stored_output_id` takes precedence.
//! Otherwise the compiler allocates a deterministic default from the existing
//! policy fields (including pane layout and cadence). This identifier is not
//! semantic identity: `SummaryDefinitionId` hashes the versioned semantic
//! definition, and several deployed outputs may share that definition.

Its two original jobs have already been split out: semantic identity belongs to SummaryDefinitionId, and explicit routing belongs to stored_output_id.

The migration is real but cannot finish on its current path

stored_output_id adoption is climbing through the stack:

branch stored_output_id occurrences
main 157
issue-752 (#761) 194
test/physical-execution-e2e (#775) 196
test/issue754-level3 (#759) 196

But PolicyFingerprint occurrences do not fall. Counted across every open PR branch, they sit at 300-302 against main's 301, and several branches are at 302. No open PR deletes crates/asap_types/src/policy_fingerprint.rs.

The reason is a loop in PolicyFingerprint::from_config:

pub fn from_config(cfg: &PrecomputeMaterialization) -> Self {
    if let Some(output) = cfg.stored_output_id {
        return output.fingerprint();   // the explicit id becomes a fingerprint too
    }
    ...
}

Adopting an explicit id does not remove a fingerprint; it changes how that fingerprint is derived. The computation is being retired. The type is not, because it is still the routing key.

What actually blocks retirement

StreamingConfig keys its materializations by the fingerprint as a bare u64:

HashMap<u64, PrecomputeMaterialization>

StreamingConfig is what the control plane POSTs to the data plane as YAML (control_plane/src/backend_client.rs). So the key is part of a published configuration format, and changing it is a format change rather than an internal refactor.

Proposed work

  1. Key StreamingConfig's materializations by StoredOutputId instead of a bare u64.
  2. Require stored_output_id, so from_config's hashing fallback has no callers, then delete the fallback and the module.
  3. Bump the published schema version, which now has a single owner and a test enforcing that every producer agrees (WORKLOAD_SNAPSHOT_VERSION, added in fix(control-plane): give the snapshot schema version one owner #782).

Step 3 is the reason this is worth doing now rather than later: #782 made a schema bump one edit plus whatever its test reports, instead of a literal that some producers follow and others quietly do not.

Not in scope

SummaryDefinitionId stays. It is semantic identity, a different thing from routing, and several deployed outputs may legitimately share one definition.

Related

Separately worth checking while in this area: plan_id is computed with std::collections::hash_map::DefaultHasher (control_plane/src/physical/compiler.rs), whose algorithm the standard library does not guarantee across Rust releases, yet plan_id is compared across processes and contract tests reject a changed installed generation. A toolchain upgrade would change it.

Activity

  1. zzylol commented on Sep 28, 2026

    @zzylol
    ContributorAuthor

    Correcting the blocker I described above: I had not checked how the key is serialized, and it is not.

    #[serde(skip)]
    pub materializations_by_policy_fingerprint: HashMap<u64, PrecomputeMaterialization>,

    StreamingConfig's materialization map carries #[serde(skip)]. It is a process-local index, rebuilt by from_precompute_plan when the config is decoded. So re-keying it is an internal refactor, not a change to the published configuration format, and step 3 above (bump the schema version) is not required.

    Two further facts make the first two steps cheaper than I said:

    StoredOutputId and PolicyFingerprint are the same u64 wearing two type names, with free conversions in both directions:

    pub struct StoredOutputId(pub u64);
    impl StoredOutputId {
        pub fn fingerprint(self) -> PolicyFingerprint { PolicyFingerprint(self.0) }
    }
    impl From<PolicyFingerprint> for StoredOutputId { ... }
    impl From<StoredOutputId> for PolicyFingerprint { ... }

    Both are #[serde(transparent)], so wherever either is serialized the bytes are identical.

    And PolicyFingerprint never appears as a serde field on any struct. The only place its value reaches a published document is inside a formatted string:

    pub(crate) fn state_schema_id(fingerprint: PolicyFingerprint) -> String {
        format!("{}:summary-state:v1:{}", BACKEND_COMPAT, fingerprint.0)
    }

    That schema_id does travel in PrecomputePlan.schemas and is validated on install, but since the two types hold the same number, passing a StoredOutputId produces the identical string.

    So the work is: re-key the map and give it a name that says what it holds, take StoredOutputId in state_schema_id, then make stored_output_id required so the hashing fallback in from_config has no callers and the module can go. No format change, no version bump, and nothing about SummaryDefinitionId moves.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions