router: add weighted priority cluster specifier plugin (POC)

[kanurag94]

Commit Message: router: add weighted priority cluster specifier plugin (POC)

Additional Description:

This is a proof of concept opened for design discussion, and it deliberately carries no tests yet. I would like agreement on the shape before fixing the surface. It follows @wbpcode's suggestion on #46624 that this kind of selection belongs in a cluster specifier rather than in a cluster implementation, since a specifier is much lighter.

The complete API

The whole configuration surface is three fields:

priority_groups:          # ordered, most preferred first; at least one group
- clusters:               # at least one cluster per group
  - name: primary_a       # cluster defined elsewhere in the configuration
    weight: 70            # relative share within this group; must be > 0

Wired into a route:

routes:
- match: {prefix: "/"}
  route:
    inline_cluster_specifier_plugin:
      extension:
        name: weighted_priority
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.router.cluster_specifiers.weighted_priority.v3.WeightedPriorityClusterSpecifier
          priority_groups:
          - clusters:
            - name: primary_a
              weight: 70
            - name: primary_b
              weight: 30
          - clusters:
            - name: fallback
              weight: 1

How it picks

Walk the groups in order. Take the first group that has at least one cluster which exists and has healthy hosts. Make a weighted choice among only those healthy clusters.

flowchart LR
    A["Request"] --> B{"Group has a healthy cluster?"}
    B -->|No| C["Next group"]
    C --> B
    B -->|Yes| D["Weighted choice among its healthy clusters"]
    D --> E["Cluster"]

Loading

With the configuration above:

primary_a	primary_b	fallback	Traffic goes to
healthy	healthy	healthy	70% primary_a, 30% primary_b
healthy	unhealthy	healthy	100% primary_a
unhealthy	healthy	healthy	100% primary_b
unhealthy	unhealthy	healthy	100% fallback
unhealthy	unhealthy	unhealthy	fallback, and the request fails there

Row two is the point of the extension. Weights are renormalized over whichever clusters of the group are healthy, so primary_b going down hands its 30% to primary_a rather than failing it.

Why not the existing mechanisms

weighted_clusters keeps sending a cluster its configured share even when every host in it is unhealthy, so row two would fail 30% of requests. It also has no notion of tiers, so there is no way to express "try these two, then that one".

The aggregate cluster fails over between priorities of a single linearized host set. This chooses between clusters, so each keeps its own health checking, outlier detection, load balancing policy, and circuit breakers, and the weighting is expressed between clusters rather than between endpoints.

Scope

The cluster is chosen once per request, when the route is resolved, so retries stay within the chosen cluster. That is a deliberate boundary. Moving between clusters across retry attempts is what #46624 is for, and the two compose: this plugin chooses which cluster a request starts in, and a retry-aware cluster type handles progression from there.

Open questions

1. Is the two-level structure right, or would a flat list of clusters each carrying an explicit priority number be preferred?
2. Should health mean "any healthy host", as implemented, or should degraded hosts count, following HostSet::degradedHosts()?
3. On the last row of the table, this makes the weighted choice over the last group while ignoring health, so the request fails against a real cluster and is attributed to it in stats. Would returning no route be preferred?
4. Per-tenant weights are the obvious follow-up, and reading weights from dynamic metadata would cover it. I left it out to keep the structural question separate.

Risk Level: Low. This is a new opt-in extension, and no existing code path changes.

Testing: none yet, by design. Unit tests covering group fallthrough, weight renormalization, and the all-unhealthy path will follow once the shape is agreed. It compiles: bazel build //source/extensions/router/cluster_specifiers/weighted_priority/...

Docs Changes: protodoc on the new message only. An architecture overview page will follow with the tests.

Release Notes: changelogs/current/new_features/router__weighted-priority-cluster-specifier.rst

Platform Specific Features: N/A

API Considerations: a new extension proto in a new package, marked alpha in the extension metadata. Nothing existing changes.

Generative AI disclosure: this change was developed with assistance from Claude Code. The author has reviewed and understands the code.

[repokitteh-read-only]

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #46636 was opened by kanurag94.

see: more, trace.