Skip to content

feat: add protocol-v2 multivariate flag evaluation - #744

Merged
roncohen merged 13 commits into
mainfrom
feature/multivariate-evaluator-v2
Nov 4, 2026
Merged

roncohen merged 13 commits into
mainfrom
feature/multivariate-evaluator-v2

Conversation

@roncohen

@roncohen roncohen commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Make evaluator v2 a clean major release: newEvaluator({ flags, segments }) prepares a complete collection and returns evaluateAll(context) and evaluateFlag(context, flagKey) methods. Remove the v1 value-rule API and legacy serialization helpers.
  • Batch evaluation returns { context, flags } with normalized context once and independent per-flag results/diagnostics. One unresolved flag does not stop the remaining outputs. Single-flag evaluation visits only its selected root and dependencies. The methods map naturally to Node SDK getFlags() / getFlagsForBootstrap() and getFlag() respectively; SDK wrappers, overrides and fallbacks remain outside the evaluator.
  • Preserve flag and segment references instead of requiring publishers to inline their filters. Flag references match explicit variant keys, never the truthiness of variant values. Segments remain reusable predicates and can reference other segments or flags.
  • Resolve references recursively for every evaluation. No dependency-result caching or memoization; repeated references are evaluated again. Retain the existing definition-level membership Sets and percentage-bound preparation.
  • Preserve a valid matching referenced selection despite diagnostics from unsuccessful independent sibling rules. Nonmatches accompanied by diagnostics remain invalid through negation, so an uncertain negative reference cannot grant access. Ordinary authored AND/OR error handling is unchanged. Propagated diagnostics identify their original flag or segment.
  • Reject duplicate keys, missing references, unknown referenced variant keys and cycles during preparation, including mixed flag/segment cycles and unreachable references. Evaluating an unknown flag key also throws.
  • Preserve generic variant values and use a discriminated resolved result, distinguishing selected undefined from failure and requiring success metadata.
  • Keep prepared Sets and percentage bounds private. Precompute lookups/bounds once, use a simple allocation scan, and preserve boolean cohort hashing.
  • Return structured diagnostics instead of console logging. Invalid numeric/date comparisons cannot match through negation.
  • Keep the legacy Node SDK on published evaluator 1.1.2 via an npm alias, retaining main's v1 diagnostic fixes without linking it to the v2 workspace. Node SDK v2 adoption is separate in feat(node-sdk): adopt collection values and Boolean-first reads #790.

Main synchronization

Merged main at abb17bd2f801a8b299ef02eca87b2d9ca774572a through merge commit 486b2099e0d03f5e86566ab31660b0cb95b20ea8. The reviewed collection evaluator source is unchanged. Main's SDK fixes, dependency updates and Vitest 4.1.11 migration are retained; an additional invalid-operator regression covers v2 fail-closed handling. No history was rewritten. The separate SDK branches were not changed and need their own synchronization with this updated base.

API

const evaluator = newEvaluator({
  flags: compiledFlags,
  segments: compiledSegments,
});
const batch = evaluator.evaluateAll(context); // { context, flags }
const result = evaluator.evaluateFlag(context, "my-flag");

Reference filters:

{ type: "flag", key: "model", variantKey: "quality" }
{ type: "segment", key: "enterprise" }

A matching selected variant is authoritative even if earlier rules produced diagnostics. A diagnostic-bearing nonmatch is deliberately more conservative than explicitly matching the selected fallback variant. Segments do not introduce independent-rule boundaries.

Validation

  • 301 evaluator tests and 265 legacy Node SDK tests passed after the main merge. Consumer regressions also passed: 208 Browser, 65 React, 13 Vue, 27 OpenFeature Browser and 28 OpenFeature Node tests. These are the v1 SDKs retained on this evaluator PR, not the separate SDK v2 branches.
  • 14 batch/single-flag API cases cover result-type narrowing, output selection, normalized-context sharing, per-flag failure isolation, parity between operations, unrelated-root exclusion, no memoization in either definition order, returned-state isolation, selected undefined values, empty collections and dangerous object keys.
  • 71 collection/reference cases cover the original independent-grant reproduction; both source-rule orders; missing context; unsupported arrays; invalid comparisons and operators; rollout/distribution behavior; nested flag/segment references; diagnostic provenance and deduplication; negative-reference safety; duplicate/missing/cyclic definitions; shared acyclic dependencies; and repeated evaluation without result caching.
  • Evaluator/test TypeScript checks and builds/declarations passed for evaluator, Node, Browser, React, Vue, both OpenFeature providers and React Native. Immutable installation, scoped formatting/lint and PR-delta whitespace checks passed. Scoped evaluator/Node lint has zero warnings/errors; repository lint has zero errors and one existing generated Next.js triple-slash-reference warning. Existing dependency peer warnings remain. Normal commit hooks passed without changing the verified content.
  • Existing definition-level benchmarks passed with the preceding collection change. No dependency-graph performance claim is made; correctness is the focus of this work.
  • Changesets release-plan validation selects evaluator 2.0.0, Node SDK 1.8.3 and OpenFeature Node 1.2.8; legacy SDKs are not migrated to the v2 protocol.
  • Full workspace build was previously attempted but blocked locally by missing Java for unrelated Management SDK OpenAPI generation; it was not rerun for this merge. No E2E was run in this checkpoint. Upstream-generated Management SDK files were retained unchanged; the PR delta contains no Management SDK changes.

Scope

Still a draft; nothing published or deployed. This layers the collection-based alternative discussed in #780 into v2 rather than adding ruleEvaluation: "independent" metadata to v1. #780 and its companion web PR are untouched.

Publisher/compiler changes and matching server-side consumers are separate in reflagcom/web#3660. SDK adoption of /flags and variant-valued public APIs are separate in #790 and draft #791. Existing v1 consumers must not receive v2 reference nodes.

Read-only inspection of web and its Advanced flags work (#3660) confirms the follow-up needs a distinct reference-preserving v2 lowering, reachable compiled segments, complete collection transport, and snapshot-level evaluator ownership. Boolean v2 rules must be compiled from normalized sources rather than copied from the already-inlined legacy rules. Keep legacy /features and config compilation unchanged, and retain the complete graph while selecting/authorizing output roots.

@roncohen

Copy link
Copy Markdown
Contributor Author

Performance review fixes pushed:

  • Added newFlagEvaluator(flag) for repeated checks. SDKs should prepare on definition refresh and reuse the returned function; evaluateFlag is the one-shot convenience API.
  • ANY_OF and NOT_ANY_OF use prebuilt Sets, including under groups and negations. Prepared filters no longer retain the original candidate arrays. Scalar checks use hash lookups; array contexts do one lookup per examined context element.
  • Percentage validation and cumulative thresholds move out of the hot path. Allocation selection uses binary search.
  • Reuse per-rule diagnostic scratch storage and avoid result-copy/group-callback allocations. Returned diagnostics remain isolated between calls.
  • Fixed the inclusive maximum hash selecting a trailing 0% allocation; it now selects the last nonzero allocation.

Validation: 285 tests passed; typecheck, workspace build, and lint passed. Added repeatable Vitest benchmarks and tests that assert Set lookup usage and no repeated percentage reads.

Local benchmarks: prepared v2 membership checks stayed around 0.7 microseconds/check across 10, 1,000, and 100,000 candidates. At 100,000 candidates, the original PR implementation measured about 77 microseconds/check versus 0.68 microseconds prepared (roughly 113x faster, excluding preparation). Prepared v2 was also about 1.2x faster than prepared legacy in the included membership benchmark. Results are local microbenchmarks, not production throughput guarantees.

@roncohen
roncohen marked this pull request as ready for review November 4, 2026 18:30
Copilot AI balanced review requested due to automatic review settings November 4, 2026 18:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the collection-based evaluator change.

No open issues 🎉

What changed in this PR

Introduces a collection-based protocol-v2 evaluator for multivariate flags, providing the foundation for separate SDK migrations while keeping the existing Node SDK on published evaluator v1.

Changes:

  • Replaces v1 value-rule APIs with prepared batch and single-flag evaluation, generic variant values, and structured diagnostics.
  • Adds recursive flag/segment references, collection validation, and conservative handling of invalid references without dependency-result caching.
  • Adds regression tests, benchmarks, migration documentation, and release isolation for the legacy Node SDK.
File Description
yarn.lock Records the published v1 alias separately from the v2 workspace.
packages/​node-sdk/​src/​types.ts Imports evaluator types through the v1 alias.
packages/​node-sdk/​src/​client.ts Uses the v1 alias for runtime evaluation.
packages/​node-sdk/​package.json Pins the legacy evaluator through an npm alias.
packages/​flag-evaluation/​test/​variants.test.ts Tests variant selection, distributions, types, and preparation.
packages/​flag-evaluation/​test/​variants.bench.ts Benchmarks prepared membership and distribution evaluation.
packages/​flag-evaluation/​test/​release.test.ts Checks v1/v2 dependency isolation.
packages/​flag-evaluation/​test/​references.test.ts Tests recursive references, validation, and diagnostic boundaries.
packages/​flag-evaluation/​test/​index.test.ts Migrates operator and context tests to the v2 API.
packages/​flag-evaluation/​test/​helpers.ts Adds a single-flag fixture helper.
packages/​flag-evaluation/​test/​collection.test.ts Tests batch/single parity, isolation, and uncached dependencies.
packages/​flag-evaluation/​src/​index.ts Implements the collection-based multivariate evaluator.
packages/​flag-evaluation/​README.md Documents v2 usage, reference semantics, and migration.
packages/​flag-evaluation/​package.json Sets the v2 prerelease version and adds prepack compilation.
.changeset/​pin-legacy-node-evaluator.md Declares the Node SDK dependency-isolation patch.
.changeset/​multivariate-evaluation.md Declares the evaluator major release and API changes.

Review effort: Balanced

@roncohen
roncohen added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit d7078e0 Nov 4, 2026
9 checks passed
@roncohen
roncohen deleted the feature/multivariate-evaluator-v2 branch November 4, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants