Declarative decoder parameter schemas: pluggable realtime decoder configuration#679
Draft
bmhowe23 wants to merge 10 commits into
Draft
Declarative decoder parameter schemas: pluggable realtime decoder configuration#679bmhowe23 wants to merge 10 commits into
bmhowe23 wants to merge 10 commits into
Conversation
…-backed storage Replaces the per-decoder typed-struct YAML traits and type-dispatch ladders in realtime config parsing with a declarative parameter-schema registry (cudaq/qec/decoder_config_schema.h). Each decoder registers a decoder_schema describing its custom args; a generic llvm::yaml CustomMappingTraits walker converts YAML <-> cudaqx::heterogeneous_map from the schema, including nested (subschema) and discriminated-union sections (schema selected by a sibling key, covering trt global_decoder_params and sliding_window inner_decoder_params with one mechanism). decoder_config::decoder_custom_args changes from a closed std::variant to a heterogeneous_map-backed wrapper; typed config structs remain as programmatic conveniences (assignment still works, reads via as<T>()). Out-of-tree decoder plugins can now make themselves realtime-ready by registering a schema from their own shared library instead of editing config.cpp. Consumer updates (tests, examples, python bindings) follow in this branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…ma tests Call sites move from std::variant access (std::get) to the map-backed wrapper (assignment of typed configs still works; reads via as<T>()). The example decoder plugin now registers a parameter schema from its own library, and new DecoderSchemaTest cases cover third-party registration, unknown-key and missing-required rejection, round-trip stability, and the plugin path. Python: decoder_config.decoder_custom_args becomes a property returning typed views for built-in decoder types and a dict for others; setter accepts typed configs or dicts. Adds config.decoder_param_schema()/registered_decoder_schemas() introspection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
… registration Emission keeps the historical behavior for unknown decoder types (args dropped, now with a warning) so configure_decoders() still reports failures via status codes instead of throwing from to_yaml_str(). Input remains strict. Adds a docs subsection showing how a custom decoder plugin registers its parameter schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
pymatching, chromobius, and trt_decoder schemas move from the central decoder_config_schema.cpp into their plugin .so files; single/multi_error_lut and sliding_window schemas move next to their decoder implementations in cudaq-qec-decoders. The central file now hosts only the nv-qldpc-decoder schema (plus its srelay_bp subschema) until the proprietary out-of-tree plugin registers it itself. Consequence: builds without a given plugin can no longer parse or emit that decoder's YAML section (arguably correct -- the decoder cannot run there either). trt-dependent YAML tests now skip when the trt schema is absent (GTEST_SKIP / pytest.mark.skipif via the new decoder_param_schema introspection), and the third-party schema test gains a synthetic discriminated + materialize_empty section so walker coverage no longer depends on the trt plugin being built. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
sliding_window_config's three typed inner-params members collapse to a plain heterogeneous_map inner_decoder_params, and trt_decoder_config's global_decoder_config variant becomes std::optional<heterogeneous_map> -- both validated against the schema registry instead of hardcoded type lists. Deletes the global_decoder_config type, its four helper functions, and the variant special-casing in the python type casters; nested params surface as plain dicts in Python. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…ruth (option 2) Deletes srelay_bp_config, nv_qldpc_decoder_config, multi/single_error_lut_config, pymatching_config, chromobius_config, trt_decoder_config, and sliding_window_config along with their to/from_heterogeneous_map conversions and the INSERT_ARG/GET_ARG machinery. decoder_custom_args is a plain parameter map in C++ and a plain dict in Python -- the same representation for built-in and third-party decoders -- with keys governed solely by each decoder's registered schema. The Python decoder_custom_args getter no longer special-cases built-in types (the last per-decoder dispatch ladder), and the typed Python classes and their re-exports are removed. Docs now teach the dict/map + decoder_param_schema() introspection flow. Each decoder parameter now exists in exactly one place: the schema its decoder registers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Programmatically built configurations (heterogeneous_map / Python dict) never pass through the YAML parser, so schema checks (unknown keys, missing required keys, per-schema validate hooks) were only applied when parsing. Add validate_custom_args(schema_name, args) to the schema registry, expose it as decoder_config::validate_custom_args() and multi_decoder_config::validate_custom_args() (C++ and Python), and call it in the real_time_complete examples. Also register the first in-tree validate hook: sliding_window rejects step_size outside [1, window_size] and empty error_rate_vec at configuration time. Unknown-key and required-key checks remain framework provided; hooks are only for cross-field constraints the declarative specs cannot express. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
decoder_config_json_schema() generates a JSON Schema (draft 2020-12) document for multi_decoder_config YAML files, so third-party tooling (check-jsonschema, python jsonschema, editor YAML language servers) can validate user configurations offline. Per-decoder decoder_custom_args sections are translated from the schemas registered at call time -- including out-of-tree plugins, whose schemas appear automatically once their library is loaded -- with required keys, additionalProperties: false, and discriminated-section dispatch mirroring the runtime parser. The fixed decoder_config envelope (id/type/block_size/H_sparse/...) is emitted alongside, and types with no registered schema accept no custom args, matching parse behavior. Exposed in Python as qec.decoder_config_json_schema(). Schema validate hooks are arbitrary code and are documented as not representable; the runtime parser remains authoritative. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
- Apply schema-declared defaults (materialize_empty sections) to programmatically built configs: decoder_custom_args_to_heterogeneous_map now materializes via the registry, so a dict/map-built trt config with global_decoder but no global_decoder_params attaches the global decoder exactly like the YAML path. New materialize_default_args() in the registry; finalize_parsed_args delegates to it plus the canonical validate walk, eliminating the duplicated recursive schema walker. - Convert Python dict values to the schema's canonical types in the decoder_custom_args setter: ints are accepted for f64 params and negative ints for int32 params (the generic conversion stored every int as size_t, rejecting negatives at assignment and breaking f64 reads at emission); type mismatches raise an error naming the parameter. - Warn when YAML emission omits a map key absent from the registered schema (typo'd programmatic args no longer vanish silently). - Make custom-args equality sign-aware: size_t(2^64-1) no longer compares equal to int(-1). - decoder_config::from_yaml_str now rejects malformed YAML like the multi_decoder_config variant (pre-existing gap). - Cleanups: drop dead includes and the unused set_decoder_custom_args_from_heterogeneous_map, use heterogeneous_map::size(), document that the Python decoder_custom_args getter returns a copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Declarative decoder parameter schemas: true third-party support for realtime decoding
TL;DR
config.cppwas a closed shop: every decoder's YAML parameters were hard-coded intotyped structs, a
std::variant, and per-type LLVM YAML traits inside the core library.A third-party decoder could implement the
decoderinterface from its own plugin, butcould never be configured through the realtime YAML without patching and rebuilding
CUDA-Q QEC.
This PR replaces all of that with one declarative schema per decoder, registered by the
same shared library that registers the decoder itself. The framework does the rest —
parsing, emission, validation, defaulting, Python conversion, and standalone JSON Schema
export are all generic. Net size: +2.5k/−2.2k lines, and the additions are mostly
tests and docs; the core config machinery is smaller than what it replaces.
Headline benefits
🔌 True third-party realtime decoders. A plugin makes itself fully YAML-configurable
with one registration in its own
.so— zero framework changes, zero rebuilds:Proven end-to-end: a demo decoder was built against a sandbox
cmake --installtreeusing only installed headers, dropped into
decoder-plugins/, and ran realtime decodingwith its own YAML parameters — no cudaqx source edits.
🧰 Maintainability: each parameter exists in exactly one place. Previously, adding
one decoder parameter meant touching ~6 places (struct field, variant traits, YAML
mapping,
to_heterogeneous_map, Python binding, docs). Now it is one line in thedecoder's own schema. The core gets rid of ~900 lines of per-decoder ladders, and the
YAML walker, validator, and JSON exporter never change when decoders are added.
🐍 Usability: plain dicts and maps, with introspection. No more per-decoder config
classes to learn:
Values are converted to the schema's declared types (Python ints are fine for float
parameters, negative ints for int32 parameters), and mistakes fail with messages like
Parameter 'bp_seed' of 'nv-qldpc-decoder' expects a 32-bit int value.✅ Validability, three ways.
Parse time: unknown keys and missing required keys are rejected by the framework —
decoders never write those checks.
Programmatic:
config.validate_custom_args()applies the same checks to dict/map-builtconfigs before use; decoders can attach cross-field hooks (e.g. sliding_window rejects
step_size > window_sizeat config time instead of deep in the constructor).Offline:
qec.decoder_config_json_schema()exports a standard JSON Schema(draft 2020-12) built from whatever plugins are loaded, so CI and editors can lint
config files without loading the library at all:
🔁 Behavior preserved. YAML round-trips byte-match the old output; trt's default
global_decoder_paramsmaterialization, sliding_window's inner-decoder handling, andconfigure_decodersstatus-code semantics are unchanged and covered by tests (31 C++ /28 Python in the touched suites, all green).
The main downside: this is a breaking change
The typed config classes are gone —
nv_qldpc_decoder_config,trt_decoder_config,sliding_window_config,pymatching_config,chromobius_config, etc. (C++ and Python),with no deprecation shims. Migration is mechanical:
One structural consequence to be aware of: a decoder's YAML section is only parseable when
its plugin (and therefore its schema) is loaded. Hosts that parse-and-forward configs for
decoders they don't have locally need that decoder's plugin present. The nv-qldpc-decoder
schema is temporarily hosted in-tree until the proprietary plugin registers it itself.
Follow-ups (out of scope here)
sparse_binary_matrixparameter kind with scipy pass-through (parked on a separatebranch for its own PR).
global_decoder) to decoder-roleschemas rather than any registered name.
🤖 Generated with Claude Code