DRAFT: QEC decoding server: transport-agnostic bridge providers, one ring per decoder, mixed host + device_graph dispatch#682
Draft
bmhowe23 wants to merge 26 commits into
Draft
Conversation
…ng-server test Extend hsb_fpga_decoding_server_test.sh to drive the nv-qldpc Relay BP decoder through the decoding server's gpu_roce transport -- the self-relaunching device-graph scheduler (GpuRoceTransceiver): enqueue_syndromes / get_corrections / reset_decoder execute as DEVICE_CALLs on the GPU and the captured RelayBP decode graph fires device-side (fire-and-forget + tail-self-relaunch) when a window completes. This replaces hololink_qldpc_graph_decoder_bridge for the decoding-server flow; the old bridge test is untouched. hsb_fpga_decoding_server_test.sh: - --transport cpu_roce|gpu_roce, defaulted from the decoder profile (pymatching -> cpu_roce HOST_CALL path, nv-qldpc-decoder -> gpu_roce), plus --gpu, --proprietary-archive, --nv-qldpc-plugin. - Data generation (surface_code-4-yaml, fresh each run) adds --use-relay-bp for the nv-qldpc profile and injects `transport: gpu_roce` into the generated YAML: DecoderServer selects its transceiver from the per-decoder transport key, the generator omits non-default optionals, and the default (cpu_roce) resolves to the not-yet-implemented CpuRoceTransceiver stub. - gpu_roce server launch: Hololink parameters go via HOLOLINK_* env (the server's gpu_roce mode ignores the cpu_roce CLI flags), remote QP converted to decimal ($((qp)) -- GpuRoceConfig::from_env parses base-10 only), CUDA_MODULE_LOADING=EAGER as with the old bridge launcher, and readiness keyed on "QEC_DECODING_SERVER_READY gpu_roce" (the transceiver prints the QP/RKey/Buffer handshake before that line, without the bridge banner). - Build phase: builds gpu_roce_transceiver (HSB) and cudaq-realtime-bridge-hololink (cuda-quantum), wires CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE / GPU_ROCE_TRANSCEIVER_LIB / CUDAQ_REALTIME_BRIDGE_HOLOLINK_LIBRARY into the cudaqx configure, clears the CMake cache (find_library NOTFOUND staleness), and symlinks the nv-qldpc plugin into build/lib/decoder-plugins. All gated on the proprietary archive's presence so pymatching-only rigs build unchanged. The plugin symlink is also created opportunistically on --build-less runs. decoding-server-cqr/CMakeLists.txt (bug fix, separable): - CUDAQ_GPU_ROCE_AVAILABLE was set as CACHE INTERNAL on success, but the plain `set(... FALSE)` at the top of the file shadows the cache under CMP0126 (NEW), so the compile-definition gate for cudaq-qec-decoding-server never fired: the library was silently built WITHOUT gpu_roce (make_transport threw "CUDAQ_GPU_ROCE_AVAILABLE is not set" at runtime) while the sibling decoding_server target -- which reads the cache -- reported "gpu_roce transport enabled". Set the normal variable TRUE alongside the cache entry. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
With CUDAQ_GPU_ROCE_AVAILABLE now actually reaching the library target (the CMP0126 normal-vs-cache shadow fix), cudaq-qec-decoding-server links libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so -- both of which reference the CUDA Driver API (cu*). Consumers of the static lib (test_decoding_server_core, test_decoders_yaml, decoding_server) inherit those dependencies, and on driverless build machines (CI containers: CUDA toolkit present, no GPU driver, no libcuda.so.1) their links fail with dozens of `undefined reference to cuStreamCreate` etc. -- the failure seen on all four QEC Build-and-test CI jobs. Link CUDA::cuda_driver alongside CUDA::cudart in the gpu_roce block: the CMake target resolves to the real driver where one is installed and to the toolkit's stubs (lib/stubs/libcuda.so) in driverless environments. This matches the existing precedent in unittests/utils (playback and the old bridge already link CUDA::cuda_driver). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
With CUDAQ_GPU_ROCE_AVAILABLE actually reaching the library target (the CMP0126 fix), every consumer of cudaq-qec-decoding-server (test_decoding_server_core, test_decoders_yaml, the CQR plugin) inherited runtime dependencies on libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so, which require libcuda.so.1 at LOAD time. On driverless CI machines (CUDA toolkit + HSB + DOCA present, no GPU driver) all four QEC Build-and-test jobs failed: gtest_discover_tests executes the test binaries at build time and the loader cannot satisfy libcuda.so.1 -- link-time driver stubs satisfy ld but not the runtime loader. Restructure so the core library carries no GPU RoCE dependencies while keeping full CI build coverage of the HSB path: - New optional component cudaq-qec-decoding-server-gpuroce holds GpuRoceTransceiver.cpp plus GpuRoceFactory.cpp, the strong definition of the factory (cudaqx_qec_make_gpu_roce_transceiver) that DecodingServer.cpp now references weakly. All DOCA / hololink / CUDA-driver link deps move to this component. - make_transport(gpu_roce) calls the factory when it is linked in and throws a clear "GPU RoCE support is not linked into this binary" error otherwise; the ifdef and the direct GpuRoceTransceiver construction are gone from the core. - The scheduler hookup no longer dynamic_casts to the concrete type: ITransceiver grows a launch_device_scheduler(void*) virtual (default: no device scheduler), overridden by GpuRoceTransceiver to forward to launch_scheduler(). The config-driven constructor keys the hookup on the transport enum instead. - decoding_server links the component WHOLE_ARCHIVE in its proprietary-archive-gated gpu_roce block -- a weak reference does not pull archive members. - A link canary (cudaq-qec-decoding-server-gpuroce-linkcheck; never executed, not installed, not a test) preserves CI link validation of the component: a static archive alone never resolves symbols, and the one real consumer (the daemon's gpu_roce block) is gated on the proprietary cudevice archive, which CI does not provision. The canary forces full link resolution against the hololink / DOCA libraries with the driver stubs -- the same check that used to happen incidentally via the test binaries. - decoding_server's QEC_HAVE_GPU_ROCE_TRANSPORT gate is split from the proprietary-archive gate: the gpu_roce CLI branch references only the (now dependency-free) core DecodingServer API, so it compiles and links whenever HSB/DOCA are present -- CI now builds that branch too. The component + cudevice whole-archive link stays behind the archive gate, so the CI daemon remains loadable on driverless runners. Branch [2a] wraps server startup in try/catch, turning what used to be std::terminate on any gpu_roce bring-up failure into an "ERROR: gpu_roce startup failed: ..." message with exit 1. Net CI coverage: GpuRoceTransceiver.cpp compiles against the HSB/DOCA headers (component builds in `all`), its link resolves (canary), and the daemon's gpu_roce CLI branch compiles and links (split gate). No binary CI executes carries a libcuda.so.1 dependency. On dev rigs the daemon keeps full gpu_roce. As a bonus, the CQR plugin .so is loadable on driverless machines again (it too had inherited the deps). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The server now contains no transport-specific code. --transport=<name> resolves to libcudaq-realtime-bridge-<name>.so next to the CUDA-Q realtime libraries (udp and cpu_roce ship there, incl. the TCP rendezvous and hsb_fpga QP exchange that previously lived inline here); --transport=/path/to/lib.so loads a partner's out-of-tree provider verbatim with no changes to this server. Unrecognized CLI arguments are forwarded to the provider's create(), so the existing flags (--port/--num-slots/--slot-size/--device/--local-ip/--qp_config/ --peer-ip/--remote-qp/--frame-size) keep working. Dispatch moves from a hand-rolled thread around cudaq_host_ring_dispatch_loop to the libcudaq-realtime dispatcher object (create/set_ringbuffer/set_function_table/set_control/start), the supported surface after cuda-quantum PR 4869; ring geometry comes from the provider's geometry query and readiness from its endpoint query (READY line format unchanged: port= hoisted first for the sscanf in test_decoding_server.cpp). gpu_roce is untouched: it predates the provider interface and takes a different dispatch shape (device-side scheduler; PR 670 is active in that code). CMake: the server no longer links the udp/cpu_roce transports, the host-dispatch archive, or ibverbs -- providers are runtime artifacts. Requires a CUDA-Q realtime install with bridge interface v2 (endpoint info + ring geometry) and the provider libraries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The Hololink transceiver bring-up moves behind the CUDA-Q realtime transport-provider interface: the constructor loads the provider (built-in libcudaq-realtime-bridge-hololink.so, or the library named by CUDAQ_REALTIME_BRIDGE_LIB) and adopts its RING_BUFFER context, geometry, and endpoint identity (v2 queries, with config-derived and banner fallbacks for a v1 provider). launch_scheduler()'s device-graph scheduler wiring is unchanged; the provider's launch() now owns the Hololink monitor thread, and disconnect()/destroy() replace direct hololink_close/destroy calls. Consequence: cudaq-qec-decoding-server no longer links Hololink / DOCA / HSB / ibverbs at all -- CUDAQ_GPU_ROCE_AVAILABLE now gates only on the CUDA-Q realtime headers, libcudaq-realtime.so, and the CUDA toolkit, so the gpu_roce path compiles on machines with none of the RDMA stack installed and fails loudly at runtime when the provider .so is absent. (Also applies PR 670's CMP0126 cache-shadowing fix so the compile definition actually fires.) Runtime validation on HSB rigs still required: this box has no DOCA / HSB, so the provider .so cannot be built or exercised here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
# Conflicts: # libs/qec/lib/realtime/decoding-server-cqr/CMakeLists.txt # libs/qec/tools/decoding-server/CMakeLists.txt Signed-off-by: Ben Howe <bhowe@nvidia.com>
…roce Follow-up to the bridge-provider refactor + PR 670 merge, closing the naming leaks that made the device path look like (and require the vocabulary of) a specific wire: - GpuRoceTransceiver/GpuRoceConfig -> DeviceGraphTransceiver/-Config (files, class, weak factory symbol cudaqx_qec_make_device_graph_transceiver, component cudaq-qec-decoding-server-device-graph, defines CUDAQ_QEC_DEVICE_GRAPH_AVAILABLE / QEC_HAVE_DEVICE_GRAPH_DISPATCH). It never was a transport: it is the GPU device-graph dispatch engine riding whatever bridge provider is loaded. - Per-decoder YAML key is now `dispatch: host|device_graph` (DecoderDispatch), naming the dispatch SHAPE instead of a wire. The legacy `transport: cpu_roce|gpu_roce` key and spellings remain input-only aliases. Fixes a subtle parser bug found while testing: the alias mapping must use two-arg mapOptional, or an absent legacy key resets the parsed dispatch value to the default. - Config env vars are QEC_DEVICE_GRAPH_* with the HOLOLINK_* spellings as fallback (values are forwarded to whatever provider is loaded, which need not be Hololink; existing orchestration scripts keep working). - decoding_server routes to the device-graph path from the YAML's declared dispatch shape, not from --transport=gpu_roce (kept as a legacy alias). The CLI/YAML mismatch that used to silently select a stub transceiver is now a loud, specific error, and a non-legacy --transport value selects the provider for the device path the same way it does for the host path. Validated: udp two-process suite 5/5, core 5/5; dispatch: device_graph, legacy transport: gpu_roce + HOLOLINK_* env, and the mismatch error all exercised end-to-end (device path fails at provider load on this box, as expected without the hololink provider .so). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…split Everything here is new in this development cycle with no deployed consumers, so the transition aliases go: - YAML: the per-decoder legacy key `transport: cpu_roce|gpu_roce` and the wire-named enum spellings are gone; `dispatch: host|device_graph` is the only form. An old config now fails at parse time with "unknown key 'transport'" (and the server reports YAML parse errors cleanly instead of terminating on the exception). - CLI: `--transport=gpu_roce` is gone; --transport only ever names the wire/provider, and the device-graph path is selected solely by the config's dispatch key (default provider: built-in hololink). READY sentinel for that path is now `QEC_DECODING_SERVER_READY device_graph`. - Env: QEC_DEVICE_GRAPH_* only; the HOLOLINK_* fallback spellings are removed. - DeviceGraphTransceiver now REQUIRES a bridge-interface-v2 provider (ring geometry + endpoint identity queries); the v1-provider fallback paths are gone. - CpuRoceTransceiver (a stub whose constructor always threw) is deleted; DecoderDispatch::host in the standalone DecodingServer path throws a direct, accurate error instead. - hsb_fpga_decoding_server_test.sh updated to the new spellings (dispatch: device_graph injection, QEC_DEVICE_GRAPH_* env, no --transport for the device path, new READY sentinel). Validated: udp two-process suite 5/5, core 5/5; device path reached via dispatch: device_graph with QEC_DEVICE_GRAPH_* env; removed spellings all fail loudly (unknown YAML key; HOLOLINK_* env ignored with a clear missing-variable error). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…n plan) Records the bridge-provider refactor end to end so it can be reproduced from the document alone: problem statement, after-architecture with the device-link vs dlopen design rules, the v2 bridge interface contract and the three providers, the server CLI/YAML/env contracts, the dispatch-vs- wire split, the ordered reproduction plan across both repos, validation gates (including the known pre-existing failure), and recorded follow-ups (rig validation, upstreaming, per-decoder rings, fan-in, vp_id). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…r_id The QEC device wrappers now route every decoding RPC through cudaq::device_call's device-id overload with device_id == decoder_id. The CUDA-Q device_call runtime keys sessions, rings, and dispatchers by device id, so each decoder gets its own ring buffer and dispatcher: no shared ring, no head-of-line blocking between decoders, per-decoder backpressure. The payload keeps decoder_id as a cross-check (and for future multi-source fan-in). The CQR service plugin needs no change for the host path: per-device sessions share the same HOST_CALL handlers and the registry still routes by payload decoder_id. New app example surface_code-5-per-decoder-rings (+ test script, registered in ctest): two pymatching decoders decoded over two rings by one kernel, verified by (1) per-decoder corrections + dispatch count, (2) two per-device session-init lines in the runtime log, and (3) the per-device channel override (CUDAQ_DEVICE_CALL_CHANNEL= host_dispatch,1=device_dispatch) steering device 1 to the GPU channel while device 0 stays on its host ring. Pairs with cuda-quantum commit '[device-call] Per-device sessions on demand + per-device channel spec' (lazy per-device init; channel spec syntax; external channels keep one shared endpoint). Design doc per_decoder_rings_design.md records the topology, the CPU-ring + GPU-ring channel matrix, the recipe for the plugin's Gpu-mode session (the remaining piece for simultaneous CPU+GPU rings), and follow-ups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The server now opens ONE provider instance per decoder in the YAML -- own endpoint, own ring buffer, own dispatcher -- completing the two-process form of the one-ring-per-decoder topology (the product path; the earlier in-process demo only proved the runtime mechanism). Wire contract additions (single-decoder output unchanged apart from a ring0 token): - READY: 'QEC_DECODING_SERVER_READY port=<P0> transport=udp ring0=<P0> ring1=<P1> ...' -- leading port stays ring 0's for existing single-endpoint consumers, which keep working (all traffic lands on ring 0, payload-demuxed). - Shutdown: one 'QEC_DECODING_SERVER_RING decoder=<id> dispatched=<n>' line per ring, sampled AFTER dispatcher_stop (the loop flushes its stats counter on exit). New test DecodingServerTwoProcess.TwoProcessPerDecoderRings: two decoders, two udp endpoints, one caller wiring each decoder's device_call session to its own ring via device-scoped channel args (udp-port=<P0> udp-port.1=<P1>); asserts correct corrections AND >=3 dispatches on EACH ring -- proving per-ring traffic, not one shared wire. Full suite 6/6. Pairs with cuda-quantum commits 'Register bridge handles created through the cached-provider path' (second bridge instance on one provider was unusable) and 'Per-device external channel endpoints'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
At run time the server is N independent 1-ring consumers constructed by
one setup loop: every decoder gets one bridge provider instance and one
ring (same CQR RPC wire format), and the decoder's dispatch: shape
selects the ring's consumer -- a host dispatcher-object thread, or the
CUDAQ device-graph scheduler. Host decoders and a device_graph decoder
now run simultaneously in one server; an all-device_graph config still
takes the standalone DecodingServer (HSB) path.
- DeviceGraphRingConsumer: the device-graph scheduler as a ring
CONSUMER (3 proprietary DEVICE_CALL entries + the decoder's captured
decode graph over a ring it does not own), extracted from
DeviceGraphTransceiver (which now delegates to it). Exposed through
a weak C ABI so server builds without the proprietary component link
and fail loudly at runtime instead.
- SessionRegistry accepts mixed dispatch shapes; the single-transceiver
DecodingServer paths still require uniformity (required_dispatch()
throws on mixed).
- CQR plugin exports cudaqx_qec_decoding_server_graph_resources(id) so
the server can wire the scheduler to a decoder hosted behind the
plugin (graphs are captured at session creation, before READY).
- Decoder YAML gains an optional per-decoder transport override
('transport: "udp --pinned-rings"', or 'hololink' for the builtin
provider slot). One EXTERNAL provider library per process (loader
limitation) is diagnosed with a clear error.
Validated locally: mixed config brings up ring0 (host) + ring1 (pinned
udp), READY publishes both rings, the non-graph decoder fails loudly at
the graph-capture requirement with clean teardown; full two-process
suite 6/6, per-decoder-rings example green. Positive decode on the
device ring needs a graph-capable decoder built against this tree's
ABI (see design doc follow-ups).
Pairs with cuda-quantum commits 'UDP transport external-rings API +
provider --pinned-rings' and 'Do not re-export transport-archive
symbols from the runtime' (ODR fix this work flushed out).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Replaces the per-decoder transport override added with the mixed-
dispatch server: the wire is deployment configuration and lives OUTSIDE
the decoders list. Transports differ between rings only by dispatch
shape (a device_graph ring must be GPU-pollable), so the top-level YAML
transport: section carries provider/args plus one shape-keyed
device_graph: override -- decoder entries carry no transport
information and stay portable across environments.
transport:
provider: udp
device_graph:
args: [--pinned-rings]
Per-ring precedence: shape override > section provider/args >
--transport CLI default; an explicit --transport still overrides the
section's provider for one-off experiments.
Validated: mixed config brings up both rings from the YAML alone (no
transport CLI); full two-process suite 6/6; per-decoder-rings example
and device_call tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
… + WSL2 wall The sc4-cqr external channel guard wires per-decoder device_call sessions to their own rings via QEC_DECODING_SERVER_PORT_<id> -> udp-port.<id>= scoped args. Design doc records the positive-validation campaign: nv-qldpc rebuilt against this ABI (cuda-qx scratch merge), two-process per-decoder-ring CONTROL run decodes correctly with two host nv-qldpc rings; the mixed run's GPU scheduler serves DEVICE_CALLs over pinned-udp rings end to end (raw RPC datagram probes: reset + full-window enqueue round-trip status=0); the remaining decode-graph firing step is blocked by a platform limitation isolated with a minimal CUDA probe -- device-side cudaStreamGraphFireAndForget launches silently never execute on this WSL2 stack -- so the final step runs on the rig. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…aunch The wedge is not a silent no-op: launching any kernel referencing device-side cudaGraphLaunch fails with cudaErrorNotSupported, visible only via cudaGetLastError() immediately after the launch (sync reports success). Documents the 10-second gating check for rig/CI machines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…s exonerated Rig data showed the original probe failing on target hardware too, which exposed the probe's API-contract violation: fire-and-forget device graph launch is only legal from a kernel inside a device-launched graph; a plain trigger kernel correctly gets cudaErrorNotSupported everywhere. The corrected two-graph probe (trigger inside a device-launched parent) PASSES on WSL2, native SASS and forced-JIT alike, and artifact arch coverage is ruled out. The wedge is a real pipeline issue, locally reproducible; next suspects recorded (triggered-graph instantiation flags/upload, tail relaunch, decode graph execution), plus the missing device-side error surfacing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Three-way split with a sharpened boundary: - transport_provider_design.md now owns ALL deployment contracts in one place: alongside the provider ABI, section 4.2 gains the top-level YAML transport section (shape-keyed override + precedence), the multi-ring READY tokens and per-ring shutdown lines, the N-ring server flow, and the caller-side per-ring endpoint contract (key.<id>= scoped args, channel spec). Stale follow-ups refreshed. - per_decoder_rings_design.md slims to pure topology (rings, consumers, dispatch shapes, mixed server), pointing at the provider doc for contracts; the validation campaign narrative is replaced by a short status section. - per_decoder_rings_validation_notes.md (new) is the explicit campaign log: what is validated, the open decode-graph wedge with ordered suspects, the probe methodology including the invalid-probe lesson, the proprietary-rebuild recipe, and the build/runtime gotchas. No code changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Add transport_provider_design.md section 4.3 with complete minimal recipes for the nine deployment shapes covered by the 4.2 contracts (udp loopback, per-decoder rings, cpu_roce rendezvous + hsb_fpga, mixed dispatch local/HSB, all-device_graph, partner drop-in, in-process channels). Renumber the two-axis-split section to 4.4 and cross-reference the cookbook from per_decoder_rings_design.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Root cause of the device-graph scheduler wedge: cooperative-launch co-residency starvation. The decode graph was captured with reserved_sms=0, sizing its cooperative grid for every SM; with the persistent dispatch graph resident, the device-side fire-and-forget launch queued forever and deadlocked at grid.sync(), stalling the tail self-relaunch. DecodingSession now captures with reserved_sms=1 (QEC_DECODE_GRAPH_RESERVED_SMS overrides, e.g. on rigs where Hololink RX/TX kernels are also resident). Same bug class, second instance (documented, plugin-side fix pending): HOST-path nv-qldpc decode launches unreserved cooperative kernels and hangs when a scheduler is resident -- in mixed deployments put a CPU decoder on host rings, or plumb reservation into the host decode. DeviceGraphRingConsumer logs the new trigger diagnostics (cudaq_dispatch_get_trigger_debug) at shutdown, and dispatched() now uses an async copy on a non-blocking stream -- a legacy default-stream cudaMemcpy synchronizes against the persistent scheduler graph and self-deadlocks. Proof (two-process, pinned-udp device ring, WSL2 laptop): sc4 app, 10 shots at p_spam=0.08, multi_error_lut host ring + nv-qldpc RelayBP device ring: trigger rc=0, fires=12 == tail_relaunches, both rings dispatched=72, decoder[1] corrections=2 logical_errors=0/10, clean teardown. Full regression suites stay green. Pairs with cuda-quantum commit 'Surface the device-side trigger-launch result'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…ss limit Pairs with cuda-quantum '[realtime] Key the bridge-provider loader by library name/path'. The server now passes each ring's resolved provider library to cudaq_bridge_create_from_library directly: no more CUDAQ_REALTIME_BRIDGE_LIB setenv side-channel for the per-ring path, no 'hololink is the builtin slot' special case, and rings may mix provider libraries freely -- the corresponding startup error and design-doc limitation are removed. DeviceGraphTransceiver likewise names its default provider library (hololink) as a plain string, with CUDAQ_REALTIME_BRIDGE_LIB still honored as the replacement override. Also note in --help that the installed cudaq-realtime is the source of truth for available providers and their arguments; the enumerated names and args are examples. Validated: two-process suite 4/4, per-decoder-rings app test, and the full mixed device-graph E2E (trigger rc=0, fires=10=tail_relaunches, both rings dispatched=60, real corrections on both decoders). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
decoding_server.cpp: - Set transport_from_cli when --transport is parsed: the documented "explicit --transport overrides the YAML transport section" precedence was dead code (the section's provider always won). - The all-device_graph path now resolves its provider with the same precedence as the per-ring loop (CLI > section device_graph override > section provider > built-in default) instead of consulting only --transport != udp. - The v1-provider endpoint fallback advertises the ring's own resolved provider, not the process default. - Guard --timeout and the provider port token against non-numeric input (clean error instead of an uncaught std::stoi/stoul exception). - Include DeviceGraphRingConsumer.h so the weak C-ABI redeclarations are compile-checked against the canonical prototypes. CQR components: - Consolidate the SM-reservation knob on the documented device-graph env family: DecodingSession reads QEC_DEVICE_GRAPH_RESERVED_SMS (was the undocumented QEC_DECODE_GRAPH_RESERVED_SMS), and the transceiver's parsed-but-unused reserved_sms field is removed. - Delete dead helpers left behind when the scheduler wiring moved into DeviceGraphRingConsumer (alloc_pinned_mapped, populate_device_call, GPU_CUDA_CHECK) -- they produced -Wunused-function noise. - DeviceGraphRingConsumer.h self-compiles with the component macro off (the C-ABI block's includes were inside the #ifdef). - Free the function table, shutdown flag, and stats allocations when the scheduler-stream create fails (the one error path without a cleanup ladder). - Fix a stale HOLOLINK_FRAME_SIZE error message (QEC_DEVICE_GRAPH_FRAME_SIZE) and a stale SessionRegistry comment; correct the READY-line doc: the leading port= token belongs to the first decoder listed in the YAML, not necessarily decoder id 0. Validated: two-process suite 4/4, per-decoder-rings app test, and the mixed device-graph E2E (trigger rc=0, fires=10=tail_relaunches, both rings dispatched=60, correct RelayBP decodes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
… path The wire's identity lives in the YAML transport section; --transport now applies only to configs that intentionally leave the wire unspecified (one YAML reused across wires, selected per launch -- the pattern every two-process test uses). A config that names a provider combined with an explicit --transport is rejected at startup instead of resolved by a precedence rule, in both the mixed per-ring path and the standalone all-device_graph path. Help text and design docs updated to match. New coverage for the YAML path, which previously had none: - DecoderYAMLTest.TransportSectionAndMixedDispatch: round-trips the transport section (provider/args + device_graph shape override) with a host + device_graph decoder mix, and pins the exact YAML key spelling against a literal document. - DecodingServerTwoProcess.TwoProcessHostDispatchYamlTransportSection: full two-process decode where the server is launched WITHOUT --transport and the YAML names the provider and its args. - DecodingServerTwoProcess.TransportCliConflictsWithYamlSection: the conflict is rejected before READY with a nonzero exit and a pointed error message. - app_examples.surface_code-4-yaml-mixed-dispatch: the flagship mixed host+device_graph two-process flow as a gated ctest (registered when the server links the device-graph component; skips at runtime without a GPU or the nv-qldpc plugin). Asserts per-decoder rings on the READY line, per-decoder results in the app, a healthy scheduler (trigger rc=0, fires == tail_relaunches > 0), and traffic on both rings. ServerProcess (test harness) learns to launch without the CLI transport args, to fold stderr into the captured output, to report the child's exit code, and to bail out on EOF instead of burning the READY timeout. 29/29 in the decoding-server/YAML/session/app regression sweep, including the new tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The after-diagram now reflects the one-ring-per-decoder consumers and the YAML transport section (--transport as fallback, conflict rejected). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Collaborator
Author
|
/ok to test 2229b85 |
1tnguyen
reviewed
Jul 13, 2026
Comment on lines
+210
to
+212
| parse_endpoint_token(endpoint_info, "qp", qp_number_); | ||
| parse_endpoint_token(endpoint_info, "rkey", rkey_); | ||
| parse_endpoint_token(endpoint_info, "buffer_addr", buffer_addr_); |
Contributor
There was a problem hiding this comment.
These seem to be HSB-specific info. Perhaps we can drop them (look like we don't use these anymore)
1tnguyen
reviewed
Jul 13, 2026
Comment on lines
+133
to
+145
| const std::vector<std::string> args = { | ||
| "--device=" + config.device_name, | ||
| "--peer-ip=" + config.peer_ip, | ||
| "--remote-qp=" + std::to_string(config.remote_qp), | ||
| "--gpu=" + std::to_string(config.gpu_id), | ||
| "--page-size=" + std::to_string(page_size), | ||
| "--num-pages=" + std::to_string(config.num_pages), | ||
| "--payload-size=" + std::to_string(payload_size), | ||
| }; | ||
| std::vector<char *> argv; | ||
| argv.reserve(args.size()); | ||
| for (auto &a : args) | ||
| argv.push_back(const_cast<char *>(a.c_str())); |
Contributor
There was a problem hiding this comment.
I think we'd need to generalize this for non-HSB bridge implementation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
VERY EARLY DRAFT; only posting for paired review with NVIDIA/cuda-quantum#4915
Note: do not actually review code changes yet as they are guaranteed to change.
Builds on #670. Some changeset would be reduced once that PR lands.
Summary
This PR re-architects the realtime decoding server around the CUDA-Q bridge-provider boundary. The server no longer contains any transport code: the wire is named in the deployment YAML (or a
--transportfallback), loaded at runtime as alibcudaq-realtime-bridge-<name>.soprovider, and every decoder gets its own ring buffer and its own consumer -- a host dispatcher thread fordispatch: hostdecoders, or the GPU device-graph scheduler fordispatch: device_graphdecoders, both side by side in one server process. A partner transport library drops in as a single.sowith zero decoding-server changes.The guiding litmus test for the layering: cudaq-realtime and its transport providers never say "QEC" or "decoder"; the QEC library code never names a wire (UDP, RoCE, hololink). Wire names appear only where deployments are described: YAML configs, launch scripts, tests, and docs.
Before
DT_NEEDEDon the server binary). Adding a wire meant editing the server.--transporton the CLI AND a per-decoder YAML key), with silent misconfiguration when they disagreed.device_id == decoder_id) were impossible.gpu_roce) was hololink-only, named after one wire, and could not coexist with host-dispatch decoders in the same process.After
decoding_serverspeaks only YAML + thecudaq_bridge_*C API. Provider libraries resolve by name next to the cudaq-realtime install (QEC_BRIDGE_PROVIDER_DIR) or verbatim by path (partner drop-in). The QEC libraries link no transport libraries; the rendezvous/hsb_fpga handshake moved down into the cpu_roce provider (companion cuda-quantum PR).dispatch: host|device_graphpicks HOW RPCs execute; the top-level, shape-keyedtransport:section (provider,args, plus adevice_graph:override for the rings that must be GPU-pollable) picks the wire for the whole deployment.--transportis a fallback for configs that intentionally leave the wire unspecified (one YAML reused across wires, selected per launch); a YAML that names a provider combined with an explicit--transportis a startup error, never a silent precedence decision. All pre-release aliases (transport:as a per-decoder key,--transport=gpu_roce,HOLOLINK_*env fallbacks) are removed -- this component is new this cycle, so there is no compatibility surface to keep.QEC_DECODING_SERVER_READY port=<p0> ... ring<id>=<port>) before any blocking connect, and shutdown reports per-ring traffic (QEC_DECODING_SERVER_RING decoder=<id> dispatched=<n>). Callers route withcudaq::device_call(decoder_id, ...)and per-device endpoint args (udp-port.<id>=<port>).DeviceGraphRingConsumerruns the CUDA-Q device-graph scheduler (self-relaunching GPU dispatch graph + the decoder's captured decode graph) as a ring consumer over any GPU-pollable ring -- hololink DOCA rings on a rig, or pinned+mapped udp rings (--pinned-rings) on any CUDA box.GpuRoceTransceiverwas renamed to what it actually is (DeviceGraphTransceiver, the dispatch engine, transport-blind) and now delegates to the ring consumer; the standalone all-device_graph path remains for the HSB flow.grid.sync().DecodingSessionnow captures withreserved_sms=1(QEC_DEVICE_GRAPH_RESERVED_SMSto raise it on rigs where hololink RX/TX kernels are also resident). The same bug class affects host-path GPU-cooperative decodes when a scheduler is resident -- documented, with the mixed-deployment guidance of a CPU decoder on host rings until the plugin plumbs reservation into its host path.transport_provider_design.md(contracts + a deployment cookbook: sample YAML + exact launch line for nine scenarios),per_decoder_rings_design.md(topology + reproduction recipe),per_decoder_rings_validation_notes.md(what was proven where, including the wedge investigation and the invalid probes it burned). New coverage: transport-section parse/round-trip with a mixed host+device_graph decoder set, a two-process decode where the YAML alone names the wire, CLI/YAML conflict rejection, per-decoder-rings two-process test asserting traffic on each ring, andapp_examples.surface_code-4-yaml-mixed-dispatch-- the flagship mixed flow as a gated ctest (registered only when the server links the device-graph component; skips without a GPU or the nv-qldpc plugin) asserting scheduler health via the new trigger diagnostics (rc=0, fires == tail_relaunches > 0).Sample configurations (from the deployment cookbook)
The full cookbook (
libs/qec/lib/realtime/transport_provider_design.md, section 4.3) covers nine scenarios with the exact launch line and caller config for each; three representative ones:Two host decoders, one udp ring per decoder -- the YAML says nothing about the wire, so
--transport(default udp) selects it per launch; the READY line publishes every ring's endpoint.Caller routes per decoder with device-scoped endpoint args:
--cudaq-device-call=udp udp-host=127.0.0.1 udp-port=<P0> udp-port.1=<P1>.Mixed dispatch (host CPU decoder + GPU device-graph decoder), runs on any CUDA box -- the wire lives in the YAML transport section; the shape-keyed override adds
--pinned-ringsonly to the device_graph ring so the GPU scheduler can poll it. No--transporton the command line (combining both is a startup error). This is exactly what the new gated ctest runs.Partner transport drop-in -- an out-of-tree provider library is named by path, verbatim; its args are forwarded untouched. Zero decoding-server changes.
Validation
Dependencies
cudaq_bridge_create_from_library, the v2 endpoint/geometry queries, per-device sessions and device-scoped channel args, the udp provider's--pinned-rings, andcudaq_dispatch_get_trigger_debug.CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE); builds without them still compile and the device-graph paths fail at runtime with a clear not-linked error, and the gated ctest is simply not registered.🤖 Generated with Claude Code