Universal heterogeneous frontier-model runtime with self-optimizing scheduling and self-healing execution

[Cuuper22]

North star

Build a universal inference runtime that accepts a frontier-model workload plus an arbitrary inventory of accelerators and automatically discovers, validates, and executes a near-optimal deployment across GPUs, CPUs, NPUs, FPGAs, memory tiers, interconnects, racks, regions, and power envelopes.

The runtime should make vendor boundaries, topology accidents, thermal limits, memory asymmetry, and partial hardware failure look like internal implementation details rather than architecture constraints.

This is not merely a scheduler. It is a closed-loop execution system that:

1. models the full hardware/software stack;
2. decomposes a model into executable regions;
3. searches candidate placements, precisions, kernels, and communication plans;
4. predicts latency, throughput, memory, energy, cost, and reliability;
5. profiles reality and corrects its model online;
6. detects novel failure modes;
7. synthesizes and validates recovery plans;
8. changes execution live without violating declared numerical or service-level contracts.

Scope merged from the original absurd prompt set

This issue deliberately folds three ideas into one gpu_stack-native program:

• Universal heterogeneous inference runtime: decompose and execute any frontier model across whatever compute exists while preserving declared numerical behavior.
• Whole-stack adaptive scheduler: model kernels, memory channels, topology, contention, thermal state, power caps, queueing, and user intent so placement is driven by actual system state rather than static device labels.
• Self-healing distributed execution: detect unseen failure modes, synthesize candidate mitigations, model-check them against the live execution graph, deploy them progressively, and generate a causal incident record before the fault becomes user-visible when physically possible.

Product boundary

The first implementation should be a research-grade planner, simulator, trace format, and reference runtime, not an immediate claim of production support for every accelerator.

The architecture must nevertheless avoid baking in CUDA, one graph IR, one collective library, one topology model, or one notion of numerical equivalence.

Primary user stories

Runtime architect

As a runtime architect, I can submit a model graph, serving objective, numerical contract, and hardware inventory, and receive an executable plan with explicit assumptions, predicted performance, uncertainty, and rejected alternatives.

Infrastructure operator

As an infrastructure operator, I can add an irregular fleet containing mixed accelerator generations and vendors, and the runtime can exploit useful capacity without requiring a hand-authored deployment per machine class.

Reliability engineer

As a reliability engineer, I can inject device, link, memory, thermal, process, kernel, and software faults and observe bounded degradation, deterministic recovery decisions, and a machine-readable explanation of every mitigation.

Researcher

As a researcher, I can compare scheduling policies, partition strategies, precision policies, and recovery mechanisms against reproducible scenarios backed by the existing gpu_stack equation and provenance system.

Proposed architecture

1. Canonical workload IR

Create a hardware-neutral execution IR that retains enough semantic information to support aggressive transformations without collapsing everything into opaque kernels.

Required concepts:

• tensor shape, dtype, sparsity, layout, and aliasing;
• operator semantics and legal rewrites;
• control flow, state, KV-cache behavior, and dynamic shapes;
• numerical sensitivity and allowed error envelopes;
• latency-critical versus throughput-oriented regions;
• prefill, decode, speculative branches, experts, routing, and communication phases;
• recomputation, checkpointing, paging, quantization, and offload legality;
• deterministic and nondeterministic operator annotations;
• side effects and externally observable ordering constraints.

Candidate adapters may ingest PyTorch FX/export, MLIR, StableHLO, ONNX, TensorRT graphs, XLA HLO, vendor compiler traces, or synthetic gpu_stack workload descriptions. The canonical IR must not become a thin alias for any one of them.

2. Capability-normalized hardware graph

Represent the available system as a live graph rather than a flat device list.

Nodes may include:

• compute engines;
• HBM, SRAM, DDR, CXL, NVMe, and remote memory tiers;
• NICs, switches, PCIe roots, coherent fabrics, and network paths;
• power domains, cooling domains, racks, zones, and regions;
• compiler/runtime versions and available kernel libraries.

Edges carry measured and uncertain properties:

• directional bandwidth and latency;
• contention domains;
• collective support;
• coherency and synchronization semantics;
• transfer setup costs;
• failure correlation;
• energy and monetary cost;
• trust and isolation constraints.

Every property must support provenance, validity ranges, uncertainty, and live updates using gpu_stack's existing symbolic registry conventions.

3. Numerical contract system

Users declare what “preserving behavior” means for each workload:

• bitwise reproducibility;
• deterministic replay under a fixed seed;
• bounded absolute/relative tensor error;
• bounded distributional divergence;
• task-level quality floors;
• exact logits for selected paths;
• mixed contracts by graph region.

All candidate transformations must produce a proof obligation or empirical validation obligation. No optimization silently weakens the numerical contract.

4. Multi-level decomposition engine

Search over:

• operator placement;
• graph partitioning;
• tensor, pipeline, data, expert, sequence, context, and speculative parallelism;
• fusion and defusion;
• kernel families;
• precision and quantization;
• KV-cache placement and movement;
• memory paging and recomputation;
• batch formation and continuous batching;
• replica count and routing;
• collective algorithms and chunk sizes;
• asynchronous overlap schedules;
• power caps and thermal-aware pacing;
• cross-region spillover when allowed.

The engine must represent incompatible choices explicitly and retain a rejection explanation for important discarded plans.

5. Hierarchical optimizer

A single monolithic optimizer will not scale. Use a hierarchy:

1. feasibility pruning from memory, operator support, numerical contracts, and policy constraints;
2. analytical lower bounds from roofline, communication, queueing, and energy models;
3. coarse partition search;
4. learned or Bayesian cost-model ranking;
5. discrete optimization for placement and routing;
6. local schedule refinement;
7. hardware profiling of finalists;
8. online adaptation after deployment.

Candidate methods may include MILP/CP-SAT, dynamic programming, graph partitioning, MCTS, evolutionary search, Bayesian optimization, differentiable surrogates, and contextual bandits. The implementation should permit policy comparison rather than canonizing one search algorithm.

6. Adaptive whole-stack scheduler

Translate the earlier OS-scheduler idea into gpu_stack's domain. The scheduler observes and predicts:

• per-kernel runtime distributions;
• queue depth and request mix;
• memory pressure and fragmentation;
• interconnect congestion;
• cache and KV locality;
• collective interference;
• power draw, thermal headroom, throttling, and cooling lag;
• hardware error counters;
• process/runtime health;
• tenant priorities, deadlines, and cost budgets.

It continuously chooses admission, batching, routing, placement, preemption, migration, replication, and pacing decisions.

The scheduler must distinguish planned adaptation from fault recovery and must expose the causal factors behind each decision.

7. Self-healing execution plane

Implement a fault-observation and response loop:

1. detect deviation from expected runtime distributions or invariants;
2. localize likely causes across graph, device, memory, link, runtime, and software layers;
3. generate candidate mitigations;
4. simulate or model-check their impact on correctness and service objectives;
5. canary the safest candidate;
6. expand, roll back, or escalate;
7. preserve a causal trace and minimal reproducer.

Mitigations may include:

• rerouting collectives;
• replacing a kernel variant;
• reducing precision only where the contract allows;
• rematerializing or relocating state;
• shrinking or reshaping batches;
• migrating graph regions;
• dropping a degraded device or link;
• changing parallelism strategy;
• power/thermal pacing;
• restarting isolated runtime components;
• falling back to a slower verified plan.

The recovery engine must never synthesize arbitrary executable patches and deploy them without a bounded validation pipeline. “Self-healing” means generated plans under explicit invariants, not YOLO production mutation.

8. Digital twin and counterfactual simulator

Before live deployment, replay candidate plans through a simulator driven by gpu_stack equations and calibrated traces.

The simulator should support:

• event-driven execution;
• uncertain durations and failure probabilities;
• contention and topology effects;
• queueing and request-arrival models;
• power and thermal dynamics;
• correlated failures;
• counterfactual replay of production traces;
• confidence intervals over predicted outcomes.

Observed production data must update calibration without erasing source provenance or conflating measured facts with inferred parameters.

9. Portable execution backends

Define a backend contract for:

• capability discovery;
• memory allocation and movement;
• kernel compilation/loading;
• synchronization and events;
• collectives;
• telemetry;
• fault injection;
• checkpoint/state migration.

Initial reference backends can be CPU plus one GPU ecosystem, followed by simulated adapters for additional accelerator types. The core planner must treat unavailable backend features as capability constraints rather than special-case branches.

10. Explanation and provenance layer

Every final plan should answer:

• why this decomposition was selected;
• which constraints were binding;
• which assumptions materially affected the result;
• what alternatives were rejected and why;
• predicted versus observed performance;
• uncertainty intervals;
• numerical validation performed;
• adaptation and recovery decisions made after launch.

Integrate with gpu_stack's dependency-cone, trace, scenario, uncertainty, audit, and documentation-freshness machinery.

BDD feature plan

Feature: ingest a heterogeneous fleet

Given an inventory containing mixed compute devices, memory tiers, and links
And each capability has units, provenance, validity, and uncertainty metadata
When the hardware graph is validated
Then unsupported or contradictory capabilities are reported explicitly
And the usable topology is exported deterministically
And no device is reduced to a misleading single throughput number.

Feature: generate a feasible execution plan

Given a model workload IR
And a numerical contract
And a validated hardware graph
When the planner searches candidate decompositions
Then every returned plan fits memory and capability constraints
And every transformation is legal under the numerical contract
And every rejected high-ranking candidate has a machine-readable rejection reason
And the result includes latency, throughput, energy, cost, reliability, and uncertainty estimates.

Feature: preserve strict numerical behavior

Given a workload marked bitwise deterministic
When the planner considers quantization, alternate kernels, reordering, or cross-device partitioning
Then candidates without a determinism proof or validated equivalent are rejected
And the accepted plan reproduces the reference outputs over the conformance corpus.

Feature: support bounded numerical relaxation

Given a workload with region-specific error budgets
When lower precision or approximate kernels improve the objective
Then the planner may use them only within declared regions
And validation reports observed error and confidence
And a violation automatically disables the candidate and selects a verified fallback.

Feature: optimize for user intent

Given identical workload and hardware inputs
And one objective prioritizes p99 latency
And another prioritizes tokens per dollar
When plans are generated
Then the selected plans may differ
And the explanation identifies which objective terms caused the difference.

Feature: adapt to live contention

Given a deployed plan meeting its service objective
And a competing workload saturates a shared interconnect
When observed communication time exceeds the calibrated envelope
Then the scheduler evaluates rebatching, rerouting, and repartitioning
And applies the lowest-risk plan predicted to restore the objective
And records predicted versus realized improvement.

Feature: adapt to thermal throttling

Given a device approaching a thermal limit
When projected throttling would violate the service objective
Then the scheduler evaluates workload migration, power capping, and pacing
And chooses a plan using the thermal time constant rather than reacting only after clocks collapse.

Feature: recover from device loss

Given an execution plan spanning multiple devices
And one device becomes unavailable during service
When the recovery engine receives the fault event
Then it localizes affected graph state
And restores or reconstructs required state
And activates a verified degraded-mode plan
And does not violate the declared numerical contract
And reports the recovery point objective, recovery time, and lost work.

Feature: recover from a degraded link

Given a collective plan using several fabric paths
And one path develops high loss or latency without fully failing
When telemetry violates the path model
Then the runtime distinguishes link degradation from compute slowdown
And reroutes or changes the collective algorithm
And validates that the new plan avoids correlated congestion.

Feature: handle a novel failure pattern

Given no exact rule exists for an observed failure signature
When anomaly detection identifies a persistent invariant violation
Then the system generates multiple bounded mitigation candidates
And rejects candidates that violate correctness or safety invariants
And tests the safest remaining candidate in simulation or a canary slice
And rolls back automatically if measured behavior diverges from prediction.

Feature: learn from execution without corrupting provenance

Given a sourced analytical parameter and a separately inferred calibration correction
When runtime traces update the cost model
Then the source value remains unchanged
And the calibration layer records dataset, timestamp, confidence, and validity range
And users can reproduce predictions using either sourced-only or calibrated modes.

Feature: compare policies reproducibly

Given a fixed scenario pack and random seed
When two planning or scheduling policies are evaluated
Then both receive identical workload, hardware, and fault traces
And results include confidence intervals and paired comparisons
And the report distinguishes model error from policy error.

Feature: deterministic replay

Given a captured execution trace
When replay mode is invoked
Then planner, scheduler, and recovery decisions can be reproduced from recorded inputs
And nondeterministic observations are represented explicitly
And causal explanation identifiers remain stable.

Milestones

M0 — terminology and contracts

• ADRs for workload IR, hardware graph, numerical contracts, backend API, and trace schema.
• Explicit non-goals and trust boundaries.
• Scenario definitions for homogeneous baseline, heterogeneous pair, irregular four-device node, and faulted cluster.

M1 — planner skeleton in simulation

• Canonical workload IR for a constrained transformer subset.
• Capability-normalized hardware graph.
• Deterministic feasibility planner.
• Analytical latency/memory/communication/energy estimates.
• Explanation and rejection traces.
• CPU-only simulated execution.

M2 — calibrated heterogeneous planning

• One real GPU backend plus CPU fallback.
• Profiling harness and calibration layer.
• Partition, precision, memory-placement, and collective search.
• Predicted-versus-observed reports.
• Numerical conformance suite.

M3 — adaptive scheduler

• Live telemetry ingestion.
• Request admission, continuous batching, routing, and migration.
• Contention, power, and thermal models.
• Policy simulator and reproducible benchmarks.

M4 — self-healing plane

• Fault taxonomy and injection framework.
• Invariant and anomaly engine.
• Bounded mitigation synthesis.
• Simulation/canary validation and rollback.
• Incident trace and causal report generation.

M5 — broader portability

• Additional backend adapters or high-fidelity simulated targets.
• Cross-vendor planning demonstrations.
• Dynamic expert routing, speculative decoding, and distributed KV-cache scenarios.
• Multi-node and optional multi-region planning.

Suggested repository modules

gpu_stack/
  runtime/
    ir.py
    numerical_contracts.py
    hardware_graph.py
    capabilities.py
    backend.py
    planner.py
    decomposition.py
    objectives.py
    feasibility.py
    scheduler.py
    telemetry.py
    adaptation.py
    recovery.py
    invariants.py
    fault_injection.py
    replay.py
    trace_schema.py
  simulation/
    event_engine.py
    queueing.py
    contention.py
    thermal.py
    power.py
    failure_models.py
  calibration/
    profiler.py
    observations.py
    cost_models.py
    validity.py
  presets/
    runtime_scenarios.py

Exact placement should follow the repository's current cohesion rules rather than this sketch blindly.

Evaluation matrix

Measure at minimum:

• feasibility rate;
• planner wall time;
• prediction error for latency, throughput, peak memory, energy, and communication;
• p50/p95/p99 latency;
• throughput and tokens per dollar;
• joules per token;
• hardware utilization and stranded capacity;
• numerical deviation and task-quality delta;
• adaptation convergence time;
• fault detection precision/recall;
• mean time to mitigation;
• recovery success rate;
• rollback rate;
• explanation completeness;
• reproducibility under replay.

Required adversarial scenarios

• heterogeneous devices where the theoretically fastest device worsens end-to-end latency because of transfer cost;
• a topology report that lies or becomes stale;
• memory fragmentation despite sufficient aggregate capacity;
• thermal throttling with delayed cooling response;
• collective interference from an unrelated workload;
• silent numerical corruption from one backend;
• intermittent link degradation rather than clean failure;
• correlated rack-level failure;
• planner model error that makes a candidate look better than reality;
• oscillation between two adaptations;
• recovery action that fixes throughput but violates numerical or tail-latency constraints;
• malicious or malformed capability metadata.

Definition of done for the first vertical slice

• A small transformer workload is represented in the canonical IR.
• A scenario contains at least CPU plus two distinct accelerator capability profiles, real or simulated.
• The planner produces multiple feasible decompositions and explains its selection.
• The selected plan executes through a reference backend or faithful simulator.
• Prediction and observation are compared with uncertainty.
• A device-loss and a degraded-link fault are injected.
• The runtime activates verified fallback plans without violating the declared numerical contract.
• The entire run is replayable from a versioned trace.
• Tests cover every BDD scenario implemented in the slice.
• verify, audit, docs-stat freshness, source cleanliness, and provenance gates remain green.

Research questions intentionally left open

• What is the minimal IR that preserves enough semantics for cross-backend optimization without becoming impossible to lower?
• Which numerical contracts can be proven statically, and which require empirical conformance?
• Where should analytical models end and learned cost models begin?
• How should the planner quantify epistemic uncertainty for hardware it has barely observed?
• How can online adaptation avoid oscillation and exploitation of cost-model errors?
• Which recovery actions can be model-checked quickly enough for live use?
• How should state migration work for massive distributed KV caches and expert shards?
• Can a policy generalize across vendors without quietly learning vendor-specific identities?
• How should performance, cost, energy, reliability, and numerical quality be composed without hiding unacceptable tradeoffs in one scalar score?

Anti-bullshit constraints

• No claim of universal support based only on a common graph importer.
• No silent fallback to CPU while reporting success.
• No benchmark that excludes data movement, warm-up, compilation, queueing, or recovery cost.
• No precision reduction without a declared and validated numerical contract.
• No learned optimizer without deterministic baselines and replayable comparisons.
• No self-healing claim that is merely restart-on-error.
• No source/calibration conflation.
• No hardcoded vendor branch in the core planner when a capability predicate can express the distinction.

This should become a long-horizon umbrella issue with child issues for each milestone and architecture contract.