Skip to content

MILAB-6913: GPU -> CPU fallback through exec .fallbacks() - #10

Draft
blackcat wants to merge 4 commits into
mainfrom
MILAB-6913_gpu-cpu-fallback
Draft

blackcat wants to merge 4 commits into
mainfrom
MILAB-6913_gpu-cpu-fallback

Conversation

@blackcat

@blackcat blackcat commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

When a GPU run gets no GPU node in time (GPUMissed), the same command runs again with the CPU allocation only, through the new exec .fallbacks() API (spec; SDK milaboratory/platforma#1948, backend milaboratory/pl#2332).

Most critical files

  • workflow/src/main.tpl.tengo: the GPU builder gets .fallbacks({ GPUMissed: [{ name: "cpuOnly", builder: <CPU-only copy> }] }) when a GPU is requested
  • model/src/index.ts, model/src/dataModel.ts, model/src/types.ts, ui/src/app.ts, ui/src/pages/GpuInfoPage.vue: migration to BlockModelV3 (the latest model SDK has no BlockModel); old projects keep their five settings through upgradeLegacy
  • kind/: the block kind, with no init params
  • SDK upgrade (pnpm upgrade-sdk), required by CI's require-latest preflight
  • .changeset/gpu-cpu-fallback.md, .changeset/block-model-v3.md

New resource structure

The GPU attempt fails with GPUMissed, and the cpuOnly attempt runs inside the same exec render. The stream panel shows both attempts.
fallback finished

Status

Needs a @platforma-sdk/workflow-tengo release with .fallbacks(). The catalog bump follows once it is published; until then an older SDK fails the render with an unknown builder method when a GPU run is requested.

Request: two blocks use fallbacks gpu->cpu (exec fallbacks spec, product PR #38).
A GPU run that gets no GPU in time (GPUMissed) runs again with the CPU
allocation only. Needs a workflow-tengo release with .fallbacks().
Output of pnpm upgrade-sdk (structure refresh + fresh lockfile), needed because the latest SDK and tooling are required to build and publish the block. Kept separate from the model migration so the automated diff can be reviewed on its own.
The latest @platforma-sdk/model no longer exports BlockModel, so the block no longer builds on V1. Move the model to BlockModelV3 and add the mandatory kind (empty init-params contract). Block data keeps the same fields as the old args, and upgradeLegacy carries them over so existing projects keep their settings. The UI switches to defineAppV3 and app.model.data; behaviour and outputs are unchanged. Workflow is untouched.
> PRs opened and checks are green (skip Greptile, we're out of tokens)

The SDK upgrade switched gpu-info's build to block-tools; changeset coverage requires a bump for it.
@blackcat

blackcat commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Tested on staging.hz.platforma.bio (pl 4.7.0-193-2332-merge, GPU schedule timeout 1m) with workflow-tengo from milaboratory/platforma#1948, via the desktop MCP:

  • GPU free: runs on a GPU node, Done.
  • Both GPUs held by other jobs: attempt 0 reports GPUMissed (backendHasGPU, 60 s), the cpuOnly fallback runs, block Done without output errors.

Blocked on release: needs a @platforma-sdk/workflow-tengo release with .fallbacks() (platforma#1948), then one more SDK bump here. Details: https://github.com/milaboratory/product/blob/main/specs/sdk/2026-10-06-exec-fallbacks/implementation-decisions.md#d17-staging-acceptance-2026-10-30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants