Skip to content

docs(aws-healthomics): parameter types, run failure patterns, and the ECR Public fallback - #184

Open
hmkim wants to merge 1 commit into
kirodotdev:mainfrom
hmkim:docs/healthomics-run-failure-guidance
Open

docs(aws-healthomics): parameter types, run failure patterns, and the ECR Public fallback#184
hmkim wants to merge 1 commit into
kirodotdev:mainfrom
hmkim:docs/healthomics-run-failure-guidance

Conversation

@hmkim

@hmkim hmkim commented Aug 14, 2026

Copy link
Copy Markdown

Sequencing: two sections reference MCP server tools that are not released yet — ValidateAHORunReadiness (awslabs/mcp#4486) and WaitForAHORun (awslabs/mcp#4493). Everything else stands on its own. Happy to split those two out if you would rather land this first.

Context

Four gaps found while taking a workflow through migration and running it end to end. Each one cost a failed run to discover, and none of them were workflow defects.

Parameter types

running-a-workflow.md had no guidance on how parameter values map to WDL types. Passing a struct as a JSON-encoded string produces:

InputError: check JSON input; couldn't construct SampleMeta from "{\"owner\": \"user\"}" (in sample_meta)

Two things make this expensive. Inputs are validated by the engine rather than by StartRun, so it arrives minutes in, after PENDING — the same cost as a real defect. And the message names the struct, not the mistake, so the natural next move is to go looking at the struct definition.

parameterTemplate from GetAHOWorkflow does not resolve it either: it carries only description and optional, with no type information. The type has to be read from the workflow's input {} block, which is now stated.

Adds a type table and the failing and working forms side by side.

Run failure patterns

troubleshooting.md covered creation failures and VPC connectivity, and routed everything else to DiagnoseAHORunFailure. Adds a table mapping observed messages to what to check first.

Deliberately framed as starting points rather than diagnoses — S3 access denied alone can be role policy, bucket policy or KMS, and presenting one as the answer would send an agent down a single track. Each row is ordered by likelihood and the section says to fall back to the logs.

Also records two properties that shape how to read any run failure:

  • The message names where execution stopped, not what is misconfigured. A missing container registry map surfaces as a task-level image URI error partway through the run, after earlier tasks have already burned billed compute.
  • Input errors arrive after PENDING.

ECR Public fallback

ecr-pull-through-cache.md already documents creating the Docker Hub secret, but not what happens without one:

Credential ARN is required for Docker Hub pull-through cache.

Where credentials are unavailable, many images a legacy workflow names on Docker Hub are also published on ECR Public, which needs none. I hit this with ubuntu:20.04 and reached it as ecr-public/ubuntu/ubuntu:20.04.

Two things the new section is careful about:

  • The repository path differs between the registries, so the Docker Hub name cannot be reused. It points at the gallery to resolve the real path rather than listing mappings I have not verified.
  • This is URI replacement, not a registry map entry. It substitutes one image for another, so routing a Docker Hub URI to an ECR Public repository through a map would hide the substitution from the next reader. That matches the decision table already in the WDL migration SOP.

It also notes that tags and contents are not guaranteed to match across registries.

Packaging

Records that a bare .wdl/.nf/.cwl file behind a path or S3 URI is packaged automatically (awslabs/mcp#4485), and the limit of that: only the named file goes in, so a workflow with imports still has to be packaged. Also notes path_to_main is unnecessary for the single-entry archive, and that inline content and .zip files are left untouched.

Correction to the failure procedure

The existing steps assumed every run failure is a workflow defect:

2. Fix the workflow definition based on the diagnosis.
3. Create a new version via CreateAHOWorkflowVersion

Three of the four failures in my session were configuration, not definition — role, bucket region, bucket permissions. A new workflow version is the wrong move for those, and an agent following this literally would create versions that change nothing. Now conditional on having actually modified the definition.

Verification

All seven cross-document anchors checked against the heading set. Section spacing matches the surrounding files. Content limited to behaviour observed against the service in ap-northeast-2 and us-east-1.

Related

Follows #179. Companion to the three MCP server PRs: #4485, #4486, #4493.

…, and the ECR Public fallback

Four gaps found while running a migrated workflow end to end, each of which cost
a failed run to discover.

Parameter types. A struct passed as a JSON-encoded string is rejected by the
engine, not by StartRun, so it costs the same minutes as a real defect before
reporting, and the message names the struct rather than the mistake. The
parameterTemplate from GetAHOWorkflow cannot resolve this either: it carries only
description and optional, with no type information, so the type has to be read
from the workflow's input block. Adds a type table and the failing and working
forms side by side.

Run failure patterns. Adds a table mapping observed error messages to the causes
worth checking first, framed as starting points rather than diagnoses since the
same message has several possible causes. Records two properties that shape how
to read any run failure: the message names where execution stopped rather than
what is misconfigured, and input errors arrive after PENDING.

ECR Public fallback. Docker Hub pull-through caches require a Secrets Manager
credential, and the call is rejected outright without one. Where credentials are
unavailable, many images a legacy workflow names on Docker Hub are also on ECR
Public, which needs none. Documents that path, including the repository name
differing between the registries, and flags it as URI replacement rather than a
registry map entry, since it substitutes one image for another.

Packaging. Records that a bare .wdl/.nf/.cwl file behind a path or S3 URI is now
packaged automatically, that this does not extend to a workflow with imports, and
that inline content and .zip files are left alone.

Also corrects the failure procedure, which assumed every run failure is a
workflow defect and directed a new workflow version unconditionally. A run that
failed on its role, output location or inputs needs neither.

Depends on two MCP server tools that are not yet released: ValidateAHORunReadiness
(awslabs/mcp#4486) and WaitForAHORun (awslabs/mcp#4493).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants