Skip to content

Wait for exact-base quality baseline artifacts before comparison #516

Description

Problem

Exact-base quality workflows can start before the corresponding baseline workflow for a freshly merged base commit has uploaded artifacts. The download action fails immediately instead of waiting, causing a false-negative PR check.

Observed on PR #515:

  • base commit: ae216cad97a11eb389585edf3613a22dde4223b7
  • performance run: 31542690250
  • Linux benchmark itself completed successfully with healthy metrics
  • comparison failed closed with Snapshot file does not exist: baseline-perf/metrics.json
  • the exact-base Performance Baseline workflow completed successfully afterward
  • rerunning only the failed Linux job then passed

Coverage has the same exact-base artifact dependency and can race similarly.

Scope

  • wait for the exact-base performance/coverage baseline workflow and required artifact before download;
  • use a bounded timeout and explicit polling interval;
  • fail immediately with a clear error if the exact-base workflow completes unsuccessfully;
  • fail clearly on timeout or missing required artifact;
  • never fall back to a moving/latest-main baseline;
  • share and unit-test the waiting logic across performance and coverage workflows.

Acceptance criteria

  • a pending exact-base workflow is polled until its required artifact is available;
  • successful, failed, timed-out, and missing-artifact paths are tested;
  • performance and coverage workflows use the helper;
  • exact-base SHA remains visible in logs/reports;
  • full CI, comparator tests, coverage, performance, and CodeQL pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIssue identified by VS Code Team member as probable bugimportantIssue identified as high-priority

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions