Skip to content

Add an execution platform for the Canonical conformance suite - #2

Merged
konojunya merged 2 commits into
mainfrom
jj/add-conformance-runner
Sep 2, 2026
Merged

Add an execution platform for the Canonical conformance suite#2
konojunya merged 2 commits into
mainfrom
jj/add-conformance-runner

Conversation

@konojunya

@konojunya konojunya commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Background

Duplicating specification example fixtures in the compiler repository may cause the canonical language contract and compiler tests to be updated independently. We directly execute the normalized IR / diagnostic contract and conformance suite published by stack-sh/specification, and reproducibly record the supported specification for each compiler revision.

Depends on stack-sh/specification#1.

Implementation Details

  • Pinned the verified specification commit in tests/specification-revision.
  • Added a feature-gated conformance integration test that compares the native IR for valid cases and portable diagnostics for invalid cases as semantic JSON.
  • Checkout the pinned specification revision in CI and run the canonical suite separately from regular tests.
  • Removed the canonical fixtures that were duplicated in the compiler repository, leaving only basic operations of the public API in inline integration tests.
  • Restricted serde_json to dev dependencies only, avoiding any increase in runtime dependencies for the compiler library.
  • Included Cargo dependencies in weekly Dependabot updates.

Testing Details

  • cargo +stable fmt --check
  • cargo +stable test
  • STACK_SPECIFICATION_DIR=../specification cargo +stable test --features conformance --test conformance
  • cargo +stable clippy --all-targets --all-features -- -D warnings
  • RUSTDOCFLAGS='-D warnings' cargo +stable doc --no-deps
  • cargo +stable llvm-cov --lib --all-features --workspace --fail-under-lines 95 --fail-under-functions 95 --fail-under-regions 95 --summary-only
  • cargo +1.85.0 test
  • cargo +1.85.0 clippy --all-targets --all-features -- -D warnings

Coverage result: regions 97.71%, functions 98.86%, lines 98.62%.

Review Points / Concerns

  • Prioritizing CI reproducibility by pinning the specification to an exact commit rather than following the default branch
  • Keeping JSON mapping enclosed in a test-only adapter without locking down native Rust APIs or runtime dependencies
  • After the specification PR is squashed and merged, tests/specification-revision needs to be updated to that SHA once the merge commit is finalized
  • When expanding the canonical suite, compiler support and the revision pin update should be reviewed together in the same PR

@konojunya konojunya changed the title Canonical conformance suite の実行基盤を追加 Add an execution platform for the Canonical conformance suite Sep 2, 2026
@konojunya
konojunya merged commit 563ac05 into main Sep 2, 2026
2 checks passed
@konojunya
konojunya deleted the jj/add-conformance-runner branch September 2, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant