Skip to content

ci: split slow integration test jobs to reduce CI wall-clock by ~17%#3570

Merged
masih merged 12 commits into
mainfrom
amir/plt-447-split-evm-interoperability-test-matrix
Jun 11, 2026
Merged

ci: split slow integration test jobs to reduce CI wall-clock by ~17%#3570
masih merged 12 commits into
mainfrom
amir/plt-447-split-evm-interoperability-test-matrix

Conversation

@amir-deris

@amir-deris amir-deris commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Split 3 slow integration test jobs into parallel matrix entries, each targeting independent test files. No test logic changed — only how tests are distributed across runners.

EVM Interoperability (2 → 4 jobs):

  • EVM Interoperability (Pointer Tests) — 7 pointer-type Hardhat tests
  • EVM Interoperability (Misc Tests) — SeiSolo, SetCodeTx, TransientStorage
  • Same split applied to Autobahn variants

EVM Module (2 → 4 jobs):

  • EVM Module (Compat) — EVMCompatabilityTest.js (identified as 70–77% of runtime via timing instrumentation)
  • EVM Module (Precompile & Endpoints) — EVMPrecompileTest.js, SeiEndpointsTest.js, AssociateTest.js + FlatKV steps
  • Same split applied to Autobahn variants

dApp Tests (1 → 2 jobs):

  • dApp Tests (Uniswap) — uniswapTest.js (219s, 62% of runtime)
  • dApp Tests (NFT & Steak) — nftMarketplaceTests.js + SteakTests.js (133s combined)

Results (measured over multiple CI runs)

Milestone Critical path job Wall-clock
Baseline (pre-PR) Autobahn EVM Interoperability (667s avg) ~19.3 min
After EVM Interop split Autobahn EVM Module (622s) 17.6 min
After EVM Module split dApp Tests (600s) 17.1 min
After dApp split Autobahn EVM Interop (Pointer Tests) (559s) 16.0 min

Total: ~3.3 min saved (~17%) per CI run.

Approach

Used timing instrumentation ([timing] log lines) on evm_tests.sh and dapp_tests.sh to identify the dominant test file in each job before splitting. Instrumentation removed after data was collected.

Bug fix: AssociateTest flake on Autobahn

Fixed a pre-existing flake in AssociateTest.js where verifyAssociation read afterEvm immediately after associateKeyStrict returned, relying on waitForBlocks(2) as a proxy for the EVM balance merge landing. On Autobahn this was insufficient. Replaced with a waitForCondition poll that waits until the EVM balance actually reaches the expected value before asserting. Same failure reproduced on PR #3566 (pre-split, original evm_tests.sh), confirming it is Autobahn-specific and unrelated to the job split.

@cursor

cursor Bot commented Jun 9, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
CI orchestration and test-runner distribution only, plus a targeted test stability fix; no production chain or application logic changes.

Overview
Splits slow Docker integration matrix jobs so EVM module, EVM interoperability, and dApp suites run on more parallel runners, targeting shorter CI wall-clock without changing test logic.

The workflow replaces monolithic script invocations with narrower entries: EVM Module becomes Compat (evm_compat_tests.shEVMCompatabilityTest.js) vs Precompile & Endpoints (evm_precompile_tests.sh plus FlatKV deploy/verify steps); EVM Interoperability splits pointer Hardhat tests from misc tests (SeiSolo, SetCodeTx, TransientStorage); dApp Tests split into Uniswap-only and NFT/Steak scripts. The same splits are mirrored for Autobahn matrix rows. New shell wrappers (evm_compat_tests.sh, evm_interoperability_misc_tests.sh, dapp_uniswap_tests.sh, dapp_nft_steak_tests.sh) hold the per-job test lists; pointer and precompile scripts are trimmed so each file runs in one job.

AssociateTest.js: verifyAssociation now polls with waitForCondition until the EVM balance matches the expected post-association value before asserting, instead of relying on fixed block waits that flake on Autobahn.

Reviewed by Cursor Bugbot for commit 72d923e. Bugbot is set up for automated code reviews on this repo. Configure here.

@amir-deris amir-deris changed the title Separated hardhat tests into two batches ci: split EVM Interoperability matrix job to halve critical-path duration Jun 9, 2026
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJun 11, 2026, 8:34 AM

@codecov

codecov Bot commented Jun 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.33%. Comparing base (09c70ba) to head (72d923e).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3570      +/-   ##
==========================================
- Coverage   59.17%   58.33%   -0.84%     
==========================================
  Files        2215     2136      -79     
  Lines      183370   174701    -8669     
==========================================
- Hits       108512   101919    -6593     
+ Misses      65047    63696    -1351     
+ Partials     9811     9086     -725     
Flag Coverage Δ
sei-db 70.41% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 118 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@amir-deris amir-deris changed the title ci: split EVM Interoperability matrix job to halve critical-path duration ci: split slow integration test jobs to reduce CI wall-clock by ~17% Jun 10, 2026
@amir-deris amir-deris requested review from bdchatham and masih June 10, 2026 17:10
@bdchatham

Copy link
Copy Markdown
Contributor

Verified the split preserves coverage — the six new scripts run exactly the same test files as the three they replace (EVM Module 4≡4, Interop 10≡10, dApp 3≡3), nothing dropped or duplicated. Wall-clock improvement looks real too.

One thing worth confirming before merge: the old scripts ran each group sequentially against a single chain, whereas the split now runs them on separate fresh clusters. So if any test depended on on-chain state left behind by a sibling that ran earlier in the same script — e.g. AssociateTest/SeiEndpoints after EVMCompatability, or the steak/nft reorder — the split would break it silently. If each npx hardhat test stands up its own state (the usual case), this is a non-issue.

If test isolation isn't a concern here, LGTM.

@amir-deris

Copy link
Copy Markdown
Contributor Author

Verified the split preserves coverage — the six new scripts run exactly the same test files as the three they replace (EVM Module 4≡4, Interop 10≡10, dApp 3≡3), nothing dropped or duplicated. Wall-clock improvement looks real too.

One thing worth confirming before merge: the old scripts ran each group sequentially against a single chain, whereas the split now runs them on separate fresh clusters. So if any test depended on on-chain state left behind by a sibling that ran earlier in the same script — e.g. AssociateTest/SeiEndpoints after EVMCompatability, or the steak/nft reorder — the split would break it silently. If each npx hardhat test stands up its own state (the usual case), this is a non-issue.

If test isolation isn't a concern here, LGTM.

@bdchatham thanks for feedback. That is a valid concern but it seems that is not causing issue here. For dApp split: SteakTests.js and nftMarketplaceTests.js both set up fresh accounts and deploy contracts in their own before() hooks.

For EVMCompatabilityTest.js, it is isolated into
evm_compat_tests.sh, while EVMPrecompileTest.js + SeiEndpointsTest.js + AssociateTest.js stayed together in evm_precompile_tests.sh and no cross-cluster dependency introduced.

@masih masih enabled auto-merge June 11, 2026 08:33
@masih masih added this pull request to the merge queue Jun 11, 2026
Merged via the queue into main with commit 5e64bf3 Jun 11, 2026
60 checks passed
@masih masih deleted the amir/plt-447-split-evm-interoperability-test-matrix branch June 11, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants