ci(test): run module unittests and integration suites in parallel - #7754
ci(test): run module unittests and integration suites in parallel#7754MrLi000001 wants to merge 2 commits into
Conversation
The test workflow executed 16 module-unittest steps plus 10 integration suite steps fully serially - measured ~18-21 min of wall time even when the Build step only takes ~3 min with a warm ccache. - Merge the 16 MODULE_* steps into one 'ctest -j $(nproc) -R MODULE_' invocation (PERF_MODULE_HSOLVER_KERNELS still excluded). - Merge the 10 integration suite steps into one 'ctest -j 4' invocation. Each suite is an independent ctest entry running Autotest.sh in its own tests/<suite> directory, so they can run concurrently. -j4 is used instead of nproc to avoid extreme oversubscription, since every suite already spawns MPI ranks with OMP_NUM_THREADS=2. Test selection is unchanged (same -R/-E filters). Expected: the test portion drops from ~18-21 min to ~6-8 min of wall time.
…erial The first version attempted to run all 10 integration suites concurrently with ctest -j4, but this caused severe oversubscription on the X64 self-hosted runner (each suite spawns 4 MPI ranks with OMP_NUM_THREADS=2, so 4 concurrent suites = 32 threads on an 8-core runner). The test step took 91 min and failed with ctest exit code 8 (timeout). This version keeps the 16 MODULE_* steps merged into a single parallel ctest -j $(nproc) invocation (gtest binaries, no MPI oversubscription, measured ~4.5 min serial -> ~1 min parallel) but reverts the integration suite steps back to serial execution.
|
Parallelizing the module tests is a reasonable direction, and this step took 1m49s in the actual run. However, MODULE_ESOLVER now runs once under All Module Unittests and again under Other Unittests. Some MODULE tests also launch 2 or 4 MPI processes internally, so please add CTest PROCESSORS metadata or use a conservative fixed parallelism instead of nproc to avoid oversubscription. The stated 6–8-minute target was not reached; the full test section still took about 19 minutes. |
|
One blocking issue that hasn't been raised yet:
AddTest(TARGET MODULE_IO_read_exit_file_test ...) # serial entry
add_test(NAME MODULE_IO_read_exit_file_test_para_4
COMMAND mpirun -np 4 ./MODULE_IO_read_exit_file_test # same binary
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}) # same directoryTwo independent ctest entries, same executable, same CWD — and the test creates and Options: Two additions to @Stardust0831's points, both root causes rather than new findings:
Finally, the description says the 10 integration suite steps were merged into one Minor: with the steps merged, |
The test workflow executed 16 module-unittest steps plus 10 integration suite steps fully serially - measured ~18-21 min of wall time even when the Build step only takes ~3 min with a warm ccache.
Test selection is unchanged (same -R/-E filters). Expected: the test portion drops from ~18-21 min to ~6-8 min of wall time.
Reminder
AGENTS.mdanddocs/developers_guide/agent_governance.md.source/changes.Linked Issue
Fix #
Unit Tests and/or Case Tests for my changes
What's changed?
Governance Notes