Repository navigation
Description
Summary
A macOS x64 Helix run of Microsoft.ML.Tests segfaulted while NonDefaultColNamesBinaryClassificationOnnxConversionTest was training SymSGD. The captured native stack enters SymSGD's OpenMP parallel region and crosses from libomp.dylib into libiomp5.dylib's __kmp_suspend_64. This is consistent with the Helix payload loading two images of the same OpenMP library under different names. The stack is strong evidence of the interaction, but the precise dyld symbol-binding mechanism and a product-level reproducer still need confirmation.
Reproducible CI evidence
- Repository: dotnet/machinelearning
- Azure Pipelines build: MachineLearning-CI build 1690596 (20261119.12)
- Source: 2a5763c
- Leg/queue:
MacOS_x64 Debug_Build/osx.15.amd64.open - OS: macOS 15.7
- Helix console log
- Crash report JSON
The test was identified from the crash report's thread stacks, not just from the last Starting test: line in the console (xUnit runs tests concurrently).
Crash evidence
The crash report marks native thread 0xc1091 as crashing. Its relevant frames are:
libiomp5.dylib!__kmp_suspend_64<false, true>
libomp.dylib!kmp_flag_64<false, true>::wait
libomp.dylib!__kmp_hyper_barrier_release
libomp.dylib!__kmp_fork_barrier
libomp.dylib!__kmp_launch_thread
libomp.dylib!__kmp_launch_worker
Another captured thread shows the application path entering this OpenMP region:
libiomp5.dylib!__kmp_suspend_64<false, true>
libomp.dylib!kmp_flag_64<false, true>::wait
libomp.dylib!__kmp_hyper_barrier_gather
libomp.dylib!__kmp_join_barrier
libomp.dylib!__kmpc_fork_call
libSymSgdNative.dylib!InitializeState
libSymSgdNative.dylib!LearnAll
SymbolicSgdLogisticRegressionBinaryTrainer.TrainCore
OnnxConversionTest.TestPipeline
Microsoft.ML.Tests.OnnxConversionTest.NonDefaultColNamesBinaryClassificationOnnxConversionTest
The dump lists the two libraries as separate loaded images at bases 0x1a9c38000 (libiomp5.dylib) and 0x1abe17000 (libomp.dylib).
Relevant runtime setup
At the source commit above:
eng/helix.projsetsKMP_DUPLICATE_LIB_OK=TRUE, copies Homebrew'slibomp.dylibinto the test payload, and on macOS x64 copies that file again aslibiomp5.dylib.SymSgdNative/CMakeLists.txtlinks SymSGD against Homebrew'slibompon macOS.- The loaded
libiomp5.dylibname is needed by MKL'slibMklImports.dylib.
This makes the duplicate-runtime setup a plausible cause, but it does not by itself prove why dyld bound this particular call across the two images.
Requested investigation
- Reproduce or otherwise confirm the crash and determine whether loading the same OpenMP binary under both
libomp.dylibandlibiomp5.dylibcauses the cross-image call/state mismatch. - Adjust the macOS x64 Helix native-library setup so MKL, SymSGD, and other native dependencies use a compatible single OpenMP runtime, without changing Linux's separate fix in Use MKL's libiomp5 for SymSgdNative OpenMP on Linux x64 #7927.
- Consider improving
eng/testing/runTests.sh: when a test process exits due to a native crash, print the crashing thread's native frames from the generated crash-report JSON to the console. This would make future failures easier to identify and provide a stable Build Analysis signature even when Helix artifacts are no longer available.
Related context: #7061, #7734, and #5569. #7927 addresses a Linux x64 OpenMP-linking issue and does not change macOS.
Note
This issue draft was prepared with GitHub Copilot; the root-cause mechanism above is a hypothesis to verify.
Activity
MihuBot commentedon Nov 20, 2026
I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):