Intermittent SIGSEGV in JIT'd code under parallel load on linux-arm64 — matches #127776; requesting release/10.0 backport of #127843

[nstojiljkovic]

Description

Since moving our .NET CI onto native linux-arm64 hosts we see intermittent SIGSEGV crashes of dotnet build, dotnet publish and xUnit testhost processes, strongly correlated with parallel load (multiple concurrent builds/test hosts on one machine). The crash signature matches #127776 (intermittent SIGSEGV in GetThreadLocalStaticBase on macOS arm64, thread-pool threads enumerating immutable collections): our captured crash has a managed thread faulting in dynamically generated (JIT) code with System.Collections.Immutable frames directly above it, on a thread-pool thread.

#127843 ("Fix a series of possible race conditions in thread static variable initialization", merged to main 2026-05-13) appears to address exactly this — missing VolatileStore publication in the TLS index machinery, which is precisely the class of bug a weakly-ordered arm64 machine exposes. We could not find a release/10.0 backport. We would like to request servicing of #127843 to release/10.0 — this affects production CI on current LTS-line runtimes.

Environment

• Hosts: NVIDIA DGX Spark (GB10): 20-core arm64, heterogeneous 10× Cortex-X925 + 10× Cortex-A725, 128 GB unified memory
• OS: Ubuntu 24.04 arm64 (DGX OS), Docker containers from mcr.microsoft.com/dotnet/sdk:10.0.300-noble
• Runtime in the captured crash: 10.0.826.23019 @Commit: 94ea82652cdd4e0f8046b5bd5becbd11461482ca (runtime 10.0.8)
• DOTNET_EnableWriteXorExecute=0 was already set (crashes occur regardless); server GC defaults
• Crashes reproduce on both of our GB10 machines; frequency scales with build/test concurrency (roughly 1 in 5–10 heavily parallel CI runs)

Evidence (createdump crash report, testhost.dll)

process_name: testhost.dll
runtime: 10.0.826.23019 @Commit: 94ea82652cdd4e0f8046b5bd5becbd11461482ca
arch: arm64, threads in dump: 47
parameters: {"ExceptionType": "0x20000000"}
crashed thread (native_id 0x3fb4, is_managed=true):
  #0  [dynamic/JIT code]                   native=0xe86352593138 module_base=0x0
  #1  System.Collections.Immutable.dll     native=0xe86355913c24 module_base=0xe863558d0000
  #2  System.Collections.Immutable.dll     native=0xe8635593f904 module_base=0xe863558d0000
  #3  System.Collections.Immutable.dll     native=0xe8635590e5f0 module_base=0xe863558d0000
  #4  System.Collections.Immutable.dll     native=0xe86355915614 module_base=0xe863558d0000
  #5  System.Linq.dll                      native=0xe863544fe204
  #6  System.Private.CoreLib.dll           native=0xe86352923c14
  #7  System.Collections.Immutable.dll     native=0xe86355931720
  #8  System.Collections.Immutable.dll     native=0xe8635592f580
  #9  Akka.Cluster.dll                     native=0xe86356d63eb0
  ... (Akka.Cluster / Akka frames continue)

The workload at crash time is Akka.NET cluster tests — cluster membership state is ImmutableSortedSet-based, i.e. heavy concurrent immutable-collection enumeration on thread-pool threads, the same pattern as #127776's Roslyn/MEF callers.

We also observe occasional non-fatal state corruption in the same suites on these hosts (System.ArgumentException: An item with the same key has already been added in code that is correct on x64 and on serial arm64 runs), consistent with the same publication race hitting non-faulting paths.

Full minidump (23 MB, DOTNET_DbgMiniDumpType=1) plus the complete crash-report JSON are available on request; we capture one for every occurrence now.

What we can offer

• The crash reproduces within hours under a 10-way parallel build/test loop on the GB10 hosts — we can validate a private/serviced build quickly.
• We are rolling out SDK 10.0.301 (runtime 10.0.9); based on the absence of a backport we expect no change and will confirm either way.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

[nstojiljkovic]

Backport PR opened: #130499 (clean cherry-pick of #127843 onto release/10.0, no conflicts). We are building a runtime from that branch on the reporting GB10 hardware to validate against our parallel-CI reproduction; results will be posted on the PR.

[nstojiljkovic]

Root-cause update: after further investigation, the crash reported here is actually the generic-dictionary layout publication race, not the thread-static race.

The captured dump's System.Collections.Immutable frames are the caller — the fault is one level down, in GenericsHelpers.GenericHandleWorker / ClassWithSlotAndModule (lazy generic-dictionary layout resolution), where m_pDictLayout was published without release semantics. That is fixed by #129763, which I've backported to release/10.0 as #130548, with a clean A/B on 2× GB10 (NVIDIA DGX Spark, arm64): unpatched 5 SIGSEGV / 166 runs, patched 0 / 165.

We also confirmed #127843 does not address this specific crash — an A/B built with #127843 still crashed identically. #127843 remains a valid fix for a distinct thread-static race (the backport #130499 is worthwhile on its own merits), but it is not the fix for the CI SIGSEGV described in this issue.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[VSadov]

@davidwrighton - should the corresponding fix be backported to net10?