CanDisassembleInlinableBenchmarks failed on arm64 platform when running with .NET 10

[filzrev]

On PR #3206,
CanDisassembleInlinableBenchmarks failed with following error.

CanDisassembleInlinableBenchmarks(jit: RyuJit, platform: Arm64, toolchain: .NET 10.0) [FAIL]
System.InvalidOperationException : Sequence contains no matching element
Stack Trace:
  at System.Linq.ThrowHelper.ThrowNoMatchException()
  at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source, Func`2 predicate)
     /_/tests/BenchmarkDotNet.IntegrationTests/DisassemblyDiagnoserTests.cs(166,0): at BenchmarkDotNet.IntegrationTests.DisassemblyDiagnoserTests.CanDisassembleInlinableBenchmarks(Jit jit, Platform platform, IToolchain toolchain)

It's occurred when using arm64 architecture and CsProjCoreToolchain.NetCoreApp10_0 toolchain is used.

disassemblyResult expected to contains following name.

BenchmarkDotNet.IntegrationTests.DisassemblyDiagnoserTests+WithInlineable.JustReturn()

But on arm64/net10.0 environment following result seems be returned.

BenchmarkDotNet.Autogenerated.Runnable_0.__ForDisassemblyDiagnoser__(Int32)

[filzrev]

On arm64 environment, benchmark method seems be inlined and following code path is used.

BenchmarkDotNet/src/BenchmarkDotNet/Disassemblers/ClrMdDisassembler.cs

Lines 97 to 98 in 5d5280a

	var filteredMethods = disassembledMethods.Length == 1
	? disassembledMethods // if there is only one method we want to return it (most probably benchmark got inlined)

I've added a commit 2fbd7ff

[filzrev]

Close this issue.
Because error is resolved by 2fbd7ff workaround code.

[timcassell]

On arm64 environment, benchmark method seems be inlined and following code path is used.

That should not happen. Benchmark methods are annotated with NoInlining. If it really did get inlined, then it's a JIT bug, but I suspect it actually failed to disassemble the method for some reason.

[filzrev]

That should not happen. Benchmark methods are annotated with NoInlining.

Target benchmark is not marked with [MethodImpl(MethodImplOptions.NoInlining)].

BenchmarkDotNet/tests/BenchmarkDotNet.IntegrationTests/DisassemblyDiagnoserTests.cs

Lines 151 to 154 in 5d5280a

	public class WithInlineable
	{
	[Benchmark] public void JustReturn() { }
	}

So I though it's possible to be inlined by JIT stage.
(On Windows(x64) environment it can't be reproduced with AggressiveInlining option though)

If it really did get inlined, then it's a JIT bug, but I suspect it actually failed to disassemble the method for some reason.

I'll try to get disassembly results on arm64 environment later.

[timcassell]

It's applied automatically by the weaver (and it strips AggressiveInlining).

[filzrev]

It's applied automatically by the weaver (and it strips AggressiveInlining).

When running IntegrationTests.
It use BuildPartition.ForcedNoDependenciesForIntegrationTests code path.

BenchmarkDotNet/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommand.cs

Lines 70 to 87 in 5d5280a

	// On our CI, Integration tests take too much time, because each benchmark run rebuilds BenchmarkDotNet itself.
	// To reduce the total duration of the CI workflows, we build all the projects without dependencies
	if (BuildPartition.ForcedNoDependenciesForIntegrationTests)
	{
	var restoreResult = await DotNetCliCommandExecutor.ExecuteAsync(
	WithArguments(GetRestoreCommand(GenerateResult.ArtifactsPaths, BuildPartition, FilePath, $"{Arguments} --no-dependencies", "restore-no-deps", excludeOutput: true)),
	cancellationToken).ConfigureAwait(false);
	if (!restoreResult.IsSuccess)
	return BuildResult.Failure(GenerateResult, restoreResult.AllInformation);
	var result = await DotNetCliCommandExecutor.ExecuteAsync(
	WithArguments(
	GetBuildCommand(GenerateResult.ArtifactsPaths, BuildPartition, FilePath, TargetFrameworkMoniker, $"{Arguments} --no-restore --no-dependencies", "build-no-restore-no-deps", excludeOutput: true)
	),
	cancellationToken).ConfigureAwait(false);
	return result.ToBuildResult(GenerateResult);
	}

It assembly weaver task is applied in this code path?

I've tried also normal project build on CI by disabling ForcedNoDependenciesForIntegrationTests.
But arm64 CI behavior seems not changed.

[timcassell]

Yes, the weaver is applied to regular project builds, not just benchmark builds (it's required for in process toolchains), which means it has already been applied for the no dependencies build. The benchmarks also validate the flags, so it will fail the run if it wasn't applied.

[filzrev]

Following results show exported disasm results on ubuntu-24.04-arm on CI.
2. Results of CsProjCoreToolchain.NetCoreApp80 results indicating it's unexpectedly inlined.

1. Results of InProcessEmitToolchain

(Job: Dry(Jit=RyuJit, Platform=Arm64, BuildConfiguration=Release, Toolchain=InProcessEmitToolchain, IterationCount=1, LaunchCount=1, RunStrategy=Monitoring, UnrollFactor=1))

; BenchmarkDotNet.IntegrationTests.DisassemblyDiagnoserTests+WithInlineable.JustReturn()
       stp       x29, x30, [sp, #-0x20]!
       mov       x29, sp
       str       x0, [x29, #0x18]
       ldp       x29, x30, [sp], #0x20
       ret       
; Total bytes of code 20

2. Results of CsProjCoreToolchain.NetCoreApp80 toolchain

.NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a (Job: Dry(Jit=RyuJit, Platform=Arm64, BuildConfiguration=Release, Toolchain=.NET 10.0, IterationCount=1, LaunchCount=1, RunStrategy=Monitoring, UnrollFactor=1))

; BenchmarkDotNet.Autogenerated.Runnable_0.__ForDisassemblyDiagnoser__(Int32)
       stp       x29, x30, [sp, #-0x20]!
       mov       x29, sp
       str       x0, [x29, #0x18]
       str       w1, [x29, #0x14]
;             if (notEleven == 11)
;             ^^^^^^^^^^^^^^^^^^^^
       ldr       w0, [x29, #0x14]
       cmp       w0, #0xb
       b.ne      M00_L00
;                     base.JustReturn();
;                     ^^^^^^^^^^^^^^^^^^
       ldr       x0, [x29, #0x18]
       movz      x1, #0x6dc0
       movk      x1, #0xe8a2, lsl #16
       movk      x1, #0xff0c, lsl #32
       ldr       x1, [x1]
       blr       x1
M00_L00:
       ldp       x29, x30, [sp], #0x20
       ret       
; Total bytes of code 60

[filzrev]

I've captured CI artifacts of ubuntu-24.04-arm

Modified settings

1. InProcessEmitToolchain is removed (It's required to avoid disasm artifact is overwritten by same path)
2. ForcedNoDependenciesForIntegrationTests is disabled.

Captured Artifacts

Run selected tests (ubuntu-24.04-arm --framework net10.0 --filter CanDisassembleInlinableBenchmarks)
https://github.com/filzrev/BenchmarkDotNet/actions/runs/30141095854

How to confirm artifacts

1. Download artifact ZIP file.
2. Open ZIP file with 7z (Windows Explorer don't support zip that contains long path)
3. Verify disassembler results and confirms it's inlined on arm64 results
4. Verify binary with Disassembler(e.g. dotPeek) and confirm `[MethodImpl(MethodImplOptions.NoInlining)] attribute is properly added by weaver as expected.

So it looks like MethodImplOptions.NoInlining attribute seems be ignored on arm64 JIT (ubuntu/macos)
It need to create minimum reproduceable sample, though it's hard without actual arm64 machine.

[filzrev]

I've tracked the reason why disassembledMethods don't contain JustReturn results.
And found following runtime.GetMethodByHandle API failed when using CsProjCoreToolchain10_0/arm64

BenchmarkDotNet/src/BenchmarkDotNet/Disassemblers/ClrMdDisassembler.cs

Line 306 in 5d5280a

var methodDescriptor = runtime.GetMethodByHandle(address);

Above code successfully return result when running with .NET 8.0/9.0 toolchains.

The cause is thought to be one of the following, but it is difficult to investigate the cause any further.

1. ClrMd issue.
2. Arm64Disassembler's address calculation relating issue (Including Capstone.NET side issue on arm64 platform that may be fixed in newer version)
3. .NET 10 JIT related issue.