diff --git a/docs/design/coreclr/jit/runtime-async-inlining.md b/docs/design/coreclr/jit/runtime-async-inlining.md new file mode 100644 index 00000000000000..9d8c84ea6fc6a0 --- /dev/null +++ b/docs/design/coreclr/jit/runtime-async-inlining.md @@ -0,0 +1,348 @@ +# General runtime async inlining + +This document outlines some challenges and thoughts behind general inlining of runtime async calls. + +## Background + +Runtime async comes with several user-visible behaviors that must be preserved when inlining. +These can be categorized in behaviors attached to the call site and behaviors attached to the callee. + +### Await/call site behaviors + +Custom awaiters and synchronous Task/ValueTask-returning functions can be inlined today, so below we consider only runtime async backed implementations. + +- Every suspending await saves/restores the `ExecutionContext`. +Concretely this is what saves and restores the values of `AsyncLocal` if suspension/resumption happens around an await. +The behavior of this cannot be configured per-await, but it can be configured globally via `ExecutionContext.SuppressFlow()`. +For our purposes we can consider this to always require saving/restoration. +- For suspending `Task` and `ValueTask` awaits the resumption behavior is configurable: + + By default, the current `SynchronizationContext` or `TaskScheduler` is captured and the continuation is posted to it. + + If `ConfigureAwait(false)` is applied the continuation is run on the thread pool. + + The "location to continue", either a `SynchronizationContext`, `TaskScheduler` or the thread pool, is collectively called the _continuation context_ in this document. + +For awaits that finish synchronously (i.e. do not suspend) no additional work happens at the call site. + +To accomplish the above the JIT works together with async infrastructure: +- Suspension generates code to obtain and save the `ExecutionContext` into the `Continuation` object. +On resumption, async infrastructure restores the `ExecutionContext` before calling back into JIT'ed code. +- Suspension generates code to capture continuation context information into the `Continuation` object. +On resumption, async infrastructure ensures that the continuation runs in the captured continuation context. + +The above are accomplished with several helpers returned via the `getAsyncInfo` JIT-EE helper: +- `AsyncHelpers.CaptureExecutionContext` +- `AsyncHelpers.CaptureContinuationContext` +- `AsyncHelpers.FinishSuspensionNoContinuationContext` (optimized case) +- `AsyncHelpers.FinishSuspensionWithContinuationContext` (optimized case) + +### Function/callee behaviors +Every runtime async function body is wrapped with save and restore of the async contexts on the current Thread object. +This restore happens only when the function finishes synchronously. +Concretely a runtime async function ends up looking like: +```csharp +ExecutionContext execContext; +SynchronizationContext syncContext; +AsyncHelpers.CaptureContexts(out execContext, out syncContext); +try +{ + // actual user code +} +finally +{ + AsyncHelpers.RestoreContexts(resumed, execContext, syncContext); +} +``` + +Note the `resumed` argument, used to check if the runtime async function finished synchronously or not. +If the runtime async function finished after resuming then contexts are not restored. + +Additionally, every suspension _also_ restores the contexts. +This is done by inserting a similar call +```csharp +AsyncHelpers.RestoreContextsOnSuspension(resumed, execContext, syncContext) +``` +in suspensions. +This is folded into the `AsyncHelpers.FinishSuspensionNoContinuationContext` and `AsyncHelpers.FinishSuspensionWithContinuationContext` helpers. +The JIT uses those helpers whenever possible. + +## Inlining runtime async calls +When inlining a runtime async call we remove both a call site and the callee. +This removes the user-visible behaviors discussed above. +To inline correctly we need to ensure the JIT preserves these behaviors. + +### Suspending inside an inlinee + +Consider a scenario like: + +```csharp +async Task A() +{ + ... + await B(); + ... +} + +async Task B() +{ + ... + await C(); + ... +} + +async Task C() +{ + ... + await Task.Yield(); + ... +} +``` +Now assume that we have available bools `resumed_A`, `resumed_B`, `resumed_C` that represent whether `A`, `B` and `C` are currently running after being resumed (true), or whether they are running because they were started and have never suspended before (false). +Note that `resumed_C` and `resumed_B` can switch between true/false multiple times during the execution if `B` and `C` are in loops, while `resumed_A` will switch to `true` and then stay `true`. + +Assume we inlined `B()` and `C()` into `A`, and that we are logically suspending on `Task.Yield()` in `C`. +What does this suspension capture? That depends on `resumed_B` and `resumed_C`. +- Let's first consider `!resumed_C`. +This means that we just started running `C`, and on suspension we would restore `ExecutionContext` and `SynchronizationContext` from the beginning of `C` into the `Thread` object. +If we hadn't inlined then `B`'s frame would be physically present on the stack due to `!resumed_C`, so after returning it would create and link its own continuation. +This continuation would capture the `ExecutionContext` restored by `C` and also capture continuation context information based on the `SynchronizationContext` restored by `C`. +- If `resumed_C` then `B`'s frame would not be physically present on the stack. +Rather, the existing `B` continuation would be linked to the newly created `C` continuation by async infrastructure. +It would maintain its values from the time we suspended in `C` with `!resumed_C`. + +The take away is that the suspension for the inlined `Task.Yield()` needs to roughly accomplish the following, in addition to storing its normal state: +```csharp +continuation = AllocOrReuseContinuation(); +CaptureContinuationContext(continuation, ref continuation.ContinuationContext, ref continuation.Flags); // In the general case, but not for Task.Yield() +CaptureExecutionContext(continuation, ref continuation.ExecutionContext); +if (!resumed_C) +{ + RestoreContextsOnSuspension(false, execContext_C, syncContext_C); + // Logical return to B and linking in B's continuation + CaptureContinuationContext(continuation, ref continuation.ContinuationContextForB, ref continuation.FlagsForB) + CaptureExecutionContext(continuation, ref continuation.ExecutionContextForB); + if (!resumed_B) + { + RestoreContextsOnSuspension(false, execContext_B, syncContext_B); + // Logical return to A and linking in A's continuation + CaptureContinuationContext(continuation, ref continuation.ContinuationContextForA, ref continuation.FlagsForA) + CaptureExecutionContext(continuation, ref continuation.ExecutionContextForA); + + RestoreContextsOnSuspension(resumed_A, execContext_A, syncContext_A); + } + else + { + // keep continuation.ContinuationContextForA, FlagsForA, ExecutionContextForA + } +} +else +{ + // keep continuation.ContinuationContextForB, FlagsForB, ExecutionContextForB +} +``` +Note also that we have the implications `resumed_C` implies `resumed_B` implies `resumed_A` simplifying the `else` cases here. +The else cases also rely on `continuation.ContinuationContextForB/A` staying assigned in the continuations, which is the case only because the JIT allocates a single continuation and reuses it throughout the execution of `A`. + +This may look like a large amount of code for every suspension inside `C`, but sharing of tails is possible since all the inlined suspensions end with the same code that does not change per suspension site. + +### Resuming inside an inlinee + +Let's imagine we resume at the inlined `Task.Yield()` in `C`. +Initially such a resumption progresses as normal: async infrastructure restores the `ExecutionContext` that was saved. +If this was a task await (which it isn't, but often will be), the async infrastructure ensures it is running in the right continuation context. +Then it resumes in the code. + +When `C` logically returns to `B` we need to handle what the async infrastructure would have handled for us. +Call this the _post-inline IR_. +- We need to restore the `ExecutionContext` that we captured for the suspension at `C()`. +Logically, we need to do something like +```csharp +if (resumed_C) + AsyncHelpers.RestoreExecutionContext(Thread.CurrentThreadAssumedInitialized, continuation.ExecutionContextForB); +``` +- We need to ensure we're running on the proper continuation context. +The JIT cannot accomplish this on its own; there is no way to directly set up the environment without posting a callback. +Hence, in the case where we are not running in the right context we need help from async infrastructure to proceed with a suspension+switch+resumption: +```csharp +if (resumed_C && !AsyncHelpers.IsOnRightContext(continuation.ContinuationContextForB, continuation.FlagsForB)) + await AsyncHelpers.SwitchContext(continuation.ContinuationContextForB, continuation.FlagsForB); +``` + +We expect `IsOnRightContext` to be true almost always, since it is very rare that the current synchronization context is modified, especially during the execution of async methods. +If this was a common case then it is unlikely that inlining would ever be profitable. + +The implementation expands only the `resumed_C` check as JIT IR and leaves the rest as a single +`AsyncHelpers.RestoreInlinedFrameContexts` call, which does the `ExecutionContext` restore, the +`IsOnRightContext` check and the suspension itself internally. +The synchronous case is the one worth optimizing for, and anything past the `resumed_C` check has +already paid for at least one suspension and resumption. + +### Handling synchronous saves and restores of contexts + +Recall that every runtime async function's body was wrapped with a save/restore of the contexts. +Given `resumed_A`, `resumed_B` and `resumed_C` it is not hard to insert these for every inlinee. +For example, when inlining `C` we insert code similar to: + +```csharp +ExecutionContext execContext; +SynchronizationContext syncContext; +AsyncHelpers.CaptureContexts(out execContext, out syncContext); +try +{ + // actual user code +} +finally +{ + AsyncHelpers.RestoreContexts(resumed_C, execContext, syncContext); +} +``` + +### Special cases + +The only case supported today is when we know the inlinee never suspends, by virtue of having no awaits at all. +This simplifies the cases above since `resumed_B` and `resumed_C` are always false, and all suspension points generated belong to `A`. + +### Computing `resumed_A`, `resumed_B`, `resumed_C` ... + +Today we compute the `resumed` value only for the top-level async function, since it is computed based on the async continuation parameter. +With inlinees this strategy no longer works. +Instead insert IR to compute these values at several points: +1. When starting a function `F` (whether it be the top function or start of an inlined function), assign `resumed_F = false` +2. When resuming `F` at one of its resumption points assign `resumed_F = true` +3. When logically returning from `F` to its caller, inherit the resumption status: `resumed_caller |= resumed_F`. + +Note that these are introduced at different points. (1) and (3) will likely be inserted during `PHASE_SAVE_ASYNC_CONTEXTS` while (2) will be inserted in `PHASE_ASYNC`. + +(2) is a store to `resumed_F` inserted very late. +Until the async transformation we will model this as a `LCL_ADDR` well-known argument added to all async calls. + +It is important that the JIT can reason about `resumed` booleans to be able to eliminate the cruft for inlinees that are proven to never suspend. +We would like constant propagation to realize `resumed_C` is always false if `C` has no suspension point, for example. + +### Representation of post-inline IR + +The post-inline IR we need to insert when `C` returns to `B` looks like: + +```csharp +if (resumed_C) +{ + AsyncHelpers.RestoreExecutionContext(Thread.CurrentThreadAssumedInitialized, continuation.ExecutionContextForB); + + if (!AsyncHelpers.IsOnRightContext(continuation.ContinuationContextForB, continuation.FlagsForB)) + { + await AsyncHelpers.SwitchContext(continuation.ContinuationContextForB, continuation.FlagsForB); + } + + resumed_B = true; +} +``` + +It is not clear how to represent `continuation.ExecutionContextForB`, `continuation.ContinuationContextForB` and `continuation.FlagsForB`. +This IR is likely something to insert as part of inlining, yet at that point we have not laid out the continuation yet. +To solve this we will introduce a `GT_CONTINUATION_MEMBER_OFFSET` node that represents the member offset and that is replaced by a constant as part of the async transformation. + +### Exceptions + +Inlined functions can finish by throwing exceptions too. +Under normal circumstances the async infrastructures catches the exception and looks for the next continuation that may handle the exception. +Once located, the same resumption mechanism applies: the `ExecutionContext` is restored and the proper continuation context is ensured. + +Correct handling requires catching the possible exception and then rethrowing it in the post-inline IR. +The expansion for calls in try clauses would look roughly like: +```csharp +try +{ + // user code for C +} +catch (throwable ex) when (resumed_C) +{ + throwable_C = ex; +} + +if (resumed_C) +{ + AsyncHelpers.RestoreExecutionContext(Thread.CurrentThreadAssumedInitialized, continuation.ExecutionContextForB); + + if (!AsyncHelpers.IsOnRightContext(continuation.ContinuationContextForB, continuation.FlagsForB)) + { + await AsyncHelpers.SwitchContext(continuation.ContinuationContextForB, continuation.FlagsForB); + } + + resumed_B = true; + + if (throwable_C != null) + ThrowExact(throwable_C); +} +``` + +Initially we will skip inlining async calls that may suspend in try clauses. + +## Some examples + +In the following assume the presence of two synchronization contexts `_syncContext1` and `_syncContext2` that set up the global environment in some distinguishable way. + +```csharp +public static async Task Foo() +{ + SynchronizationContext.SetSynchronizationContext(_syncContext1); + await Bar(); +} + +private static async Task Bar() +{ + SynchronizationContext.SetSynchronizationContext(_syncContext2); + await Baz(); +} + +private static async Task Baz() +{ + await new AlwaysThreadPoolAwaitable(); +} + +private struct AlwaysThreadPoolAwaitable : INotifyCompletion +{ + public bool IsCompleted => false; + public void OnCompleted(Action continuation) + { + ThreadPool.QueueUserWorkItem(_ => { Thread.Sleep(100); continuation(); }); + } + public void GetResult() { } + + public AlwaysThreadPoolAwaitable GetAwaiter() => this; +} +``` + +- `Foo()` runs in the ambient synchronization context but switches its own synchronization context before awaiting `Bar()`. Its continuation captures `_syncContext1` as the continuation context. +- `Bar()` also switches its own synchronization context before awaiting `Baz()`. Its continuation captures `_syncContext2` as the continuation context. +- `Baz()` always suspends, switching onto a thread pool thread. No continuation context is captured for custom awaitables. +- On resumption, async infrastructure runs `Baz` directly in the context of `AlwaysThreadPoolAwaitable` +- Next, async infrastructure switches to `_syncContext2` to run `Bar`'s continuation +- Next, async infrastructure switches to `_syncContext1` to run `Foo`'s continuation + +Inlining in this case would need to suspend+switch+resume in both the `IsOnRightContext` checks. + +```csharp +private static AsyncLocal s_local = new(); +public static async Task Foo() +{ + s_local.Value = 1; + await Bar(); + Console.WriteLine(s_local.Value); +} + +private static async Task Bar() +{ + s_local.Value = 2; + await Baz(); + Console.WriteLine(s_local.Value); +} + +private static async Task Baz() +{ + s_local.Value = 3; + await new AlwaysThreadPoolAwaitable(); + Console.WriteLine(s_local.Value); +} +``` + +Similar to above, except this time we do not need a full bail out; rather, the post-inline handling will restore the `ExecutionContext` to make sure the value of `s_local` is properly restored. +Proper output is 3,2,1 and relies on `ExecutionContext`s having been restored by the post-inline handling. diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs index fd3e75881619ff..ad89e7be28fe2e 100644 --- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs +++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs @@ -223,6 +223,8 @@ private ref struct RuntimeAsyncStackState public INotifyCompletion? Notifier; public ValueTaskSourceContinuation? ValueTaskSourceContinuation; public RuntimeAsyncTaskContinuation? TaskContinuation; + public delegate* AwaiterContinuation; + public int AwaiterOffset; // When we suspend in the leaf, the contexts are captured into these fields. public ExecutionContext? LeafExecutionContext; @@ -265,6 +267,8 @@ public void CaptureContexts() [NonVersionable] public void Push(RuntimeAsyncStackState* stackState) { + stackState->AwaiterContinuation = null; + stackState->AwaiterOffset = 0; stackState->Next = StackState; StackState = stackState; CurrentThread ??= Thread.CurrentThread; @@ -805,7 +809,15 @@ internal unsafe bool HandleSuspended(ref RuntimeAsyncAwaitState state) try { - if (stackState->CriticalNotifier is { } critNotifier) + if (stackState->AwaiterContinuation != null) + { + // The awaiter is stored in the continuation for the caller of + // AwaitAwaiterInContinuation or UnsafeAwaitAwaiterInContinuation. + Debug.Assert((headContinuation.Flags & ContinuationFlags.AllContinuationFlags) == 0); + stackState->AwaiterContinuation( + headContinuation, stackState->AwaiterOffset, GetContinuationAction()); + } + else if (stackState->CriticalNotifier is { } critNotifier) { // Result of async call to AwaitAwaiter or UnsafeAwaitAwaiter. // These never have special continuation context handling. @@ -1477,6 +1489,131 @@ private static void CaptureContinuationContextFlags(ref ContinuationFlags flags, flags |= ContinuationFlags.ContinueOnThreadPool; } + // Restore the contexts that an inlined async frame captured when it logically returned to + // its caller, after that frame was resumed inside its own body. + // + // Used by the JIT when inlining runtime async calls. The JIT emits the check of whether + // the frame was resumed at all and calls this when it was; everything the async + // infrastructure would otherwise have done at that frame boundary happens here. + // + // Unlike the synchronous restore at the end of a method, the ExecutionContext restore runs + // only after a resumption, so it must target the thread we were resumed on rather than the + // one whose contexts were captured on entry. + // + // The continuation context check determines whether resuming the caller's continuation here + // would be dispatched inline. It mirrors the "can inline" conditions in + // RuntimeAsyncTaskContinuation.QueueIfNecessary; the two must be kept in sync. + // + // 'flags' must contain only ContinuationFlags.AllContinuationFlags bits. + [BypassReadyToRun] + [MethodImpl(MethodImplOptions.Async)] + private static void RestoreInlinedFrameContexts(ExecutionContext? previousExecCtx, object? continuationContext, ContinuationFlags flags) + { + Debug.Assert((flags & ~ContinuationFlags.AllContinuationFlags) == 0); + + // We are inside a runtime async chain, so the thread has already been cached. Use it + // instead of Thread.CurrentThreadAssumedInitialized to keep this to one TLS lookup. + ref RuntimeAsyncAwaitState state = ref t_runtimeAsyncAwaitState; + Thread? currentThread = state.CurrentThread; + Debug.Assert(currentThread != null); + + RestoreExecutionContext(currentThread, previousExecCtx); + + if ((flags & ContinuationFlags.ContinueOnThreadPool) != 0) + { + SynchronizationContext? syncCtx = currentThread._synchronizationContext; + if (syncCtx is null || syncCtx.GetType() == typeof(SynchronizationContext)) + { + TaskScheduler? sched = TaskScheduler.InternalCurrent; + if (sched is null || sched == TaskScheduler.Default) + { + return; + } + } + } + else if ((flags & ContinuationFlags.ContinueOnCapturedSynchronizationContext) != 0) + { + Debug.Assert(continuationContext is SynchronizationContext); + if (continuationContext == currentThread._synchronizationContext) + { + return; + } + } + else if ((flags & ContinuationFlags.ContinueOnCapturedTaskScheduler) != 0) + { + Debug.Assert(continuationContext is TaskScheduler); + if (continuationContext == TaskScheduler.InternalCurrent) + { + return; + } + } + else + { + // No continuation context was captured, so there is nothing to switch to. + return; + } + + TailAwait(); + SwitchToContinuationContext(ref state, continuationContext, flags); + } + + // Suspend and resume in the specified continuation context. + // + // Suspending on an already completed task makes the dispatcher re-dispatch the continuation + // immediately. Because that dispatch happens with canInline: false, it always posts or + // schedules onto the requested context rather than running inline here, which is what we + // want -- we only get here when we are known to be on the wrong context. + [BypassReadyToRun] + [MethodImpl(MethodImplOptions.Async)] + private static unsafe void SwitchToContinuationContext(ref RuntimeAsyncAwaitState state, object? continuationContext, ContinuationFlags flags) + { + Continuation? sentinelContinuation = state.SentinelContinuation ??= new Continuation(); + + RuntimeAsyncTaskContinuation? taskCont = state.CachedTaskContinuation; + if (taskCont != null) + { + state.CachedTaskContinuation = null; + } + else + { + taskCont = new RuntimeAsyncTaskContinuation(); + } + + taskCont.Initialize(Task.CompletedTask); + taskCont.ContinuationContext = continuationContext; + taskCont.Flags |= flags; + + sentinelContinuation.Next = taskCont; + state.StackState->TaskContinuation = taskCont; + + state.CaptureContexts(); + AsyncSuspend(taskCont); + } + + // Capture the contexts an inlined async frame hands to its caller when it logically + // returns during a suspension, i.e. what the caller's continuation would have captured + // had the callee's frame been physically present. + // + // No-ops when the frame has already resumed: in that case the caller's continuation + // already exists and keeps the values it captured when the frame first suspended. + // + // The suspension walks the inlined frames outward, and a frame having resumed implies + // its caller has too, so these can be emitted as a straight line: once one frame has + // resumed, this and every subsequent restore for the frames outside it no-op. + private static void CaptureInlinedFrameTransition(bool resumed, + ref object? continuationContext, + ref ContinuationFlags flags, + ref ExecutionContext? execContext) + { + if (resumed) + { + return; + } + + CaptureContinuationContext(ref continuationContext, ref flags); + execContext = CaptureExecutionContext(); + } + // Finish suspension in the common case of a custom await or for a ConfigureAwait(false) task await: // - Capture current ExecutionContext into the continuation // - Restore ExecutionContext and SynchronizationContext to the current Thread object diff --git a/src/coreclr/inc/corinfo.h b/src/coreclr/inc/corinfo.h index 4c39bed1aea091..c92f38cf269b10 100644 --- a/src/coreclr/inc/corinfo.h +++ b/src/coreclr/inc/corinfo.h @@ -1836,6 +1836,12 @@ struct CORINFO_ASYNC_INFO CORINFO_METHOD_HANDLE restoreContextsMethHnd; // Method handle for AsyncHelpers.RestoreContextsOnSuspension, used before suspending in async methods CORINFO_METHOD_HANDLE restoreContextsOnSuspensionMethHnd; + // Method handle for AsyncHelpers.RestoreInlinedFrameContexts, used when an inlined + // async callee logically returns to its caller after having been resumed + CORINFO_METHOD_HANDLE restoreInlinedFrameContextsMethHnd; + // Method handle for AsyncHelpers.CaptureInlinedFrameTransition, used on suspension to capture + // the contexts each inlined async frame hands to its caller + CORINFO_METHOD_HANDLE captureInlinedFrameTransitionMethHnd; // Finish suspension without saving continuation context (i.e. custom awaiter or ConfigureAwait(false)) CORINFO_METHOD_HANDLE finishSuspensionNoContinuationContextMethHnd; // Finish suspension with saving continuation context (i.e. normal task await) @@ -3168,6 +3174,14 @@ class ICorStaticInfo // instantiation argument that must be passed to the await call. virtual CORINFO_METHOD_HANDLE getAwaitReturnCall(CORINFO_METHOD_HANDLE callerHandle, CORINFO_CONTEXT_HANDLE* contextHandle, CORINFO_LOOKUP* instArg) = 0; + virtual CORINFO_METHOD_HANDLE getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg + ) = 0; + /*********************************************************************************/ // // Diagnostic methods diff --git a/src/coreclr/inc/icorjitinfoimpl_generated.h b/src/coreclr/inc/icorjitinfoimpl_generated.h index 21dfb633498260..aae0a091f92096 100644 --- a/src/coreclr/inc/icorjitinfoimpl_generated.h +++ b/src/coreclr/inc/icorjitinfoimpl_generated.h @@ -506,6 +506,13 @@ CORINFO_METHOD_HANDLE getAwaitReturnCall( CORINFO_CONTEXT_HANDLE* contextHandle, CORINFO_LOOKUP* instArg) override; +CORINFO_METHOD_HANDLE getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) override; + mdMethodDef getMethodDefFromMethod( CORINFO_METHOD_HANDLE hMethod) override; diff --git a/src/coreclr/inc/jiteeversionguid.h b/src/coreclr/inc/jiteeversionguid.h index ed763baa880121..916805184634bb 100644 --- a/src/coreclr/inc/jiteeversionguid.h +++ b/src/coreclr/inc/jiteeversionguid.h @@ -37,11 +37,11 @@ #include -constexpr GUID JITEEVersionIdentifier = { /* 134b051a-e1ec-4f52-a1bf-f919908aa33c */ - 0x134b051a, - 0xe1ec, - 0x4f52, - {0xa1, 0xbf, 0xf9, 0x19, 0x90, 0x8a, 0xa3, 0x3c} +constexpr GUID JITEEVersionIdentifier = { /* a41245a2-74c0-47ac-87d7-9375cb833125 */ + 0xa41245a2, + 0x74c0, + 0x47ac, + {0x87, 0xd7, 0x93, 0x75, 0xcb, 0x83, 0x31, 0x25} }; #endif // JIT_EE_VERSIONING_GUID_H diff --git a/src/coreclr/jit/ICorJitInfo_names_generated.h b/src/coreclr/jit/ICorJitInfo_names_generated.h index fe8fa377332a9a..f3265a45960177 100644 --- a/src/coreclr/jit/ICorJitInfo_names_generated.h +++ b/src/coreclr/jit/ICorJitInfo_names_generated.h @@ -126,6 +126,7 @@ DEF_CLR_API(runWithSPMIErrorTrap) DEF_CLR_API(getEEInfo) DEF_CLR_API(getAsyncInfo) DEF_CLR_API(getAwaitReturnCall) +DEF_CLR_API(getAwaitAwaiterInContinuationCall) DEF_CLR_API(getMethodDefFromMethod) DEF_CLR_API(printMethodName) DEF_CLR_API(getMethodNameFromMetadata) diff --git a/src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp b/src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp index 5291f7a8c8d287..8c08675dc5e941 100644 --- a/src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp +++ b/src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp @@ -1199,6 +1199,19 @@ CORINFO_METHOD_HANDLE WrapICorJitInfo::getAwaitReturnCall( return temp; } +CORINFO_METHOD_HANDLE WrapICorJitInfo::getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) +{ + API_ENTER(getAwaitAwaiterInContinuationCall); + CORINFO_METHOD_HANDLE temp = wrapHnd->getAwaitAwaiterInContinuationCall(callerHandle, callSig, isUnsafe, contextHandle, instArg); + API_LEAVE(getAwaitAwaiterInContinuationCall); + return temp; +} + mdMethodDef WrapICorJitInfo::getMethodDefFromMethod( CORINFO_METHOD_HANDLE hMethod) { diff --git a/src/coreclr/jit/async.cpp b/src/coreclr/jit/async.cpp index 6a47dcacc6a77b..c6c21d06f83f60 100644 --- a/src/coreclr/jit/async.cpp +++ b/src/coreclr/jit/async.cpp @@ -45,6 +45,191 @@ #include "jitstd/algorithm.h" #include "async.h" +ContinuationMember ContinuationMember::CustomAwaiterOfLayout(ClassLayout* layout) +{ + ContinuationMember member; + member.Type = ContinuationMemberType::CustomAwaiterOfLayout; + member.m_customAwaiterLayout = layout; + return member; +} + +ContinuationMember ContinuationMember::InlineFrameExecutionContext(unsigned inlineDepth) +{ + ContinuationMember member; + member.Type = ContinuationMemberType::InlineFrameExecutionContext; + member.m_inlineDepth = inlineDepth; + return member; +} + +ContinuationMember ContinuationMember::InlineFrameContinuationContext(unsigned inlineDepth) +{ + ContinuationMember member; + member.Type = ContinuationMemberType::InlineFrameContinuationContext; + member.m_inlineDepth = inlineDepth; + return member; +} + +ContinuationMember ContinuationMember::InlineFrameFlags(unsigned inlineDepth) +{ + ContinuationMember member; + member.Type = ContinuationMemberType::InlineFrameFlags; + member.m_inlineDepth = inlineDepth; + return member; +} + +ClassLayout* ContinuationMember::GetCustomAwaiterLayout() const +{ + assert(Type == ContinuationMemberType::CustomAwaiterOfLayout); + return m_customAwaiterLayout; +} + +unsigned ContinuationMember::GetInlineDepth() const +{ + assert((Type == ContinuationMemberType::InlineFrameExecutionContext) || + (Type == ContinuationMemberType::InlineFrameContinuationContext) || + (Type == ContinuationMemberType::InlineFrameFlags)); + return m_inlineDepth; +} + +//------------------------------------------------------------------------ +// ContinuationMember::GetStorageType: +// Get the type of the storage this member occupies in the continuation. +// +// Returns: +// TYP_STRUCT for custom awaiters (described by GetCustomAwaiterLayout), otherwise +// the primitive type stored. +// +var_types ContinuationMember::GetStorageType() const +{ + switch (Type) + { + case ContinuationMemberType::CustomAwaiterOfLayout: + return TYP_STRUCT; + case ContinuationMemberType::InlineFrameExecutionContext: + case ContinuationMemberType::InlineFrameContinuationContext: + return TYP_REF; + case ContinuationMemberType::InlineFrameFlags: + return TYP_INT; + default: + unreached(); + } +} + +bool ContinuationMember::AreCompatible(const ContinuationMember& a, const ContinuationMember& b) +{ + if (a.Type != b.Type) + { + return false; + } + + switch (a.Type) + { + case ContinuationMemberType::CustomAwaiterOfLayout: + return ClassLayout::AreCompatible(a.m_customAwaiterLayout, b.m_customAwaiterLayout); + case ContinuationMemberType::InlineFrameExecutionContext: + case ContinuationMemberType::InlineFrameContinuationContext: + case ContinuationMemberType::InlineFrameFlags: + // Keyed by inline depth rather than by individual inline frame. Two frames at + // the same depth can share storage because their live ranges cannot overlap: + // the value is written at a frame's first suspension and consumed by that same + // frame's post-inline IR, and one frame at a given depth must be left before + // another can be entered. + return a.m_inlineDepth == b.m_inlineDepth; + default: + unreached(); + } +} + +#ifdef DEBUG +void ContinuationMember::Print() const +{ + switch (Type) + { + case ContinuationMemberType::CustomAwaiterOfLayout: + printf("CustomAwaiter<%s>", m_customAwaiterLayout->GetClassName()); + break; + case ContinuationMemberType::InlineFrameExecutionContext: + printf("ExecutionContext for inline depth %u", m_inlineDepth); + break; + case ContinuationMemberType::InlineFrameContinuationContext: + printf("Continuation context for inline depth %u", m_inlineDepth); + break; + case ContinuationMemberType::InlineFrameFlags: + printf("Continuation flags for inline depth %u", m_inlineDepth); + break; + default: + unreached(); + } +} +#endif + +//------------------------------------------------------------------------ +// CollectWellKnownArgs: +// Collect the nodes of all arguments of a call with a specific well-known kind, in +// argument order. +// +// Parameters: +// call - The call +// wka - The well-known argument kind +// nodes - [out] Stack to push the argument nodes onto +// +// Remarks: +// Async context pseudo-args can appear multiple times on a call: general async +// inlining adds one set per enclosing inlined frame. +// +static void CollectWellKnownArgs(GenTreeCall* call, WellKnownArg wka, ArrayStack& nodes) +{ + for (CallArg& arg : call->gtArgs.Args()) + { + if (arg.GetWellKnownArg() == wka) + { + nodes.Push(arg.GetNode()); + } + } +} + +size_t Compiler::GetContinuationMemberIndex(const ContinuationMember& member) +{ + // Members describe the root method's continuation, so they are always registered + // there: an inlinee's IR ends up in the root and its member offset nodes are + // resolved against the root's layout. + Compiler* const root = impInlineRoot(); + + if (root->m_asyncContinuationMembers == nullptr) + { + root->m_asyncContinuationMembers = + new (root, CMK_Async) jitstd::vector(root->getAllocator(CMK_Async)); + } + else + { + for (size_t i = 0; i < root->m_asyncContinuationMembers->size(); i++) + { + const ContinuationMember& existingMember = root->m_asyncContinuationMembers->at(i); + + if (ContinuationMember::AreCompatible(member, existingMember)) + { + return i; + } + } + } + + root->m_asyncContinuationMembers->push_back(member); + return root->m_asyncContinuationMembers->size() - 1; +} + +size_t Compiler::GetContinuationMemberCount() +{ + Compiler* const root = impInlineRoot(); + return root->m_asyncContinuationMembers == nullptr ? 0 : root->m_asyncContinuationMembers->size(); +} + +const ContinuationMember& Compiler::GetContinuationMember(size_t index) +{ + Compiler* const root = impInlineRoot(); + assert(index < root->m_asyncContinuationMembers->size()); + return root->m_asyncContinuationMembers->at(index); +} + //------------------------------------------------------------------------ // Compiler::SaveAsyncContexts: // Insert code in async methods that saves and restores contexts. @@ -66,6 +251,11 @@ PhaseStatus Compiler::SaveAsyncContexts() return PhaseStatus::MODIFIED_NOTHING; } + // Note the OSR handling below only ever applies to the root frame: JIT_FLAG_OSR is + // cleared for inlinees in compInitOptions. An inlinee inside an OSR method starts a + // fresh logical frame and so captures its own contexts, with a resumed indicator that + // starts out false however the OSR method was entered. + // Create locals for Thread, ExecutionContext and SynchronizationContext lvaAsyncThreadObjectVar = lvaGrabTemp(false DEBUGARG("Async Thread")); lvaGetDesc(lvaAsyncThreadObjectVar)->lvType = TYP_REF; @@ -79,6 +269,11 @@ PhaseStatus Compiler::SaveAsyncContexts() lvaResumedIndicator = lvaGrabTemp(false DEBUGARG("Async Resumed")); lvaGetDesc(lvaResumedIndicator)->lvType = TYP_UBYTE; + // Mark this frame as a logical async frame. This has to happen here rather than when + // an inlinee is spliced into its caller: the frame's own inlinees are processed while + // it is still being compiled, and they need to see it in the enclosing frame chain. + compInlineContext->SetIsAsyncFrame(); + if (opts.IsOSR()) { lvaGetDesc(lvaAsyncThreadObjectVar)->lvIsOSRLocal = true; @@ -242,10 +437,7 @@ PhaseStatus Compiler::SaveAsyncContexts() for (BasicBlock* block : Blocks()) { - if (!compIsForInlining()) - { - AddContextArgsToAsyncCalls(block); - } + AddContextArgsToAsyncCalls(block); if (!block->KindIs(BBJ_RETURN) || (block == newReturnBB)) { @@ -337,11 +529,32 @@ void Compiler::AddContextArgsToAsyncCalls(BasicBlock* block) return WALK_CONTINUE; } - GenTreeCall* call = tree->AsCall(); - GenTree* resumed = m_compiler->gtNewLclVarNode(m_compiler->lvaResumedIndicator, TYP_INT); - GenTree* resumedAddr = m_compiler->gtNewLclAddrNode(m_compiler->lvaResumedIndicator, 0); - GenTree* execCtx = m_compiler->gtNewLclVarNode(m_compiler->lvaAsyncExecutionContextVar, TYP_REF); - GenTree* syncCtx = m_compiler->gtNewLclVarNode(m_compiler->lvaAsyncSynchronizationContextVar, TYP_REF); + GenTreeCall* call = tree->AsCall(); + + // Record that this body has a suspension point. Inlinees without one keep the + // cheap shape: no post-inline IR and no capture chain is emitted for them. + m_compiler->compAsyncBodyMaySuspend = true; + + if (call->gtArgs.FindWellKnownArg(WellKnownArg::AsyncResumedUse) != nullptr) + { + // Already has context args: this is an async call in an inlinee that + // inherited them from the inlining call (see + // impInheritAsyncContextsFromInliner). + assert(m_compiler->compIsForInlining()); + return WALK_CONTINUE; + } + + if (m_compiler->compIsForInlining() && !generalAsyncInliningEnabled()) + { + // Only general async inlining introduces inlinee async calls that need + // their own contexts. + return WALK_CONTINUE; + } + + GenTree* resumed = m_compiler->gtNewLclVarNode(m_compiler->lvaResumedIndicator, TYP_INT); + GenTree* resumedAddr = m_compiler->gtNewLclAddrNode(m_compiler->lvaResumedIndicator, 0); + GenTree* execCtx = m_compiler->gtNewLclVarNode(m_compiler->lvaAsyncExecutionContextVar, TYP_REF); + GenTree* syncCtx = m_compiler->gtNewLclVarNode(m_compiler->lvaAsyncSynchronizationContextVar, TYP_REF); JITDUMP( "Adding resumed use [%06u], resumed def [%06u] exec context [%06u], sync context [%06u] to async call [%06u]\n", dspTreeID(resumed), dspTreeID(resumedAddr), dspTreeID(execCtx), dspTreeID(syncCtx), dspTreeID(call)); @@ -620,7 +833,8 @@ PhaseStatus AsyncTransformation::Run() PhaseStatus result = PhaseStatus::MODIFIED_NOTHING; ArrayStack blocksWithNormalAwaits(m_compiler->getAllocator(CMK_Async)); ArrayStack blocksWithTailAwaits(m_compiler->getAllocator(CMK_Async)); - AggregatedAwaitInfo awaits = FindAwaits(blocksWithNormalAwaits, blocksWithTailAwaits); + ArrayStack continuationMemberOffsets(m_compiler->getAllocator(CMK_Async)); + AggregatedAwaitInfo awaits = FindAwaits(blocksWithNormalAwaits, blocksWithTailAwaits, continuationMemberOffsets); if (awaits.NumNormalAwaits + awaits.NumTailAwaits > 1) { @@ -640,7 +854,8 @@ PhaseStatus AsyncTransformation::Run() // This may have changed blocks, so refind the normal awaits. blocksWithNormalAwaits.Reset(); blocksWithTailAwaits.Reset(); - awaits = FindAwaits(blocksWithNormalAwaits, blocksWithTailAwaits); + continuationMemberOffsets.Reset(); + awaits = FindAwaits(blocksWithNormalAwaits, blocksWithTailAwaits, continuationMemberOffsets); } result = PhaseStatus::MODIFIED_EVERYTHING; @@ -650,6 +865,7 @@ PhaseStatus AsyncTransformation::Run() if (awaits.NumNormalAwaits <= 0) { + assert(continuationMemberOffsets.Empty()); return result; } @@ -747,31 +963,41 @@ PhaseStatus AsyncTransformation::Run() } while (any); } - if (ReuseContinuations()) + // Set up the local containing the continuation we can reuse. For OSR + // things are special: we can transition to the OSR method after having + // resumed in the tier0 method. In that case we end up with the tier0 + // continuation in the OSR method, but we cannot reuse it. + if (m_compiler->opts.IsOSR()) { - // Set up the local containing the continuation we can reuse. For OSR - // things are special: we can transition to the OSR method after having - // resumed in the tier0 method. In that case we end up with the tier0 - // continuation in the OSR method, but we cannot reuse it. - if (m_compiler->opts.IsOSR()) - { - m_reuseContinuationVar = m_compiler->lvaGrabTemp(false DEBUGARG("OSR reusable continuation")); - m_compiler->lvaGetDesc(m_reuseContinuationVar)->lvType = TYP_REF; - } - else - { - m_reuseContinuationVar = m_compiler->lvaAsyncContinuationArg; - } + m_reuseContinuationVar = m_compiler->lvaGrabTemp(false DEBUGARG("OSR reusable continuation")); + m_compiler->lvaGetDesc(m_reuseContinuationVar)->lvType = TYP_REF; + } + else + { + m_reuseContinuationVar = m_compiler->lvaAsyncContinuationArg; } GenTreeLclVarCommon* commonAsyncResumedDef = FindAndRemoveCommonAsyncResumedDef(); - CreateResumptionsAndSuspensions(); + const ContinuationLayout* continuationLayout = CreateResumptionsAndSuspensions(continuationMemberOffsets); // After transforming all async calls we have created resumption blocks; // create the resumption switch. CreateResumptionSwitch(commonAsyncResumedDef); + // Now bash all GT_CONTINUATION_MEMBER_OFFSET into appropriate constants. + for (GenTree* node : continuationMemberOffsets.BottomUpOrder()) + { + size_t memberIndex = node->AsVal()->gtVal1; + assert(memberIndex < continuationLayout->ContinuationMemberOffsets.size()); + assert(continuationLayout->ContinuationMemberOffsets[memberIndex] != UINT_MAX); + ssize_t offset = (OFFSETOF__CORINFO_Continuation__data - SIZEOF__CORINFO_Object) + + continuationLayout->ContinuationMemberOffsets[memberIndex]; + // Preserve the node's type: offsets used in address arithmetic are TYP_I_IMPL, + // while those passed to helpers are TYP_INT. + node->BashToConst(offset, node->TypeGet()); + } + m_compiler->fgInvalidateDfsTree(); if (m_compiler->opts.OptimizationDisabled()) @@ -807,12 +1033,14 @@ PhaseStatus AsyncTransformation::Run() // Parameters: // blocksWithNormalAwaits - [out] Blocks with normal awaits are pushed onto this stack // blocksWithTailAwaits - [out] Blocks with tail awaits are pushed onto this stack +// continuationMemberOffsets - [out] Symbolic continuation member offset nodes // // Returns: // Information about awaits in the function. // AggregatedAwaitInfo AsyncTransformation::FindAwaits(ArrayStack& blocksWithNormalAwaits, - ArrayStack& blocksWithTailAwaits) + ArrayStack& blocksWithTailAwaits, + ArrayStack& continuationMemberOffsets) { AggregatedAwaitInfo awaits; for (BasicBlock* block : m_compiler->Blocks()) @@ -821,6 +1049,12 @@ AggregatedAwaitInfo AsyncTransformation::FindAwaits(ArrayStack& blo bool hasTailAwait = false; for (GenTree* tree : LIR::AsRange(block)) { + if (tree->OperIs(GT_CONTINUATION_MEMBER_OFFSET)) + { + continuationMemberOffsets.Push(tree); + continue; + } + if (!tree->IsCall()) { continue; @@ -1398,6 +1632,16 @@ void ContinuationLayout::Dump(int indent) printf("%*s +%03u Keep alive object\n", indent, "", KeepAliveOffset); } + for (size_t i = 0; i < ContinuationMemberOffsets.size(); i++) + { + if (ContinuationMemberOffsets[i] != UINT_MAX) + { + printf("%*s +%03u ", indent, "", ContinuationMemberOffsets[i]); + JitTls::GetCompiler()->GetContinuationMember(i).Print(); + printf("\n"); + } + } + for (const LiveLocalInfo& inf : Locals) { printf("%*s +%03u V%02u: %u bytes\n", indent, "", inf.Offset, inf.LclNum, inf.Size); @@ -1448,10 +1692,12 @@ const ReturnInfo* ContinuationLayout::FindReturn(Compiler* comp, GenTreeCall* ca // The finalized ContinuationLayout with computed offsets and a class // handle for the continuation type. // -ContinuationLayout* ContinuationLayoutBuilder::Create() +ContinuationLayout* ContinuationLayoutBuilder::Create(ArrayStack& continuationMemberOffsets) { ContinuationLayout* layout = new (m_compiler, CMK_Async) ContinuationLayout(m_compiler); layout->Locals.reserve(m_locals.size()); + size_t continuationMemberCount = m_compiler->GetContinuationMemberCount(); + layout->ContinuationMemberOffsets.resize(continuationMemberCount, UINT_MAX); for (unsigned lclNum : m_locals) { @@ -1565,6 +1811,39 @@ ContinuationLayout* ContinuationLayoutBuilder::Create() ret.Offset = allocLayout(ret.HeapAlignment(), ret.Size); } + for (GenTree* memberOffsetNode : continuationMemberOffsets.BottomUpOrder()) + { + size_t memberIndex = memberOffsetNode->AsVal()->gtVal1; + assert(memberIndex < continuationMemberCount); + if (layout->ContinuationMemberOffsets[memberIndex] != UINT_MAX) + { + continue; + } + + const ContinuationMember& member = m_compiler->GetContinuationMember(memberIndex); + + unsigned alignment; + unsigned size; + if (member.Type == ContinuationMemberType::CustomAwaiterOfLayout) + { + ClassLayout* memberLayout = member.GetCustomAwaiterLayout(); + alignment = + memberLayout->IsCustomLayout() + ? (memberLayout->HasGCPtr() ? TARGET_POINTER_SIZE : 1) + : m_compiler->info.compCompHnd->getClassAlignmentRequirement(memberLayout->GetClassHandle()); + size = memberLayout->GetSize(); + } + else + { + var_types storageType = member.GetStorageType(); + alignment = genTypeAlignments[storageType]; + size = genTypeSize(storageType); + } + + layout->ContinuationMemberOffsets[memberIndex] = + allocLayout(std::min(alignment, (unsigned)TARGET_POINTER_SIZE), size); + } + if (m_needsKeepAlive) { layout->KeepAliveOffset = allocLayout(TARGET_POINTER_SIZE, TARGET_POINTER_SIZE); @@ -1613,6 +1892,18 @@ ContinuationLayout* ContinuationLayoutBuilder::Create() bitmapBuilder.SetType(ret.Offset, ret.Type.ReturnType, ret.Type.ReturnLayout); } + for (size_t i = 0; i < layout->ContinuationMemberOffsets.size(); i++) + { + if (layout->ContinuationMemberOffsets[i] != UINT_MAX) + { + const ContinuationMember& member = m_compiler->GetContinuationMember(i); + ClassLayout* memberLayout = member.Type == ContinuationMemberType::CustomAwaiterOfLayout + ? member.GetCustomAwaiterLayout() + : nullptr; + bitmapBuilder.SetType(layout->ContinuationMemberOffsets[i], member.GetStorageType(), memberLayout); + } + } + #ifdef DEBUG if (m_compiler->verbose) { @@ -1955,35 +2246,46 @@ bool AsyncTransformation::IsReusableSuspension(const AsyncState* state, } } - static const WellKnownArg validateArgs[] = {WellKnownArg::AsyncResumedUse, WellKnownArg::AsyncResumedDef, - WellKnownArg::AsyncExecutionContext, + // These can be duplicated: general async inlining adds one set of context args per + // enclosing inlined frame, describing the chain of frames whose handling a suspension + // has to run. All of them must agree for a suspension to be reusable. + static const WellKnownArg validateArgs[] = {WellKnownArg::AsyncAwaiter, WellKnownArg::AsyncResumedUse, + WellKnownArg::AsyncResumedDef, WellKnownArg::AsyncExecutionContext, WellKnownArg::AsyncSynchronizationContext}; + ArrayStack thisNodes(m_compiler->getAllocator(CMK_Async)); + ArrayStack otherNodes(m_compiler->getAllocator(CMK_Async)); for (WellKnownArg arg : validateArgs) { - CallArg* thisArg = call->gtArgs.FindWellKnownArg(arg); - CallArg* otherArg = predAsyncCall->gtArgs.FindWellKnownArg(arg); + thisNodes.Reset(); + otherNodes.Reset(); + CollectWellKnownArgs(call, arg, thisNodes); + CollectWellKnownArgs(predAsyncCall, arg, otherNodes); - if ((thisArg == nullptr) != (otherArg == nullptr)) + if (thisNodes.Height() != otherNodes.Height()) { - JITDUMP(" No; disagreement on presence of %s argument\n", getWellKnownArgName(arg)); + JITDUMP(" No; disagreement on number of %s arguments (%d vs %d)\n", getWellKnownArgName(arg), + thisNodes.Height(), otherNodes.Height()); return false; } - if (thisArg != nullptr) + for (int i = 0; i < thisNodes.Height(); i++) { + GenTree* thisNode = thisNodes.Bottom(i); + GenTree* otherNode = otherNodes.Bottom(i); + // The value may have been folded to a constant (e.g. when the // indicator is provably zero at this point), which is still fine to // compare and to remove from the call. - if (!thisArg->GetNode()->OperIsAnyLocal() && !thisArg->GetNode()->IsInvariant()) + if (!thisNode->OperIsAnyLocal() && !thisNode->IsInvariant()) { JITDUMP(" No; %s argument is too complex\n", getWellKnownArgName(arg)); return false; } - if (!GenTree::Compare(thisArg->GetNode(), otherArg->GetNode())) + if (!GenTree::Compare(thisNode, otherNode)) { JITDUMP(" No; disagreement on value of %s argument ([%06u] vs [%06u])\n", getWellKnownArgName(arg), - Compiler::dspTreeID(thisArg->GetNode()), Compiler::dspTreeID(otherArg->GetNode())); + Compiler::dspTreeID(thisNode), Compiler::dspTreeID(otherNode)); return false; } } @@ -2011,13 +2313,15 @@ bool AsyncTransformation::IsReusableSuspension(const AsyncState* state, // void AsyncTransformation::HandleReusedSuspension(BasicBlock* callBlock, GenTreeCall* call) { - static const WellKnownArg argsToRemove[] = {WellKnownArg::AsyncResumedUse, WellKnownArg::AsyncResumedDef, - WellKnownArg::AsyncExecutionContext, + static const WellKnownArg argsToRemove[] = {WellKnownArg::AsyncAwaiter, WellKnownArg::AsyncResumedUse, + WellKnownArg::AsyncResumedDef, WellKnownArg::AsyncExecutionContext, WellKnownArg::AsyncSynchronizationContext}; for (WellKnownArg wka : argsToRemove) { - CallArg* arg = call->gtArgs.FindWellKnownArg(wka); - if (arg != nullptr) + // These can be duplicated: general async inlining adds one set of context args + // per enclosing inlined frame. All of them must go. + for (CallArg* arg = call->gtArgs.FindWellKnownArg(wka); arg != nullptr; + arg = call->gtArgs.FindWellKnownArg(wka)) { assert(arg->GetNode()->OperIsAnyLocal() || arg->GetNode()->IsInvariant()); LIR::AsRange(callBlock).Remove(arg->GetNode()); @@ -2118,7 +2422,7 @@ void AsyncTransformation::CreateSuspension(BasicBlock* call LIR::AsRange(suspendBB).InsertAtEnd(storeNewContinuation); SaveSet tailSaveSet = SaveSet::All; - if (ReuseContinuations() && resumeReachable) + if (resumeReachable) { // Split suspendBB into suspendBB -> [reuse continuation with Next store] -> [allocNewBlock with allocation // call] -> suspendBBTail [empty] @@ -2244,10 +2548,81 @@ void AsyncTransformation::CreateSuspension(BasicBlock* call LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, storeFlags)); FillInDataOnSuspension(layout, subLayout, suspendBB, mutatedSinceResumption, tailSaveSet); + StoreAsyncAwaiter(callBlock, call, suspendBB, layout); FinishContextHandlingAndSuspension(callBlock, call, suspendBB, layout, subLayout); } +//------------------------------------------------------------------------ +// AsyncTransformation::StoreAsyncAwaiter: +// Move the pseudo awaiter argument into its reserved continuation member. +// +void AsyncTransformation::StoreAsyncAwaiter(BasicBlock* callBlock, + GenTreeCall* call, + BasicBlock* suspendBB, + const ContinuationLayout& layout) +{ + CallArg* awaiterArg = call->gtArgs.FindWellKnownArg(WellKnownArg::AsyncAwaiter); + if (awaiterArg == nullptr) + { + return; + } + + ClassLayout* awaiterLayout = awaiterArg->GetSignatureLayout(); + assert(awaiterLayout != nullptr); + size_t memberIndex = + m_compiler->GetContinuationMemberIndex(ContinuationMember::CustomAwaiterOfLayout(awaiterLayout)); + assert(memberIndex < layout.ContinuationMemberOffsets.size()); + assert(layout.ContinuationMemberOffsets[memberIndex] != UINT_MAX); + + GenTree* awaiter = awaiterArg->GetNode(); + assert(varTypeIsStruct(awaiter)); + + if (awaiter->OperIs(GT_FIELD_LIST)) + { + GenTreeFieldList* fieldList = awaiter->AsFieldList(); + for (GenTreeFieldList::Use& use : fieldList->Uses()) + { + if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR)) + { + LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList); + lirUse.ReplaceWithLclVar(m_compiler); + } + + GenTree* field = use.GetNode(); + LIR::AsRange(callBlock).Remove(field); + + GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF); + unsigned offset = + OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex] + use.GetOffset(); + GenTree* store = StoreAtOffset(continuation, offset, field, use.GetType()); + LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store)); + } + + LIR::AsRange(callBlock).Remove(fieldList); + } + else + { + if (!awaiter->OperIs(GT_LCL_VAR)) + { + LIR::Use use(LIR::AsRange(callBlock), &awaiterArg->NodeRef(), call); + use.ReplaceWithLclVar(m_compiler); + awaiter = use.Def(); + } + + LIR::AsRange(callBlock).Remove(awaiter); + + GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF); + unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex]; + GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL); + GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode); + GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING); + LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store)); + } + + call->gtArgs.RemoveUnsafe(awaiterArg); +} + //------------------------------------------------------------------------ // AsyncTransformation::CreateAllocContinuationCall: // Create a call to the JIT helper that allocates a continuation. @@ -2561,7 +2936,12 @@ void AsyncTransformation::FinishContextHandlingAndSuspension(BasicBlock* assert(suspendBB->KindIs(BBJ_RETURN)); - if (m_sharedReturnBB != nullptr) + BasicBlock* const frameTail = CreateInlinedFrameSuspensionTail(callBlock, call, layout); + if (frameTail != nullptr) + { + suspendBB->SetKindAndTargetEdge(BBJ_ALWAYS, m_compiler->fgAddRefPred(frameTail, suspendBB)); + } + else if (m_sharedReturnBB != nullptr) { suspendBB->SetKindAndTargetEdge(BBJ_ALWAYS, m_compiler->fgAddRefPred(m_sharedReturnBB, suspendBB)); } @@ -2646,7 +3026,12 @@ void AsyncTransformation::FinishContextHandlingAndSuspensionWithHelper(BasicBloc LIR::AsRange(callBlock).Remove(syncContext); call->gtArgs.RemoveUnsafe(syncContextArg); - if (sharedFinish != nullptr) + // A call inside an inlined async frame needs the enclosing frames' handling to run + // after this one, so it cannot use a finish block shared with suspensions in other + // frames. + BasicBlock* const frameTail = CreateInlinedFrameSuspensionTail(callBlock, call, layout); + + if ((sharedFinish != nullptr) && (frameTail == nullptr)) { // Store the vars to the shared locals that the shared finish block will take them from. if (m_sharedFinishContextHandlingResumedVar != BAD_VAR_NUM) @@ -2677,8 +3062,13 @@ void AsyncTransformation::FinishContextHandlingAndSuspensionWithHelper(BasicBloc // Otherwise insert a new call InsertFinishContextHandlingCall(suspendBB, layout, helper, resumed, execContext, syncContext); - // And return either via a new GT_RETURN_SUSPEND or via the shared return BB. - if (m_sharedReturnBB != nullptr) + // And continue with the enclosing frames, or return either via a new + // GT_RETURN_SUSPEND or via the shared return BB. + if (frameTail != nullptr) + { + suspendBB->SetKindAndTargetEdge(BBJ_ALWAYS, m_compiler->fgAddRefPred(frameTail, suspendBB)); + } + else if (m_sharedReturnBB != nullptr) { suspendBB->SetKindAndTargetEdge(BBJ_ALWAYS, m_compiler->fgAddRefPred(m_sharedReturnBB, suspendBB)); } @@ -2691,6 +3081,167 @@ void AsyncTransformation::FinishContextHandlingAndSuspensionWithHelper(BasicBloc } } +//------------------------------------------------------------------------ +// AsyncTransformation::CreateInlinedFrameSuspensionTail: +// Create the block that runs the context handling of the frames enclosing an inlined +// async frame on suspension. +// +// Parameters: +// callBlock - The block containing the async call +// call - The async call +// layout - Information about the continuation layout +// +// Returns: +// The block to continue suspension in, or nullptr if this call is not inside an +// inlined async frame. +// +// Remarks: +// Each frame that has not yet resumed must capture the contexts it would hand to its +// caller, and restore its caller's contexts onto the thread, as if its physical frame +// had returned. A frame having resumed implies its caller has too, so the frames can be +// walked outward as a straight line: once one frame has resumed, the helpers for it and +// every frame outside it no-op. +// +// TODO-Async: suspensions in the same frame could share a tail when they agree on all +// of the values below, which would save some cold code. +// +BasicBlock* AsyncTransformation::CreateInlinedFrameSuspensionTail(BasicBlock* callBlock, + GenTreeCall* call, + const ContinuationLayout& layout) +{ + unsigned const numFrames = call->GetAsyncInfo().InlineFrameDepth + 1; + if (numFrames < 2) + { + // Not inside an inlined async frame, so there are no logical frame transitions to + // run and no context args beyond the ones the suspension itself consumed. + assert(call->gtArgs.FindWellKnownArg(WellKnownArg::AsyncResumedUse) == nullptr); + return nullptr; + } + + JITDUMP(" Call [%06u] is inside an inlined async frame; %u frames in chain\n", Compiler::dspTreeID(call), + numFrames); + + // Take the values of the enclosing frames off the call. These args are the source of + // truth for what to store: the optimizer may have rewritten them since they were + // added, for example folding a "resumed" indicator to a constant, and the stores must + // reflect that rather than re-reading the locals the args originally named. + ArrayStack values(m_compiler->getAllocator(CMK_Async)); + + for (unsigned i = 0; i < numFrames; i++) + { + CallArg* frameArgs[3] = {call->gtArgs.FindWellKnownArg(WellKnownArg::AsyncResumedUse), + call->gtArgs.FindWellKnownArg(WellKnownArg::AsyncExecutionContext), + call->gtArgs.FindWellKnownArg(WellKnownArg::AsyncSynchronizationContext)}; + + for (unsigned j = 0; j < 3; j++) + { + CallArg* arg = frameArgs[j]; + assert(arg != nullptr); + + GenTree* node = arg->GetNode(); + if (!node->IsInvariant() && !node->OperIs(GT_LCL_VAR)) + { + // We are going to reference this from a different block, so spill it. + LIR::Use use(LIR::AsRange(callBlock), &arg->NodeRef(), call); + use.ReplaceWithLclVar(m_compiler); + node = use.Def(); + } + + LIR::AsRange(callBlock).Remove(node); + call->gtArgs.RemoveUnsafe(arg); + values.Push(node); + } + } + + if (m_sharedReturnBB == nullptr) + { + CreateSharedReturnBB(); + } + + BasicBlock* tailBB = m_compiler->fgNewBBbefore(BBJ_ALWAYS, m_sharedReturnBB, false); + tailBB->bbSetRunRarely(); + tailBB->clearTryIndex(); + tailBB->clearHndIndex(); + tailBB->SetKindAndTargetEdge(BBJ_ALWAYS, m_compiler->fgAddRefPred(m_sharedReturnBB, tailBB)); + + if (m_compiler->fgIsUsingProfileWeights()) + { + tailBB->SetFlags(BBF_PROF_WEIGHT); + } + + JITDUMP(" Created inlined frame suspension tail " FMT_BB " for %u frames\n", tailBB->bbNum, numFrames); + + for (unsigned i = 0; i + 1 < numFrames; i++) + { + GenTree* const frameResumed = values.Bottom((int)(i * 3)); + GenTree* const outerResumed = values.Bottom((int)((i + 1) * 3)); + GenTree* const outerExec = values.Bottom((int)((i + 1) * 3 + 1)); + GenTree* const outerSync = values.Bottom((int)((i + 1) * 3 + 2)); + unsigned const depth = numFrames - 1 - i; + + // Capture what this frame hands to its caller. + GenTreeCall* captureCall = + m_compiler->gtNewUserCallNode(m_asyncInfo->captureInlinedFrameTransitionMethHnd, TYP_VOID); + captureCall->gtArgs + .PushFront(m_compiler, + NewCallArg::Primitive( + ContinuationMemberAddress(layout, ContinuationMember::InlineFrameExecutionContext(depth)))); + captureCall->gtArgs.PushFront(m_compiler, + NewCallArg::Primitive( + ContinuationMemberAddress(layout, + ContinuationMember::InlineFrameFlags(depth)))); + captureCall->gtArgs.PushFront(m_compiler, + NewCallArg::Primitive( + ContinuationMemberAddress(layout, + ContinuationMember::InlineFrameContinuationContext( + depth)))); + captureCall->gtArgs.PushFront(m_compiler, NewCallArg::Primitive(m_compiler->gtCloneExpr(frameResumed))); + + m_compiler->compCurBB = tailBB; + m_compiler->fgMorphTree(captureCall); + LIR::AsRange(tailBB).InsertAtEnd(LIR::SeqTree(m_compiler, captureCall)); + + // Then restore the caller's contexts, as its physical frame's return would have. + GenTreeCall* restoreCall = + m_compiler->gtNewUserCallNode(m_asyncInfo->restoreContextsOnSuspensionMethHnd, TYP_VOID); + restoreCall->gtArgs.PushFront(m_compiler, NewCallArg::Primitive(m_compiler->gtCloneExpr(outerSync))); + restoreCall->gtArgs.PushFront(m_compiler, NewCallArg::Primitive(m_compiler->gtCloneExpr(outerExec))); + restoreCall->gtArgs.PushFront(m_compiler, NewCallArg::Primitive(m_compiler->gtCloneExpr(outerResumed))); + + m_compiler->compCurBB = tailBB; + m_compiler->fgMorphTree(restoreCall); + LIR::AsRange(tailBB).InsertAtEnd(LIR::SeqTree(m_compiler, restoreCall)); + } + + DISPRANGE(LIR::AsRange(tailBB)); + + return tailBB; +} + +//------------------------------------------------------------------------ +// AsyncTransformation::ContinuationMemberAddress: +// Create the address of a continuation member in the newly created continuation. +// +// Parameters: +// layout - Information about the continuation layout +// member - The member +// +// Returns: +// IR node computing the address. +// +GenTree* AsyncTransformation::ContinuationMemberAddress(const ContinuationLayout& layout, + const ContinuationMember& member) +{ + size_t const memberIndex = m_compiler->GetContinuationMemberIndex(member); + assert(memberIndex < layout.ContinuationMemberOffsets.size()); + assert(layout.ContinuationMemberOffsets[memberIndex] != UINT_MAX); + + unsigned const offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex]; + GenTree* const continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF); + return m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, + m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL)); +} + //------------------------------------------------------------------------ // AsyncTransformation::RestoreContexts: // Create IR to restore contexts on suspension. @@ -3159,18 +3710,15 @@ BasicBlock* AsyncTransformation::RethrowExceptionOnResumption(BasicBlock* GenTree* storeException = m_compiler->gtNewStoreLclVarNode(exceptionLclNum, exceptionInd); LIR::AsRange(resumeBB).InsertAtEnd(LIR::SeqTree(m_compiler, storeException)); - if (ReuseContinuations()) - { - // If we may reuse this continuation later then make sure we don't see the same exception again. - GenTree* continuation = m_compiler->gtNewLclvNode(m_compiler->lvaAsyncContinuationArg, TYP_REF); - unsigned exceptionOffset = OFFSETOF__CORINFO_Continuation__data + layout.ExceptionOffset; - GenTree* null = m_compiler->gtNewNull(); - GenTree* nullException = StoreAtOffset(continuation, exceptionOffset, null, TYP_REF); - LIR::AsRange(resumeBB).InsertAtEnd(LIR::SeqTree(m_compiler, nullException)); - } + // Since we may reuse this continuation later we make sure we don't see the same exception again. + continuation = m_compiler->gtNewLclvNode(m_compiler->lvaAsyncContinuationArg, TYP_REF); + exceptionOffset = OFFSETOF__CORINFO_Continuation__data + layout.ExceptionOffset; + GenTree* null = m_compiler->gtNewNull(); + GenTree* nullException = StoreAtOffset(continuation, exceptionOffset, null, TYP_REF); + LIR::AsRange(resumeBB).InsertAtEnd(LIR::SeqTree(m_compiler, nullException)); GenTree* exception = m_compiler->gtNewLclVarNode(exceptionLclNum, TYP_REF); - GenTree* null = m_compiler->gtNewNull(); + null = m_compiler->gtNewNull(); GenTree* neNull = m_compiler->gtNewOperNode(GT_NE, TYP_INT, exception, null); GenTree* jtrue = m_compiler->gtNewOperNode(GT_JTRUE, TYP_VOID, neNull); LIR::AsRange(resumeBB).InsertAtEnd(exception, null, neNull, jtrue); @@ -3294,10 +3842,7 @@ void AsyncTransformation::CopyReturnValueOnResumption(GenTreeCall* LIR::AsRange(storeResultBB).InsertAtEnd(LIR::SeqTree(m_compiler, storeResult)); } - if (ReuseContinuations()) - { - ClearReturnValueOnResumption(retInfo, resultOffset, storeResultBB); - } + ClearReturnValueOnResumption(retInfo, resultOffset, storeResultBB); } //------------------------------------------------------------------------ @@ -3859,9 +4404,10 @@ GenTreeLclVarCommon* AsyncTransformation::FindAndRemoveCommonAsyncResumedDef() // Walk all recorded async states and create the suspension and resumption // IR, continuation layouts, and debug info for each one. // -void AsyncTransformation::CreateResumptionsAndSuspensions() +const ContinuationLayout* AsyncTransformation::CreateResumptionsAndSuspensions( + ArrayStack& continuationMemberOffsets) { - bool useSharedLayout = (m_states.size() > 1) && ReuseContinuations(); + bool useSharedLayout = m_states.size() > 1; ContinuationLayout* sharedLayout = nullptr; if (useSharedLayout) @@ -3869,7 +4415,7 @@ void AsyncTransformation::CreateResumptionsAndSuspensions() JITDUMP("Creating shared layout:\n"); ContinuationLayoutBuilder* sharedLayoutBuilder = ContinuationLayoutBuilder::CreateSharedLayout(m_compiler, m_states); - sharedLayout = sharedLayoutBuilder->Create(); + sharedLayout = sharedLayoutBuilder->Create(continuationMemberOffsets); unsigned numSharedSuspensionsWithContinuationContext = 0; unsigned numSharedSuspensionsWithoutContinuationContext = 0; @@ -3955,12 +4501,14 @@ void AsyncTransformation::CreateResumptionsAndSuspensions() } } + ContinuationLayout* layout = nullptr; JITDUMP("Creating suspensions and resumptions for %zu states\n", m_states.size()); for (const AsyncState& state : m_states) { JITDUMP("State %u suspend @ " FMT_BB ", resume @ " FMT_BB "\n", state.Number, state.SuspensionBB->bbNum, state.ResumptionBB->bbNum); - ContinuationLayout* layout = sharedLayout == nullptr ? state.Layout->Create() : sharedLayout; + layout = sharedLayout == nullptr ? state.Layout->Create(continuationMemberOffsets) : sharedLayout; + CreateSuspension(state.CallBlock, state.Call, state.SuspensionBB, state.Number, *layout, *state.Layout, state.ResumeReachable, state.MutatedSincePreviousResumption); CreateResumption(state.CallBlock, state.Call, state.ResumptionBB, state.CallDefInfo, *layout, *state.Layout); @@ -3968,28 +4516,8 @@ void AsyncTransformation::CreateResumptionsAndSuspensions() JITDUMP("\n"); } -} - -//------------------------------------------------------------------------ -// AsyncTransformation::ReuseContinuations: -// Returns true if continuation reuse is enabled. -// -// Returns: -// True if so. -// -bool AsyncTransformation::ReuseContinuations() -{ -#ifdef DEBUG - static ConfigMethodRange s_range; - s_range.EnsureInit(JitConfig.JitAsyncReuseContinuationsRange()); - - if (!s_range.Contains(m_compiler->info.compMethodHash())) - { - return false; - } -#endif - return JitConfig.JitAsyncReuseContinuations() != 0; + return layout; } //------------------------------------------------------------------------ @@ -4246,12 +4774,9 @@ void AsyncTransformation::CreateResumptionSwitch(GenTreeLclVarCommon* commonAsyn GenTree* jtrue = m_compiler->gtNewOperNode(GT_JTRUE, TYP_VOID, eqZero); LIR::AsRange(checkOSRAddressOffsetBB).InsertAtEnd(LIR::SeqTree(m_compiler, jtrue)); - if (ReuseContinuations()) - { - // Also, save the fact that we have a reusable continuation - continuationArg = m_compiler->gtNewLclvNode(m_compiler->lvaAsyncContinuationArg, TYP_REF); - GenTree* storeReusable = m_compiler->gtNewStoreLclVarNode(m_reuseContinuationVar, continuationArg); - LIR::AsRange(onContinuationBB).InsertAtBeginning(continuationArg, storeReusable); - } + // Also, save the fact that we have a reusable continuation + continuationArg = m_compiler->gtNewLclvNode(m_compiler->lvaAsyncContinuationArg, TYP_REF); + GenTree* storeReusable = m_compiler->gtNewStoreLclVarNode(m_reuseContinuationVar, continuationArg); + LIR::AsRange(onContinuationBB).InsertAtBeginning(continuationArg, storeReusable); } } diff --git a/src/coreclr/jit/async.h b/src/coreclr/jit/async.h index ea9f3943b1b3bb..0871050776bb95 100644 --- a/src/coreclr/jit/async.h +++ b/src/coreclr/jit/async.h @@ -1,6 +1,46 @@ // Licensed to the .NET Foundation under one or more agreements. // The .NET Foundation licenses this file to you under the MIT license. +enum class ContinuationMemberType +{ + // A custom awaiter, stored so that it can be awaited from the continuation. + // Deduplicated by layout: all awaits of compatible awaiter types share one slot. + CustomAwaiterOfLayout, + // The ExecutionContext an inlined async frame captured when it logically returned + // to its caller. Deduplicated by inline depth. + InlineFrameExecutionContext, + // The continuation context an inlined async frame captured when it logically + // returned to its caller. Deduplicated by inline depth. + InlineFrameContinuationContext, + // The ContinuationFlags describing the above continuation context. Deduplicated by + // inline depth. + InlineFrameFlags, +}; + +struct ContinuationMember +{ + ContinuationMemberType Type; + + void Print() const; + +private: + ClassLayout* m_customAwaiterLayout; + unsigned m_inlineDepth; + +public: + ClassLayout* GetCustomAwaiterLayout() const; + unsigned GetInlineDepth() const; + + // The type of the storage this member requires in the continuation. + var_types GetStorageType() const; + + static ContinuationMember CustomAwaiterOfLayout(ClassLayout* layout); + static ContinuationMember InlineFrameExecutionContext(unsigned inlineDepth); + static ContinuationMember InlineFrameContinuationContext(unsigned inlineDepth); + static ContinuationMember InlineFrameFlags(unsigned inlineDepth); + static bool AreCompatible(const ContinuationMember& a, const ContinuationMember& b); +}; + struct ReturnTypeInfo { var_types ReturnType = TYP_UNDEF; @@ -121,7 +161,7 @@ struct ContinuationLayoutBuilder static bool Equals(const ContinuationLayoutBuilder& a, const ContinuationLayoutBuilder& b); - struct ContinuationLayout* Create(); + struct ContinuationLayout* Create(ArrayStack& continuationMemberOffsets); static ContinuationLayoutBuilder* CreateSharedLayout(Compiler* comp, const jitstd::vector& states); @@ -137,11 +177,13 @@ struct ContinuationLayout unsigned ExecutionContextOffset = UINT_MAX; jitstd::vector Locals; jitstd::vector Returns; + jitstd::vector ContinuationMemberOffsets; CORINFO_CLASS_HANDLE ClassHnd = NO_CLASS_HANDLE; ContinuationLayout(Compiler* comp) : Locals(comp->getAllocator(CMK_Async)) , Returns(comp->getAllocator(CMK_Async)) + , ContinuationMemberOffsets(comp->getAllocator(CMK_Async)) { } @@ -374,7 +416,8 @@ class AsyncTransformation unsigned m_sharedFinishContextHandlingSyncContextVar = BAD_VAR_NUM; AggregatedAwaitInfo FindAwaits(ArrayStack& blocksWithNormalAwaits, - ArrayStack& blocksWithTailAwaits); + ArrayStack& blocksWithTailAwaits, + ArrayStack& continuationMemberOffsets); void TransformTailAwaits(ArrayStack& blocksWithTailAwaits); void TransformTailAwait(BasicBlock* block, GenTreeCall* call, BasicBlock** remainder); @@ -455,6 +498,10 @@ class AsyncTransformation const ContinuationLayoutBuilder& subLayout, SuspensionContextHelper helper); void RestoreContexts(BasicBlock* block, GenTreeCall* call, BasicBlock* insertionBB); + void StoreAsyncAwaiter(BasicBlock* callBlock, + GenTreeCall* call, + BasicBlock* suspendBB, + const ContinuationLayout& layout); void CreateCheckAndSuspendAfterCall(BasicBlock* block, GenTreeCall* call, const CallDefinitionInfo& callDefInfo, @@ -491,29 +538,32 @@ class AsyncTransformation var_types storeType, GenTreeFlags indirFlags = GTF_IND_NONFAULTING); - void CreateDebugInfoForSuspensionPoint(const ContinuationLayout& layout, - const ContinuationLayoutBuilder& subLayout); - unsigned GetReturnedContinuationVar(); - unsigned GetNewContinuationVar(); - unsigned GetResultBaseVar(); - unsigned GetExceptionVar(); - void CreateSharedReturnBB(); - BasicBlock* CreateSharedFinishContextHandlingBB(SuspensionContextHelper helper, - const ContinuationLayout& layout, - GenTree* invariantResumed, - bool execContextMayVary, - bool syncContextMayVary); - void InsertFinishContextHandlingCall(BasicBlock* block, - const ContinuationLayout& layout, - SuspensionContextHelper helper, - GenTree* resumed, - GenTree* execContext, - GenTree* syncContext); - bool ReuseContinuations(); - - GenTreeLclVarCommon* FindAndRemoveCommonAsyncResumedDef(); - void CreateResumptionsAndSuspensions(); - void CreateResumptionSwitch(GenTreeLclVarCommon* commonAsyncResumedDef); + void CreateDebugInfoForSuspensionPoint(const ContinuationLayout& layout, + const ContinuationLayoutBuilder& subLayout); + unsigned GetReturnedContinuationVar(); + unsigned GetNewContinuationVar(); + unsigned GetResultBaseVar(); + unsigned GetExceptionVar(); + void CreateSharedReturnBB(); + BasicBlock* CreateSharedFinishContextHandlingBB(SuspensionContextHelper helper, + const ContinuationLayout& layout, + GenTree* invariantResumed, + bool execContextMayVary, + bool syncContextMayVary); + void InsertFinishContextHandlingCall(BasicBlock* block, + const ContinuationLayout& layout, + SuspensionContextHelper helper, + GenTree* resumed, + GenTree* execContext, + GenTree* syncContext); + GenTreeLclVarCommon* FindAndRemoveCommonAsyncResumedDef(); + const ContinuationLayout* CreateResumptionsAndSuspensions(ArrayStack& continuationMemberOffsets); + void CreateResumptionSwitch(GenTreeLclVarCommon* commonAsyncResumedDef); + + BasicBlock* CreateInlinedFrameSuspensionTail(BasicBlock* callBlock, + GenTreeCall* call, + const ContinuationLayout& layout); + GenTree* ContinuationMemberAddress(const ContinuationLayout& layout, const ContinuationMember& member); public: AsyncTransformation(Compiler* comp) diff --git a/src/coreclr/jit/compiler.h b/src/coreclr/jit/compiler.h index d2ea1d99d592dc..10657288fc5f6a 100644 --- a/src/coreclr/jit/compiler.h +++ b/src/coreclr/jit/compiler.h @@ -95,6 +95,7 @@ enum class WasmValueType : unsigned; #ifdef DEBUG struct IndentStack; #endif +struct ContinuationMember; class Lowering; // defined in lower.h @@ -4406,6 +4407,12 @@ class Compiler // Variable representing "have we resumed?" for async methods unsigned lvaResumedIndicator = BAD_VAR_NUM; + // For async methods with CORINFO_ASYNC_SAVE_CONTEXTS: whether the body contains any + // async call, i.e. any point at which this method may suspend. Computed by + // SaveAsyncContexts. When false for an inlinee, its resumed indicator is provably + // always false and no post-inline async frame IR needs to be emitted for it. + bool compAsyncBodyMaySuspend = false; + #if defined(DEBUG) && defined(TARGET_XARCH) unsigned lvaReturnSpCheck = BAD_VAR_NUM; // Stores SP to confirm it is not corrupted on return. @@ -5267,6 +5274,15 @@ class Compiler unsigned clsFlags, bool isReadonlyCall); + void impTryOptimizeAwaitAwaiter(GenTreeCall* call, + CORINFO_RESOLVED_TOKEN* pResolvedToken, + CORINFO_SIG_INFO* sig, + CORINFO_CALL_INFO* callInfo, + CORINFO_METHOD_HANDLE* methHnd, + CORINFO_CONTEXT_HANDLE* exactContextHnd, + GenTree** instParam, + NamedIntrinsic ni); + void impSetupAsyncCall(GenTreeCall* call, CORINFO_METHOD_HANDLE methHnd, OPCODE opcode, unsigned prefixFlags, NamedIntrinsic ni, const DebugInfo& callDI); void impInsertAsyncArgsForLdvirtftnCall(GenTreeCall* call); @@ -5638,6 +5654,12 @@ class Compiler bool m_nextAwaitIsTail = false; + // Set while importing an inlinee for an async call that gets its own context + // handling instead of inheriting it from the inlining call, i.e. an await that may + // suspend inside a generally-inlined async callee. Consumed by + // impInheritAsyncContextsFromInliner. + bool m_nextAsyncCallUsesOwnContexts = false; + bool impSpillStackEntry(unsigned level, unsigned varNum #ifdef DEBUG @@ -6258,6 +6280,11 @@ class Compiler PhaseStatus placeLoopAlignInstructions(); #endif + jitstd::vector* m_asyncContinuationMembers = nullptr; + size_t GetContinuationMemberIndex(const ContinuationMember& member); + size_t GetContinuationMemberCount(); + const ContinuationMember& GetContinuationMember(size_t index); + PhaseStatus SaveAsyncContexts(); void AddContextArgsToAsyncCalls(BasicBlock* block); BasicBlock* CreateReturnBB(unsigned* mergedReturnLcl); @@ -7506,6 +7533,10 @@ class Compiler void fgInsertInlineeArgument(const InlArgInfo& argInfo, BasicBlock* block, Statement** afterStmt, Statement** newStmt, const DebugInfo& callDI); Statement* fgInlinePrependStatements(InlineInfo* inlineInfo); void fgInlineAppendStatements(InlineInfo* inlineInfo, BasicBlock* block, Statement* stmt); + void fgInlineAppendAsyncFrameStatements(InlineInfo* inlineInfo, BasicBlock* joinBlock); + void fgAppendEnclosingAsyncFrameContextArgs(InlineInfo* inlineInfo); + void fgSetupAsyncFrameTransitionCall(GenTreeCall* call, const DebugInfo& di); + GenTree* gtNewContinuationMemberIndir(const struct ContinuationMember& member, var_types type); #ifdef DEBUG static fgWalkPreFn fgDebugCheckInlineCandidates; @@ -11927,6 +11958,14 @@ class Compiler return (info.compMethodInfo->options & CORINFO_ASYNC_VERSION) != 0; } + // Is general inlining of runtime async calls enabled, i.e. inlining of async callees + // that may suspend? When disabled only the restricted cases are inlined: callees + // without any awaits, async versions of synchronous methods, and tail awaits. + static bool generalAsyncInliningEnabled() + { + return JitConfig.JitAsyncInlining() != 0; + } + //------------------------------------------------------------------------ // compMethodReturnsMultiRegRetType: Does this method return a multi-reg value? // diff --git a/src/coreclr/jit/fginline.cpp b/src/coreclr/jit/fginline.cpp index b11ab31fea2d16..a78f66ae1ea632 100644 --- a/src/coreclr/jit/fginline.cpp +++ b/src/coreclr/jit/fginline.cpp @@ -2,6 +2,7 @@ // The .NET Foundation licenses this file to you under the MIT license. #include "jitpch.h" +#include "async.h" #ifdef _MSC_VER #pragma hdrstop @@ -1644,6 +1645,10 @@ void Compiler::fgInsertInlineeBlocks(InlineInfo* pInlineInfo) // Append statements to null out gc ref locals, if necessary. fgInlineAppendStatements(pInlineInfo, iciBlock, stmtAfter); + + // An inlinee that does async context handling always has multiple blocks: its + // SaveAsyncContexts creates a try/fault region and a merged return block. + assert(InlineeCompiler->lvaResumedIndicator == BAD_VAR_NUM); insertInlineeBlocks = false; } else @@ -1890,6 +1895,10 @@ void Compiler::fgInsertInlineeBlocks(InlineInfo* pInlineInfo) // Append statements to null out gc ref locals, if necessary. fgInlineAppendStatements(pInlineInfo, bottomBlock, nullptr); JITDUMPEXEC(fgDispBasicBlocks(InlineeCompiler->fgFirstBB, InlineeCompiler->fgLastBB, true)); + + // Handle the inlined async frame logically returning to its caller. + fgAppendEnclosingAsyncFrameContextArgs(pInlineInfo); + fgInlineAppendAsyncFrameStatements(pInlineInfo, bottomBlock); } // @@ -2422,6 +2431,416 @@ Statement* Compiler::fgInlinePrependStatements(InlineInfo* inlineInfo) return afterStmt; } +//------------------------------------------------------------------------ +// fgAppendEnclosingAsyncFrameContextArgs: Add the context values of the frames +// enclosing an inlined async frame to the async calls inside it. +// +// Arguments: +// inlineInfo - information about the inline +// +// Notes: +// When one of these calls suspends, every enclosing frame that has not yet resumed +// must have its contexts captured into the continuation and restored onto the thread, +// as if its physical frame had returned. The IR for that is created by the async +// transformation, long after the optimizer has run, so the values have to be kept +// live and GC reported at the call until then. That is what the existing per-frame +// context args do, so the enclosing frames simply add more of them. +// +// The enclosing frames are exactly the ones the call being inlined already describes, +// so its context args are simply copied over. That composes for nested inlines: a +// frame picks up its immediate caller's chain when it is spliced in, and every call in +// it, including ones that came from its own inlinees, is extended again when that +// caller is itself inlined. +// +// These are pseudo-args, so duplicates are fine: they take no registers and are +// expanded out by the async transformation. The innermost frame stays first, since +// the inlinee pushed its own args to the front. +// +void Compiler::fgAppendEnclosingAsyncFrameContextArgs(InlineInfo* inlineInfo) +{ + if (!generalAsyncInliningEnabled()) + { + return; + } + + if ((InlineeCompiler->lvaResumedIndicator == BAD_VAR_NUM) || !InlineeCompiler->compAsyncBodyMaySuspend) + { + return; + } + + // Context args of the inlining call. It carries one set for the frame it suspends in, + // which that frame's own handling consumes, followed by that frame's chain of frames + // out to the root. + ArrayStack callerArgs(getAllocator(CMK_Async)); + for (CallArg& arg : inlineInfo->iciCall->gtArgs.Args()) + { + switch (arg.GetWellKnownArg()) + { + case WellKnownArg::AsyncResumedUse: + case WellKnownArg::AsyncExecutionContext: + case WellKnownArg::AsyncSynchronizationContext: + callerArgs.Push(&arg); + break; + default: + break; + } + } + + if (callerArgs.Height() == 0) + { + // The inlining call does no context handling, so no enclosing frame is a logical + // async frame. The inlinee's frame has nothing to hand back and there are no frame + // transitions to run on suspension. + JITDUMP("Inlining call [%06u] has no context args; inlinee has no enclosing async frame\n", + dspTreeID(inlineInfo->iciCall)); + return; + } + + assert((callerArgs.Height() % 3) == 0); + + // The frames enclosing the inlinee are the inlining call's chain. When the call has + // only the one set it sits in the root frame, whose chain is never materialized + // because it has no transitions of its own; that frame is then the only enclosing one. + int const firstEnclosing = (callerArgs.Height() == 3) ? 0 : 3; + + unsigned const inlineeResumed = InlineeCompiler->lvaResumedIndicator; + unsigned const inlineeExecCtx = InlineeCompiler->lvaAsyncExecutionContextVar; + unsigned const inlineeSyncCtx = InlineeCompiler->lvaAsyncSynchronizationContextVar; + + struct Visitor : GenTreeVisitor + { + enum + { + DoPreOrder = true, + }; + + ArrayStack& m_callerArgs; + int m_firstEnclosing; + unsigned m_inlineeResumed; + unsigned m_inlineeExecCtx; + unsigned m_inlineeSyncCtx; + + Visitor(Compiler* comp, + ArrayStack& callerArgs, + int firstEnclosing, + unsigned inlineeResumed, + unsigned inlineeExecCtx, + unsigned inlineeSyncCtx) + : GenTreeVisitor(comp) + , m_callerArgs(callerArgs) + , m_firstEnclosing(firstEnclosing) + , m_inlineeResumed(inlineeResumed) + , m_inlineeExecCtx(inlineeExecCtx) + , m_inlineeSyncCtx(inlineeSyncCtx) + { + } + + fgWalkResult PreOrderVisit(GenTree** use, GenTree* user) + { + GenTree* tree = *use; + if ((tree->gtFlags & GTF_CALL) == 0) + { + return WALK_SKIP_SUBTREES; + } + + if (!tree->IsCall() || !tree->AsCall()->IsAsync()) + { + return WALK_CONTINUE; + } + + GenTreeCall* const call = tree->AsCall(); + + unsigned numOwn = 0; + for (CallArg& arg : call->gtArgs.Args()) + { + if (arg.GetWellKnownArg() == WellKnownArg::AsyncResumedUse) + { + numOwn++; + } + } + + if (numOwn == 0) + { + // Call does no context handling, so it has no frame chain to extend. + return WALK_CONTINUE; + } + + if (numOwn == 1) + { + // The call only has the set describing the frame it suspends in, so that + // frame is the inlinee's. The suspension needs the inlinee's resumed + // indicator again to decide whether to capture the contexts it hands to + // its caller, since the first set is consumed by the innermost handling. + call->gtArgs.PushBack(m_compiler, + NewCallArg::Primitive(m_compiler->gtNewLclVarNode(m_inlineeResumed, TYP_INT)) + .WellKnown(WellKnownArg::AsyncResumedUse)); + call->gtArgs.PushBack(m_compiler, + NewCallArg::Primitive(m_compiler->gtNewLclVarNode(m_inlineeExecCtx, TYP_REF)) + .WellKnown(WellKnownArg::AsyncExecutionContext)); + call->gtArgs.PushBack(m_compiler, + NewCallArg::Primitive(m_compiler->gtNewLclVarNode(m_inlineeSyncCtx, TYP_REF)) + .WellKnown(WellKnownArg::AsyncSynchronizationContext)); + } + + // Then the frames enclosing the inlinee, taken from the inlining call. These + // are the values as they appear in the caller's IR, which is what the + // suspension has to store. + for (int i = m_firstEnclosing; i < m_callerArgs.Height(); i++) + { + CallArg* const arg = m_callerArgs.Bottom(i); + call->gtArgs.PushBack(m_compiler, NewCallArg::Primitive(m_compiler->gtCloneExpr(arg->GetNode())) + .WellKnown(arg->GetWellKnownArg())); + } + + unsigned numFrames = 0; + for (CallArg& arg : call->gtArgs.Args()) + { + if (arg.GetWellKnownArg() == WellKnownArg::AsyncResumedUse) + { + numFrames++; + } + } + + // One set is consumed by the handling of the frame the call suspends in, and + // the rest describe that frame and the ones enclosing it. + assert(numFrames >= 2); + call->GetAsyncInfo().InlineFrameDepth = numFrames - 2; + + JITDUMP("Extended async call [%06u] to %u frames in chain\n", Compiler::dspTreeID(call), numFrames - 1); + return WALK_CONTINUE; + } + }; + + Visitor visitor(this, callerArgs, firstEnclosing, inlineeResumed, inlineeExecCtx, inlineeSyncCtx); + for (BasicBlock* block = InlineeCompiler->fgFirstBB; block != nullptr; block = block->Next()) + { + for (Statement* const stmt : block->Statements()) + { + visitor.WalkTree(stmt->GetRootNodePointer(), nullptr); + } + + if (block == InlineeCompiler->fgLastBB) + { + break; + } + } +} + +//------------------------------------------------------------------------ +// fgSetupAsyncFrameTransitionCall: Turn a call to AsyncHelpers.RestoreInlinedFrameContexts +// into a proper async call. +// +// Arguments: +// call - the call, with its user arguments already added +// di - debug info to use +// +// Notes: +// The call is an await: it suspends when it has to switch continuation context. +// +// Unlike normal awaits it gets no context pseudo-args. Those model what a frame +// captures and restores when a suspension unwinds through it, and none of that applies +// here: we only run at all because the frame we are returning out of was resumed, so +// every frame in the chain already has a continuation holding the values it captured +// when it first suspended, and a suspension here returns straight back to the +// dispatcher, which restores the thread's contexts itself. +// +void Compiler::fgSetupAsyncFrameTransitionCall(GenTreeCall* call, const DebugInfo& di) +{ + AsyncCallInfo asyncInfo; + // The helper resumes on the requested context, so no further handling is attached to + // this await. + asyncInfo.ContinuationContextHandling = ContinuationContextHandling::None; + asyncInfo.CallAsyncDebugInfo = di; + call->SetIsAsync(new (this, CMK_Async) AsyncCallInfo(asyncInfo)); + + if (Target::g_tgtArgOrder == Target::ARG_ORDER_R2L) + { + call->gtArgs.PushFront(this, + NewCallArg::Primitive(gtNewNull(), TYP_REF).WellKnown(WellKnownArg::AsyncContinuation)); + } + else + { + call->gtArgs.PushBack(this, + NewCallArg::Primitive(gtNewNull(), TYP_REF).WellKnown(WellKnownArg::AsyncContinuation)); + } +} + +//------------------------------------------------------------------------ +// gtNewContinuationMemberIndir: Create a load of a continuation member from the +// continuation this method was resumed with. +// +// Arguments: +// member - the continuation member to load +// type - type of the member +// +// Returns: +// A load of the member. +// +// Notes: +// The member offset is not known until the async transformation has laid out the +// continuation, so it is represented symbolically until then. The offset is relative +// to the instance data, hence the additional object header adjustment. +// +GenTree* Compiler::gtNewContinuationMemberIndir(const ContinuationMember& member, var_types type) +{ + size_t const memberIndex = GetContinuationMemberIndex(member); + + GenTree* const memberOffset = + new (this, GT_CONTINUATION_MEMBER_OFFSET) GenTreeVal(GT_CONTINUATION_MEMBER_OFFSET, TYP_I_IMPL, memberIndex); + GenTree* const headerSize = gtNewIconNode(SIZEOF__CORINFO_Object, TYP_I_IMPL); + GenTree* const offset = gtNewOperNode(GT_ADD, TYP_I_IMPL, memberOffset, headerSize); + + GenTree* const continuation = gtNewLclvNode(lvaAsyncContinuationArg, TYP_REF); + GenTree* const addr = gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offset); + + GenTree* const indir = gtNewIndir(type, addr, GTF_IND_NONFAULTING); + return indir; +} + +//------------------------------------------------------------------------ +// fgInlineAppendAsyncFrameStatements: Emit the IR that handles an inlined async +// frame logically returning to its caller. +// +// Arguments: +// inlineInfo - information about the inline +// joinBlock - the block all of the inlinee's returns converge on +// +// Notes: +// When an inlined async callee is resumed inside its own body, the async +// infrastructure only restored the context of the suspension point itself. The work +// it would otherwise have done when the callee's frame returned to its caller has to +// be performed here instead: +// +// if (resumed_F) +// { +// await AsyncHelpers.RestoreInlinedFrameContexts(continuation.ExecutionContextFor, +// continuation.ContinuationContextFor, +// continuation.FlagsFor); +// resumed_caller = true; +// } +// +// Only the check is expanded as IR; the restore itself is left as a single call. The +// synchronous case is the one worth optimizing, and by the time we get past the check +// we have already suspended and resumed at least once, so one more call is cheap. In +// particular the "are we already on the right context?" test, which is what actually +// decides whether we suspend, stays inside the helper. +// +// The members are read off the continuation this method was resumed with. That is +// only valid when resumed_F is true, which is exactly the guard: resumed_F can only +// become true at a resumption point belonging to this inlinee's own async calls, and +// those belong to this method's continuation layout. +// +void Compiler::fgInlineAppendAsyncFrameStatements(InlineInfo* inlineInfo, BasicBlock* joinBlock) +{ + if (!generalAsyncInliningEnabled()) + { + return; + } + + if (InlineeCompiler->lvaResumedIndicator == BAD_VAR_NUM) + { + // Inlinee does not do context handling, so it inherited everything from this + // call and there is no logical frame transition to handle. + return; + } + + InlineContext* const inlineContext = inlineInfo->inlineContext; + + if (!InlineeCompiler->compAsyncBodyMaySuspend) + { + // The inlinee has no suspension point, so its resumed indicator is provably + // always false and all of the below would be dead code. + JITDUMP("Inlinee cannot suspend; no async frame transition IR needed\n"); + return; + } + + // The resumed indicator of the frame we are logically returning into. The inlining + // call carries it as the definition of its own frame's indicator, which is a local + // address and so something we can store to. + CallArg* const resumedDefArg = inlineInfo->iciCall->gtArgs.FindWellKnownArg(WellKnownArg::AsyncResumedDef); + if (resumedDefArg == nullptr) + { + // The inlining call does no context handling, so there is no enclosing logical + // async frame to hand anything back to. + JITDUMP("Inlining call does no context handling; no async frame transition IR needed\n"); + return; + } + + assert(resumedDefArg->GetNode()->OperIs(GT_LCL_ADDR)); + unsigned const resumedCaller = resumedDefArg->GetNode()->AsLclVarCommon()->GetLclNum(); + + unsigned const resumedInlinee = InlineeCompiler->lvaResumedIndicator; + unsigned const inlineDepth = inlineContext->GetAsyncFrameDepth(); + + JITDUMP("Adding async frame transition IR for async frame depth %u: resumed V%02u -> V%02u\n", inlineDepth, + resumedInlinee, resumedCaller); + + const DebugInfo& di = inlineInfo->iciStmt->GetDebugInfo(); + + // Split so that the transition IR precedes the remainder of the caller's code. + BasicBlock* const restBlock = fgSplitBlockAtBeginning(joinBlock); + + // joinBlock is now empty and jumps to restBlock. Turn it into the "did we resume + // inside the inlinee?" test. + BasicBlock* const restoreBlock = fgNewBBafter(BBJ_ALWAYS, joinBlock, /* extendRegion */ true); + restoreBlock->inheritWeightPercentage(joinBlock, 0); + + // joinBlock: if (resumed_F == 0) goto restBlock; else goto restoreBlock + { + GenTree* const resumed = gtNewLclvNode(resumedInlinee, TYP_INT); + GenTree* const isZero = gtNewOperNode(GT_EQ, TYP_INT, resumed, gtNewIconNode(0)); + GenTree* const jtrue = gtNewOperNode(GT_JTRUE, TYP_VOID, isZero); + fgInsertStmtAtEnd(joinBlock, gtNewStmt(jtrue)); + + fgRemoveRefPred(joinBlock->GetTargetEdge()); + FlowEdge* const toRest = fgAddRefPred(restBlock, joinBlock); + FlowEdge* const toRestore = fgAddRefPred(restoreBlock, joinBlock); + joinBlock->SetCond(toRest, toRestore); + // Resuming inside the inlinee is the cold path. + toRest->setLikelihood(1.0); + toRestore->setLikelihood(0.0); + } + + // restoreBlock: restore the contexts the caller's continuation captured, then record + // that the caller's frame has observed a resumption too. + { + CORINFO_ASYNC_INFO* const asyncInfo = eeGetAsyncInfo(); + + GenTreeCall* const restoreCall = gtNewUserCallNode(asyncInfo->restoreInlinedFrameContextsMethHnd, TYP_VOID); + restoreCall->gtArgs.PushFront(this, NewCallArg::Primitive( + gtNewContinuationMemberIndir(ContinuationMember::InlineFrameFlags( + inlineDepth), + TYP_INT))); + restoreCall->gtArgs + .PushFront(this, + NewCallArg::Primitive( + gtNewContinuationMemberIndir(ContinuationMember::InlineFrameContinuationContext(inlineDepth), + TYP_REF))); + restoreCall->gtArgs.PushFront(this, + NewCallArg::Primitive( + gtNewContinuationMemberIndir(ContinuationMember::InlineFrameExecutionContext( + inlineDepth), + TYP_REF))); + fgSetupAsyncFrameTransitionCall(restoreCall, di); + + // The helper is small on the path that does not suspend, which is the one we care + // about, so let the inliner have a look at it. + CORINFO_CALL_INFO callInfo = {}; + callInfo.hMethod = restoreCall->gtCallMethHnd; + callInfo.methodFlags = info.compCompHnd->getMethodAttribs(callInfo.hMethod); + impMarkInlineCandidate(restoreCall, MAKE_METHODCONTEXT(callInfo.hMethod), &callInfo, compInlineContext); + + fgInsertStmtAtEnd(restoreBlock, gtNewStmt(restoreCall)); + + GenTree* const store = gtNewStoreLclVarNode(resumedCaller, gtNewIconNode(1)); + fgInsertStmtAtEnd(restoreBlock, gtNewStmt(store)); + + restoreBlock->SetKindAndTargetEdge(BBJ_ALWAYS, fgAddRefPred(restBlock, restoreBlock)); + } + + JITDUMPEXEC(fgDispBasicBlocks(joinBlock, restBlock, true)); +} + //------------------------------------------------------------------------ // fgInlineAppendStatements: Append statements that are needed // after the inlined call. diff --git a/src/coreclr/jit/gentree.cpp b/src/coreclr/jit/gentree.cpp index d8f28778b0e2af..a74f26d9a330de 100644 --- a/src/coreclr/jit/gentree.cpp +++ b/src/coreclr/jit/gentree.cpp @@ -13,6 +13,7 @@ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX #include "jitpch.h" #include "hwintrinsic.h" #include "simd.h" +#include "async.h" #ifdef _MSC_VER #pragma hdrstop @@ -1648,6 +1649,7 @@ bool CallArgs::GetCustomRegister(Compiler* comp, CorInfoCallConvExtension cc, We #endif // HAS_FIXED_REGISTER_SET case WellKnownArg::StackArrayLocal: + case WellKnownArg::AsyncAwaiter: case WellKnownArg::AsyncExecutionContext: case WellKnownArg::AsyncSynchronizationContext: case WellKnownArg::AsyncResumedUse: @@ -11289,6 +11291,7 @@ GenTree* Compiler::gtCloneExpr(GenTree* tree) case GT_JMP: case GT_RECORD_ASYNC_RESUME: case GT_ASYNC_RESUME_INFO: + case GT_CONTINUATION_MEMBER_OFFSET: copy = new (this, oper) GenTreeVal(oper, tree->gtType, tree->AsVal()->gtVal1); goto DONE; @@ -12045,6 +12048,7 @@ GenTreeUseEdgeIterator::GenTreeUseEdgeIterator(GenTree* node) case GT_CATCH_ARG: case GT_ASYNC_CONTINUATION: case GT_ASYNC_RESUME_INFO: + case GT_CONTINUATION_MEMBER_OFFSET: case GT_LABEL: case GT_FTN_ADDR: case GT_FTN_ENTRY: @@ -14268,6 +14272,15 @@ void Compiler::gtDispLeaf(GenTree* tree, IndentStack* indentStack) case GT_WASM_JEXCEPT: break; + case GT_CONTINUATION_MEMBER_OFFSET: + { + printf(" index=%zu ", tree->AsVal()->gtVal1); + + const ContinuationMember& member = GetContinuationMember(tree->AsVal()->gtVal1); + member.Print(); + break; + } + case GT_RET_EXPR: { GenTreeCall* inlineCand = tree->AsRetExpr()->gtInlineCandidate; diff --git a/src/coreclr/jit/gentree.h b/src/coreclr/jit/gentree.h index ab02812f445863..14764c4fdc74b5 100644 --- a/src/coreclr/jit/gentree.h +++ b/src/coreclr/jit/gentree.h @@ -4554,6 +4554,15 @@ struct AsyncCallInfo // and suspend unconditionally. bool AlwaysSuspends = false; + // Depth of the inlined frame this call belongs to, counting only frames that have + // async context handling. 0 is the root method's frame. + // + // On suspension every enclosing frame that has not yet resumed must capture the + // contexts it would hand to its caller. Those live in continuation members keyed by + // this depth, and the enclosing frames' depths follow by decrementing while draining + // the call's chain of context args. + unsigned InlineFrameDepth = 0; + bool NeedsToSaveAndRestoreExecutionContext() const { return true; @@ -5273,6 +5282,12 @@ struct GenTreeCall final : public GenTree return *asyncInfo; } + AsyncCallInfo& GetAsyncInfo() + { + assert(IsAsync()); + return *asyncInfo; + } + //--------------------------------------------------------------------------- // GetRegNumByIdx: get i'th return register allocated to this call node. // diff --git a/src/coreclr/jit/gtlist.h b/src/coreclr/jit/gtlist.h index be81ca9f22052d..42ff6a03a84018 100644 --- a/src/coreclr/jit/gtlist.h +++ b/src/coreclr/jit/gtlist.h @@ -41,6 +41,12 @@ GTNODE(GCPOLL , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTLI GTNODE(ASYNC_RESUME_INFO, GenTreeVal ,0,0,GTK_LEAF) // Address of async resume info for a state GTNODE(FTN_ENTRY , GenTree ,0,0,GTK_LEAF) // Address of this function's entry point +// Offset into continuation, starting after the MethodTable*, of a special +// member. GenTreeVal::gtVal1 is an index into Compiler::m_asyncContinuationMembers. +// The async transformation allocates and lays out the continuation and +// substitutes these by constants at that time. +GTNODE(CONTINUATION_MEMBER_OFFSET, GenTreeVal ,0,0,GTK_LEAF) + //----------------------------------------------------------------------------- // Constant nodes: //----------------------------------------------------------------------------- diff --git a/src/coreclr/jit/gtstructs.h b/src/coreclr/jit/gtstructs.h index 59cd282c9544e1..bec1093d34d0b9 100644 --- a/src/coreclr/jit/gtstructs.h +++ b/src/coreclr/jit/gtstructs.h @@ -50,7 +50,7 @@ GTSTRUCT_0(UnOp , GT_OP) GTSTRUCT_0(Op , GT_OP) -GTSTRUCT_N(Val , GT_JMP, GT_RECORD_ASYNC_RESUME, GT_ASYNC_RESUME_INFO) +GTSTRUCT_N(Val , GT_JMP, GT_RECORD_ASYNC_RESUME, GT_ASYNC_RESUME_INFO, GT_CONTINUATION_MEMBER_OFFSET) GTSTRUCT_2_SPECIAL(IntConCommon, GT_CNS_INT, GT_CNS_LNG) GTSTRUCT_1(IntCon , GT_CNS_INT) GTSTRUCT_1(LngCon , GT_CNS_LNG) diff --git a/src/coreclr/jit/importer.cpp b/src/coreclr/jit/importer.cpp index 806c8334838b4e..91213fbce50b9d 100644 --- a/src/coreclr/jit/importer.cpp +++ b/src/coreclr/jit/importer.cpp @@ -12015,10 +12015,17 @@ bool Compiler::impWrapTopOfStackInAwait() JITDUMP("Inheriting continuation handling %d from caller [%06u]\n", (unsigned)inlCall->GetAsyncInfo().ContinuationContextHandling, dspTreeID(inlCall)); asyncInfo->ContinuationContextHandling = inlCall->GetAsyncInfo().ContinuationContextHandling; + + // Mark the call async first: inheriting the contexts also inherits the inlined + // frame depth, which lives in the async call info. + awaitCall->SetIsAsync(asyncInfo); impInheritAsyncContextsFromInliner(awaitCall); } - awaitCall->SetIsAsync(asyncInfo); + if (!awaitCall->IsAsync()) + { + awaitCall->SetIsAsync(asyncInfo); + } if (awaitCall->IsInlineCandidate()) { diff --git a/src/coreclr/jit/importercalls.cpp b/src/coreclr/jit/importercalls.cpp index 775339048a6af1..bfff818d6992c3 100644 --- a/src/coreclr/jit/importercalls.cpp +++ b/src/coreclr/jit/importercalls.cpp @@ -2,6 +2,7 @@ // The .NET Foundation licenses this file to you under the MIT license. #include "jitpch.h" +#include "async.h" //------------------------------------------------------------------------ // impGetInstParamArg: compute the hidden instantiation / generic-context argument @@ -127,6 +128,79 @@ GenTree* Compiler::impGetInstParamArg(CORINFO_RESOLVED_TOKEN* pResolvedToken, return instParam; } +//------------------------------------------------------------------------ +// impTryOptimizeAwaitAwaiter: +// Rewrite a struct AwaitAwaiter or UnsafeAwaitAwaiter call to use the +// corresponding helper that reads the awaiter from the continuation. +// +void Compiler::impTryOptimizeAwaitAwaiter(GenTreeCall* call, + CORINFO_RESOLVED_TOKEN* pResolvedToken, + CORINFO_SIG_INFO* sig, + CORINFO_CALL_INFO* callInfo, + CORINFO_METHOD_HANDLE* methHnd, + CORINFO_CONTEXT_HANDLE* exactContextHnd, + GenTree** instParam, + NamedIntrinsic ni) +{ + CallArg* awaiterArg = call->gtArgs.GetUserArgByIndex(0); + if (!varTypeIsStruct(awaiterArg->GetSignatureType())) + { + return; + } + + CORINFO_LOOKUP newInstArgLookup; + CORINFO_SIG_INFO lookupSig = *sig; + if (pResolvedToken->pMethodSpec != nullptr) + { + lookupSig.pSig = pResolvedToken->pMethodSpec; + lookupSig.cbSig = pResolvedToken->cbMethodSpec; + lookupSig.scope = pResolvedToken->tokenScope; + } + + bool isUnsafe = ni == NI_System_Runtime_CompilerServices_AsyncHelpers_UnsafeAwaitAwaiter; + CORINFO_CONTEXT_HANDLE newExactContextHnd; + CORINFO_METHOD_HANDLE newMethod = + info.compCompHnd->getAwaitAwaiterInContinuationCall(info.compMethodHnd, &lookupSig, isUnsafe, + &newExactContextHnd, &newInstArgLookup); + + CORINFO_SIG_INFO newSig; + info.compCompHnd->getMethodSig(newMethod, &newSig); + + GenTree* newInstParam = nullptr; + if (newSig.hasTypeArg()) + { + newInstParam = impLookupToTree(&newInstArgLookup, GTF_ICON_METHOD_HDL, newMethod); + if (newInstParam == nullptr) + { + JITDUMP("Failed to optimize awaiter call [%06u] because its replacement lookup could not be created\n", + dspTreeID(call)); + return; + } + } + + JITDUMP("Optimizing awaiter call [%06u] to read its struct awaiter from the continuation\n", dspTreeID(call)); + + *methHnd = newMethod; + *exactContextHnd = newExactContextHnd; + call->gtCallMethHnd = newMethod; + callInfo->hMethod = newMethod; + callInfo->methodFlags = info.compCompHnd->getMethodAttribs(newMethod); + callInfo->sig = newSig; + *instParam = newInstParam; + + GenTree* awaiter = awaiterArg->GetNode(); + var_types awaiterType = awaiterArg->GetSignatureType(); + ClassLayout* awaiterLayout = awaiterArg->GetSignatureLayout(); + call->gtArgs.Remove(awaiterArg); + call->gtArgs + .PushFront(this, NewCallArg::Struct(awaiter, awaiterType, awaiterLayout).WellKnown(WellKnownArg::AsyncAwaiter)); + + size_t memberIndex = GetContinuationMemberIndex(ContinuationMember::CustomAwaiterOfLayout(awaiterLayout)); + GenTree* offset = + new (this, GT_CONTINUATION_MEMBER_OFFSET) GenTreeVal(GT_CONTINUATION_MEMBER_OFFSET, TYP_INT, memberIndex); + call->gtArgs.PushBack(this, NewCallArg::Primitive(offset)); +} + //------------------------------------------------------------------------ // impImportCall: import a call-inspiring opcode // @@ -952,6 +1026,13 @@ var_types Compiler::impImportCall(OPCODE opcode, impPopCallArgs(sig, call->AsCall()); + if (opts.OptimizationEnabled() && ((ni == NI_System_Runtime_CompilerServices_AsyncHelpers_AwaitAwaiter) || + (ni == NI_System_Runtime_CompilerServices_AsyncHelpers_UnsafeAwaitAwaiter))) + { + impTryOptimizeAwaitAwaiter(call->AsCall(), pResolvedToken, sig, callInfo, &methHnd, &exactContextHnd, + &instParam, ni); + } + // Extra args if ((instParam != nullptr) || (asyncContinuation != nullptr) || (varArgsCookie != nullptr)) { @@ -7313,38 +7394,64 @@ void Compiler::impSetupAsyncCall(GenTreeCall* call, if (compIsForInlining()) { - if (!m_nextAwaitIsTail && !compIsAsyncVersion()) + // Two cases are inlined cheaply: async versions of synchronous methods, where + // all async calls are in tail position, and explicit tail awaits. In both the + // inlinee's tail can run in the caller's context, so we can inherit all context + // handling from the inlining call and no logical frame transition is needed when + // the inlinee returns. + bool inheritsCallerContexts = m_nextAwaitIsTail || compIsAsyncVersion(); + m_nextAsyncCallUsesOwnContexts = false; + + // We cannot inline if the callee returns valueTask.AsTask(). We need to preserve + // the continuation in this case to be able to mark it with + // CORINFO_CONTINUATION_VALUETASK_ADAPTED_TO_TASK. + if ((prefixFlags & PREFIX_IS_ADAPTED_FROM_VALUETASK) != 0) { compInlineResult->NoteFatal(InlineObservation::CALLEE_AWAIT); return; } - // We cannot inline if the callee returns valueTask.AsTask() in an - // async version. We need to preserve the continuation in this case to - // be able to mark it with CORINFO_CONTINUATION_VALUETASK_ADAPTED_TO_TASK. - if ((prefixFlags & PREFIX_IS_ADAPTED_FROM_VALUETASK) != 0) + if (!inheritsCallerContexts) { - compInlineResult->NoteFatal(InlineObservation::CALLEE_AWAIT); - return; - } + if (!generalAsyncInliningEnabled()) + { + compInlineResult->NoteFatal(InlineObservation::CALLEE_AWAIT); + return; + } + + // General case: this is a real await inside the inlinee that may suspend. It + // gets its own context handling below, exactly like an await in a non-inlined + // method, and the inlinee's own contexts (created by its SaveAsyncContexts) + // are used rather than the caller's. - // For async versions of synchronous methods all async calls are in - // tail position. Inlining is simple for these cases: we can just - // inherit all context handling from the inlining call. - assert(!compIsAsyncVersion() || ((prefixFlags & PREFIX_IS_ASYNC_VERSION_TAIL_AWAIT) != 0)); + // Suspending inside a protected region additionally requires catching and + // rethrowing around the post-inline handling, which is not implemented yet. + if ((compCurBB != nullptr) && (compCurBB->hasTryIndex() || compCurBB->hasHndIndex())) + { + compInlineResult->NoteFatal(InlineObservation::CALLEE_AWAIT_IN_TRY); + return; + } + + JITDUMP("Call [%06u] is an await in an inlinee that may suspend\n", dspTreeID(call)); + m_nextAsyncCallUsesOwnContexts = true; + } + else + { + assert(!compIsAsyncVersion() || ((prefixFlags & PREFIX_IS_ASYNC_VERSION_TAIL_AWAIT) != 0)); - GenTreeCall* inlCall = impInlineInfo->iciCall; - JITDUMP("Call [%06u] is to function with a tail async call [%06u]\n", dspTreeID(inlCall), dspTreeID(call)); + GenTreeCall* inlCall = impInlineInfo->iciCall; + JITDUMP("Call [%06u] is to function with a tail async call [%06u]\n", dspTreeID(inlCall), dspTreeID(call)); - assert(inlCall->IsAsync()); + assert(inlCall->IsAsync()); - asyncInfo.ContinuationContextHandling = inlCall->GetAsyncInfo().ContinuationContextHandling; - // Validate that below code won't override the handling - assert((prefixFlags & PREFIX_IS_TASK_AWAIT) == 0); + asyncInfo.ContinuationContextHandling = inlCall->GetAsyncInfo().ContinuationContextHandling; + // Validate that below code won't override the handling + assert((prefixFlags & PREFIX_IS_TASK_AWAIT) == 0); - asyncInfo.IsTailAwait = - inlCall->GetAsyncInfo().IsTailAwait && (m_nextAwaitIsTail || (call->gtReturnType == info.compRetType)); - m_nextAwaitIsTail = false; + asyncInfo.IsTailAwait = + inlCall->GetAsyncInfo().IsTailAwait && (m_nextAwaitIsTail || (call->gtReturnType == info.compRetType)); + m_nextAwaitIsTail = false; + } } else { @@ -7429,6 +7536,14 @@ void Compiler::impInheritAsyncContextsFromInliner(GenTreeCall* call) return; } + if (m_nextAsyncCallUsesOwnContexts) + { + // General async inlining: this await may suspend, so it uses the inlinee's own + // contexts, which are added by the inlinee's SaveAsyncContexts phase. + m_nextAsyncCallUsesOwnContexts = false; + return; + } + GenTreeCall* inlCall = impInlineInfo->iciCall; CallArg* resumedUseArg = inlCall->gtArgs.FindWellKnownArg(WellKnownArg::AsyncResumedUse); CallArg* resumedDefArg = inlCall->gtArgs.FindWellKnownArg(WellKnownArg::AsyncResumedDef); @@ -7463,6 +7578,41 @@ void Compiler::impInheritAsyncContextsFromInliner(GenTreeCall* call) call->gtArgs.PushFront(this, NewCallArg::Primitive(execNode).WellKnown(WellKnownArg::AsyncExecutionContext)); call->gtArgs.PushFront(this, NewCallArg::Primitive(resumedDefNode).WellKnown(WellKnownArg::AsyncResumedDef)); call->gtArgs.PushFront(this, NewCallArg::Primitive(resumedUseNode).WellKnown(WellKnownArg::AsyncResumedUse)); + + // The inlining call may carry further sets describing the frames enclosing it, which + // this call inherits as well: it ends up in the same frame, so a suspension in it has + // to run the same chain of frame transitions. Dropping them would silently lose the + // handling for every frame outside the immediate one. + bool skippedFirst = false; + for (CallArg& arg : inlCall->gtArgs.Args()) + { + WellKnownArg wka = arg.GetWellKnownArg(); + if ((wka != WellKnownArg::AsyncResumedUse) && (wka != WellKnownArg::AsyncExecutionContext) && + (wka != WellKnownArg::AsyncSynchronizationContext)) + { + continue; + } + + if ((wka == WellKnownArg::AsyncResumedUse) && !skippedFirst) + { + // Already inherited above, along with the resumed def that only the innermost + // frame has. + skippedFirst = true; + continue; + } + + if (&arg == execArg || &arg == syncArg) + { + continue; + } + + call->gtArgs.PushBack(this, NewCallArg::Primitive(gtCloneExpr(arg.GetNode())).WellKnown(wka)); + } + + if (call->IsAsync()) + { + call->GetAsyncInfo().InlineFrameDepth = inlCall->GetAsyncInfo().InlineFrameDepth; + } } //------------------------------------------------------------------------ diff --git a/src/coreclr/jit/inline.def b/src/coreclr/jit/inline.def index aaa7ce52dc82d1..e2a63f2e783616 100644 --- a/src/coreclr/jit/inline.def +++ b/src/coreclr/jit/inline.def @@ -57,6 +57,7 @@ INLINE_OBSERVATION(STFLD_NEEDS_HELPER, bool, "stfld needs helper", INLINE_OBSERVATION(TOO_MANY_ARGUMENTS, bool, "too many arguments", FATAL, CALLEE) INLINE_OBSERVATION(TOO_MANY_LOCALS, bool, "too many locals", FATAL, CALLEE) INLINE_OBSERVATION(AWAIT, bool, "has await", FATAL, CALLEE) +INLINE_OBSERVATION(AWAIT_IN_TRY, bool, "has await inside try region", FATAL, CALLEE) INLINE_OBSERVATION(ASYNC_SUSPEND, bool, "has async suspend", FATAL, CALLEE) // ------ Callee Performance ------- diff --git a/src/coreclr/jit/inline.h b/src/coreclr/jit/inline.h index 939a7b767fd81c..4318cf603ff9b8 100644 --- a/src/coreclr/jit/inline.h +++ b/src/coreclr/jit/inline.h @@ -793,6 +793,28 @@ class InlineContext return m_Parent; } + // Depth of the inlined frame this context represents, counting only frames that do + // async context handling. The shallowest such frame is depth 1 when the root method + // does context handling itself, and depth 0 otherwise (for example when the root is + // the async version of a synchronous method). + // + // Continuation members holding the contexts captured at frame transitions are keyed + // by this, so frames at the same depth share storage. That is safe because their live + // ranges cannot overlap. + unsigned GetAsyncFrameDepth() const + { + unsigned depth = 0; + for (InlineContext* parent = m_Parent; parent != nullptr; parent = parent->m_Parent) + { + if (parent->IsAsyncFrame()) + { + depth++; + } + } + + return depth; + } + // Get the sibling context. InlineContext* GetSibling() const { @@ -899,6 +921,19 @@ class InlineContext return (m_PgoInfo.PgoSchema != nullptr) && (m_PgoInfo.PgoSchemaCount > 0) && (m_PgoInfo.PgoData != nullptr); } + // Whether the frame this context represents does async context handling, i.e. whether + // it is a logical async frame. Set when the corresponding Compiler creates its context + // locals, so that it is already known while the frame's own inlinees are processed. + void SetIsAsyncFrame() + { + m_isAsyncFrame = true; + } + + bool IsAsyncFrame() const + { + return m_isAsyncFrame; + } + private: InlineContext(InlineStrategy* strategy); @@ -919,6 +954,8 @@ class InlineContext unsigned m_Ordinal; // Ordinal number of this inline bool m_Success : 1; // true if this was a successful inline + bool m_isAsyncFrame = false; + #if defined(DEBUG) InlinePolicy* m_Policy; // policy that evaluated this inline diff --git a/src/coreclr/jit/jitconfigvalues.h b/src/coreclr/jit/jitconfigvalues.h index 5e3095e94dd87c..b497598377ea40 100644 --- a/src/coreclr/jit/jitconfigvalues.h +++ b/src/coreclr/jit/jitconfigvalues.h @@ -620,11 +620,11 @@ OPT_CONFIG_STRING(JitAsyncDefaultValueAnalysisRange, // a continuation is being reused. OPT_CONFIG_STRING(JitAsyncPreservedValueAnalysisRange, "JitAsyncPreservedValueAnalysisRange") -// Enable continuation reuse based on method hash range -OPT_CONFIG_STRING(JitAsyncReuseContinuationsRange, "JitAsyncReuseContinuationsRange") -// Save and reuse continuation instances in runtime async functions. Also -// implies use of shared continuation layouts for all suspension points. -RELEASE_CONFIG_INTEGER(JitAsyncReuseContinuations, "JitAsyncReuseContinuations", 1) +// Enable general inlining of runtime async calls, i.e. inlining of async +// callees that may suspend. When zero, only the restricted cases are inlined: +// callees without any awaits, async versions of synchronous methods, and tail +// awaits. +RELEASE_CONFIG_INTEGER(JitAsyncInlining, "JitAsyncInlining", 1) RELEASE_CONFIG_INTEGER(JitEnableOptRepeat, "JitEnableOptRepeat", 1) // If zero, do not allow JitOptRepeat RELEASE_CONFIG_METHODSET(JitOptRepeat, "JitOptRepeat") // Runs optimizer multiple times on specified methods diff --git a/src/coreclr/jit/morph.cpp b/src/coreclr/jit/morph.cpp index 6e24f05cddd930..9c19b45f595621 100644 --- a/src/coreclr/jit/morph.cpp +++ b/src/coreclr/jit/morph.cpp @@ -2214,6 +2214,19 @@ bool Compiler::fgTryMorphStructArg(CallArg* arg) GenTree* argNode = *use; assert(varTypeIsStruct(argNode)); + if (arg->AbiInfo.NumSegments == 0) + { + // Pseudo arg. One case is WellKnownArg::AsyncAwaiter. We just handle + // these as arbitrary struct operands that can be expanded into + // FIELD_LIST. The async transformation will later store the value into + // the continuation, so FIELD_LIST allows using decomposed stores. + if (fgTryReplaceStructLocalWithFields(&arg->NodeRef())) + { + arg->GetNode()->SetMorphed(this, true); + } + return true; + } + bool isSplit = arg->AbiInfo.IsSplitAcrossRegistersAndStack(); #ifdef TARGET_ARM if ((isSplit && (arg->AbiInfo.CountRegsAndStackSlots() > 4)) || (!isSplit && arg->AbiInfo.HasAnyStackSegment())) diff --git a/src/coreclr/jit/valuenum.cpp b/src/coreclr/jit/valuenum.cpp index 96e46cd73a5812..f1e59da35cba08 100644 --- a/src/coreclr/jit/valuenum.cpp +++ b/src/coreclr/jit/valuenum.cpp @@ -13455,6 +13455,7 @@ void Compiler::fgValueNumberTree(GenTree* tree) case GT_CATCH_ARG: case GT_ASYNC_CONTINUATION: + case GT_CONTINUATION_MEMBER_OFFSET: case GT_SWIFT_ERROR: // We know nothing about the value of these. tree->gtVNPair.SetBoth(vnStore->VNForExpr(compCurBB, tree->TypeGet())); diff --git a/src/coreclr/jit/wellknownargs.h b/src/coreclr/jit/wellknownargs.h index 56e3ad962095dd..23976312e5e844 100644 --- a/src/coreclr/jit/wellknownargs.h +++ b/src/coreclr/jit/wellknownargs.h @@ -36,6 +36,7 @@ WELL_KNOWN_ARG(SwiftSelf, "swift self", true, false) WELL_KNOWN_ARG(X86TailCallSpecialArg, "tail call", false, false) WELL_KNOWN_ARG(StackArrayLocal, "&lcl arr", false, false) WELL_KNOWN_ARG(RuntimeMethodHandle, "meth hnd", false, false) +WELL_KNOWN_ARG(AsyncAwaiter, "awaiter", false, false) WELL_KNOWN_ARG(AsyncExecutionContext, "exec ctx", false, false) WELL_KNOWN_ARG(AsyncSynchronizationContext, "sync ctx", false, false) WELL_KNOWN_ARG(AsyncResumedUse, "resumed", false, false) diff --git a/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs b/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs index acbf063ada8c5b..68175209d25a61 100644 --- a/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs +++ b/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs @@ -3650,6 +3650,8 @@ private void getAsyncInfo(ref CORINFO_ASYNC_INFO pAsyncInfoOut) pAsyncInfoOut.captureContextsMethHnd = ObjectToHandle(asyncHelpers.GetKnownMethod("CaptureContexts"u8, null)); pAsyncInfoOut.restoreContextsMethHnd = ObjectToHandle(asyncHelpers.GetKnownMethod("RestoreContexts"u8, null)); pAsyncInfoOut.restoreContextsOnSuspensionMethHnd = ObjectToHandle(asyncHelpers.GetKnownMethod("RestoreContextsOnSuspension"u8, null)); + pAsyncInfoOut.restoreInlinedFrameContextsMethHnd = ObjectToHandle(asyncHelpers.GetKnownMethod("RestoreInlinedFrameContexts"u8, null)); + pAsyncInfoOut.captureInlinedFrameTransitionMethHnd = ObjectToHandle(asyncHelpers.GetKnownMethod("CaptureInlinedFrameTransition"u8, null)); pAsyncInfoOut.finishSuspensionNoContinuationContextMethHnd = ObjectToHandle(asyncHelpers.GetKnownMethod("FinishSuspensionNoContinuationContext"u8, null)); pAsyncInfoOut.finishSuspensionWithContinuationContextMethHnd = ObjectToHandle(asyncHelpers.GetKnownMethod("FinishSuspensionWithContinuationContext"u8, null)); } @@ -3735,6 +3737,61 @@ private void getWasmWellKnownGlobals(ref CORINFO_WASM_WELLKNOWN_GLOBALS pWellKno return ObjectToHandle(targetMethod); } + private CORINFO_METHOD_STRUCT_* getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_STRUCT_* callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_STRUCT** contextHandle, + ref CORINFO_LOOKUP instArg) + { + Debug.Assert(callSig->sigInst.methInstCount == 1); + + instArg.lookupKind.needsRuntimeLookup = false; + instArg.constLookup.accessType = InfoAccessType.IAT_VALUE; + instArg.constLookup.addr = null; + + MethodDesc caller = HandleToObject(callerHandle); + TypeDesc awaiterType = HandleToObject(callSig->sigInst.methInst[0]); + CompilerTypeSystemContext context = _compilation.TypeSystemContext; + DefType asyncHelpers = + context.SystemModule.GetKnownType("System.Runtime.CompilerServices"u8, "AsyncHelpers"u8); + MethodSignature signature = new MethodSignature( + MethodSignatureFlags.Static, + 1, + context.GetWellKnownType(WellKnownType.Void), + [context.GetWellKnownType(WellKnownType.Int32)]); + MethodDesc result = asyncHelpers + .GetKnownMethod( + isUnsafe ? "UnsafeAwaitAwaiterInContinuation"u8 : "AwaitAwaiterInContinuation"u8, + signature) + .MakeInstantiatedMethod(awaiterType); + MethodDesc targetMethod = result.GetCanonMethodTarget(CanonicalFormKind.Specific); + *contextHandle = contextFromMethod(result); + + if (targetMethod.RequiresInstArg()) + { +#if READYTORUN + instArg.constLookup = CreateConstLookupToSymbol( + _compilation.SymbolNodeFactory.CreateReadyToRunHelper( + ReadyToRunHelperId.MethodDictionary, + new MethodWithToken( + result, + _compilation.NodeFactory.Resolver.GetModuleTokenForMethod( + result, + allowDynamicallyCreatedReference: true, + throwIfNotFound: true), + constrainedType: null, + unboxing: false, + genericContextObject: caller))); +#else + ComputeLookup(caller != MethodBeingCompiled, result, ReadyToRunHelperId.MethodDictionary, caller, + ref instArg); +#endif + } + + return ObjectToHandle(targetMethod); + } + private CORINFO_CLASS_STRUCT_* getContinuationType(nuint dataSize, ref bool objRefs, nuint objRefsSize) { Debug.Assert(objRefsSize == (dataSize + (nuint)(PointerSize - 1)) / (nuint)PointerSize); diff --git a/src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs b/src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs index 5be5fd01b69eb9..2c705be59d0e0c 100644 --- a/src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs +++ b/src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs @@ -142,6 +142,7 @@ static ICorJitInfoCallbacks() s_callbacks.getEEInfo = &_getEEInfo; s_callbacks.getAsyncInfo = &_getAsyncInfo; s_callbacks.getAwaitReturnCall = &_getAwaitReturnCall; + s_callbacks.getAwaitAwaiterInContinuationCall = &_getAwaitAwaiterInContinuationCall; s_callbacks.getMethodDefFromMethod = &_getMethodDefFromMethod; s_callbacks.printMethodName = &_printMethodName; s_callbacks.getMethodNameFromMetadata = &_getMethodNameFromMetadata; @@ -327,6 +328,7 @@ static ICorJitInfoCallbacks() public delegate* unmanaged getEEInfo; public delegate* unmanaged getAsyncInfo; public delegate* unmanaged getAwaitReturnCall; + public delegate* unmanaged getAwaitAwaiterInContinuationCall; public delegate* unmanaged getMethodDefFromMethod; public delegate* unmanaged printMethodName; public delegate* unmanaged getMethodNameFromMetadata; @@ -2195,6 +2197,21 @@ private static void _getAsyncInfo(IntPtr thisHandle, IntPtr* ppException, CORINF } } + [UnmanagedCallersOnly] + private static CORINFO_METHOD_STRUCT_* _getAwaitAwaiterInContinuationCall(IntPtr thisHandle, IntPtr* ppException, CORINFO_METHOD_STRUCT_* callerHandle, CORINFO_SIG_INFO* callSig, byte isUnsafe, CORINFO_CONTEXT_STRUCT** contextHandle, CORINFO_LOOKUP* instArg) + { + var _this = GetThis(thisHandle); + try + { + return _this.getAwaitAwaiterInContinuationCall(callerHandle, callSig, isUnsafe != 0, contextHandle, ref *instArg); + } + catch (Exception ex) + { + *ppException = _this.AllocException(ex); + return default; + } + } + [UnmanagedCallersOnly] private static mdToken _getMethodDefFromMethod(IntPtr thisHandle, IntPtr* ppException, CORINFO_METHOD_STRUCT_* hMethod) { diff --git a/src/coreclr/tools/Common/JitInterface/CorInfoTypes.cs b/src/coreclr/tools/Common/JitInterface/CorInfoTypes.cs index c11a4cae141efc..d421e9b5905e58 100644 --- a/src/coreclr/tools/Common/JitInterface/CorInfoTypes.cs +++ b/src/coreclr/tools/Common/JitInterface/CorInfoTypes.cs @@ -972,6 +972,8 @@ public unsafe struct CORINFO_ASYNC_INFO public CORINFO_METHOD_STRUCT_* captureContextsMethHnd; public CORINFO_METHOD_STRUCT_* restoreContextsMethHnd; public CORINFO_METHOD_STRUCT_* restoreContextsOnSuspensionMethHnd; + public CORINFO_METHOD_STRUCT_* restoreInlinedFrameContextsMethHnd; + public CORINFO_METHOD_STRUCT_* captureInlinedFrameTransitionMethHnd; public CORINFO_METHOD_STRUCT_* finishSuspensionNoContinuationContextMethHnd; public CORINFO_METHOD_STRUCT_* finishSuspensionWithContinuationContextMethHnd; } diff --git a/src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt b/src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt index f478dc79563f29..1036b4fde3dfa9 100644 --- a/src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt +++ b/src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt @@ -297,6 +297,7 @@ FUNCTIONS void getEEInfo(CORINFO_EE_INFO* pEEInfoOut); void getAsyncInfo(CORINFO_ASYNC_INFO* pAsyncInfoOut); CORINFO_METHOD_HANDLE getAwaitReturnCall(CORINFO_METHOD_HANDLE callerHandle, CORINFO_CONTEXT_HANDLE* contextHandle, CORINFO_LOOKUP* instArg); + CORINFO_METHOD_HANDLE getAwaitAwaiterInContinuationCall(CORINFO_METHOD_HANDLE callerHandle, CORINFO_SIG_INFO* callSig, bool isUnsafe, CORINFO_CONTEXT_HANDLE* contextHandle, CORINFO_LOOKUP* instArg); mdMethodDef getMethodDefFromMethod(CORINFO_METHOD_HANDLE hMethod); size_t printMethodName(CORINFO_METHOD_HANDLE ftn, char* buffer, size_t bufferSize, size_t* pRequiredBufferSize) const char* getMethodNameFromMetadata(CORINFO_METHOD_HANDLE ftn, const char **className, const char **namespaceName, const char **enclosingClassNames, size_t maxEnclosingClassNames); diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs index 18d4f644407ddb..aadf13417548eb 100644 --- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs +++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs @@ -1067,6 +1067,8 @@ private void AddNecessaryAsyncReferences(MethodDesc method) asyncHelpers.GetKnownMethod("TransparentAwait"u8, new MethodSignature(MethodSignatureFlags.Static, 0, voidType, [valueTaskType])), asyncHelpers.GetKnownMethod("TransparentAwait"u8, new MethodSignature(MethodSignatureFlags.Static, 1, methodVar, [taskOfTType.MakeInstantiatedType(methodVar)])), asyncHelpers.GetKnownMethod("TransparentAwait"u8, new MethodSignature(MethodSignatureFlags.Static, 1, methodVar, [valueTaskOfTType.MakeInstantiatedType(methodVar)])), + asyncHelpers.GetKnownMethod("AwaitAwaiterInContinuation"u8, null), + asyncHelpers.GetKnownMethod("UnsafeAwaitAwaiterInContinuation"u8, null), ]; var moduleForNewReferences = ((EcmaMethod)method.GetPrimaryMethodDesc().GetTypicalMethodDefinition()).Module; _tokenManager.EnsureDefTokensAreAvailable([..requiredMethods, ..requiredTypes, ..requiredFields], moduleForNewReferences, true); diff --git a/src/coreclr/tools/aot/jitinterface/jitinterface_generated.h b/src/coreclr/tools/aot/jitinterface/jitinterface_generated.h index 75648980ac0a7b..c1387f628af346 100644 --- a/src/coreclr/tools/aot/jitinterface/jitinterface_generated.h +++ b/src/coreclr/tools/aot/jitinterface/jitinterface_generated.h @@ -133,6 +133,7 @@ struct JitInterfaceCallbacks void (* getEEInfo)(void * thisHandle, CorInfoExceptionClass** ppException, CORINFO_EE_INFO* pEEInfoOut); void (* getAsyncInfo)(void * thisHandle, CorInfoExceptionClass** ppException, CORINFO_ASYNC_INFO* pAsyncInfoOut); CORINFO_METHOD_HANDLE (* getAwaitReturnCall)(void * thisHandle, CorInfoExceptionClass** ppException, CORINFO_METHOD_HANDLE callerHandle, CORINFO_CONTEXT_HANDLE* contextHandle, CORINFO_LOOKUP* instArg); + CORINFO_METHOD_HANDLE (* getAwaitAwaiterInContinuationCall)(void * thisHandle, CorInfoExceptionClass** ppException, CORINFO_METHOD_HANDLE callerHandle, CORINFO_SIG_INFO* callSig, bool isUnsafe, CORINFO_CONTEXT_HANDLE* contextHandle, CORINFO_LOOKUP* instArg); mdMethodDef (* getMethodDefFromMethod)(void * thisHandle, CorInfoExceptionClass** ppException, CORINFO_METHOD_HANDLE hMethod); size_t (* printMethodName)(void * thisHandle, CorInfoExceptionClass** ppException, CORINFO_METHOD_HANDLE ftn, char* buffer, size_t bufferSize, size_t* pRequiredBufferSize); const char* (* getMethodNameFromMetadata)(void * thisHandle, CorInfoExceptionClass** ppException, CORINFO_METHOD_HANDLE ftn, const char** className, const char** namespaceName, const char** enclosingClassNames, size_t maxEnclosingClassNames); @@ -1384,6 +1385,19 @@ class JitInterfaceWrapper : public ICorJitInfo return temp; } + virtual CORINFO_METHOD_HANDLE getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) +{ + CorInfoExceptionClass* pException = nullptr; + CORINFO_METHOD_HANDLE temp = _callbacks->getAwaitAwaiterInContinuationCall(_thisHandle, &pException, callerHandle, callSig, isUnsafe, contextHandle, instArg); + if (pException != nullptr) throw pException; + return temp; +} + virtual mdMethodDef getMethodDefFromMethod( CORINFO_METHOD_HANDLE hMethod) { diff --git a/src/coreclr/tools/superpmi/superpmi-shared/agnostic.h b/src/coreclr/tools/superpmi/superpmi-shared/agnostic.h index 42c1f462755b4d..8b8e5e5f39b8aa 100644 --- a/src/coreclr/tools/superpmi/superpmi-shared/agnostic.h +++ b/src/coreclr/tools/superpmi/superpmi-shared/agnostic.h @@ -237,6 +237,8 @@ struct Agnostic_CORINFO_ASYNC_INFO DWORDLONG captureContextsMethHnd; DWORDLONG restoreContextsMethHnd; DWORDLONG restoreContextsOnSuspensionMethHnd; + DWORDLONG restoreInlinedFrameContextsMethHnd; + DWORDLONG captureInlinedFrameTransitionMethHnd; DWORDLONG finishSuspensionNoContinuationContextMethHnd; DWORDLONG finishSuspensionWithContinuationContextMethHnd; }; @@ -256,6 +258,13 @@ struct Agnostic_GetAwaitReturnCallResult Agnostic_CORINFO_LOOKUP instArg; }; +struct Agnostic_GetAwaitAwaiterInContinuationCall +{ + DWORDLONG callerHnd; + Agnostic_CORINFO_SIG_INFO callSig; + DWORD isUnsafe; +}; + struct Agnostic_GetOSRInfo { DWORD index; diff --git a/src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h b/src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h index 1616fc79d2916f..bc1bedc95085a8 100644 --- a/src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h +++ b/src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h @@ -80,6 +80,7 @@ LWM(GetDelegateCtor, Agnostic_GetDelegateCtorIn, Agnostic_GetDelegateCtorOut) LWM(GetEEInfo, DWORD, Agnostic_CORINFO_EE_INFO) LWM(GetAsyncInfo, DWORD, Agnostic_CORINFO_ASYNC_INFO) LWM(GetAwaitReturnCall, DWORDLONG, Agnostic_GetAwaitReturnCallResult) +LWM(GetAwaitAwaiterInContinuationCall, Agnostic_GetAwaitAwaiterInContinuationCall, Agnostic_GetAwaitReturnCallResult) LWM(GetEHinfo, DLD, Agnostic_CORINFO_EH_CLAUSE) LWM(GetStaticFieldContent, DLDDD, DD) LWM(GetObjectContent, DLDD, DD) diff --git a/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp b/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp index 7d90db055b4d3d..1fa1e4c1e3204f 100644 --- a/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp +++ b/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp @@ -4394,6 +4394,8 @@ void MethodContext::recGetAsyncInfo(const CORINFO_ASYNC_INFO* pAsyncInfo) value.captureContextsMethHnd = CastHandle(pAsyncInfo->captureContextsMethHnd); value.restoreContextsMethHnd = CastHandle(pAsyncInfo->restoreContextsMethHnd); value.restoreContextsOnSuspensionMethHnd = CastHandle(pAsyncInfo->restoreContextsOnSuspensionMethHnd); + value.restoreInlinedFrameContextsMethHnd = CastHandle(pAsyncInfo->restoreInlinedFrameContextsMethHnd); + value.captureInlinedFrameTransitionMethHnd = CastHandle(pAsyncInfo->captureInlinedFrameTransitionMethHnd); value.finishSuspensionNoContinuationContextMethHnd = CastHandle(pAsyncInfo->finishSuspensionNoContinuationContextMethHnd); value.finishSuspensionWithContinuationContextMethHnd = CastHandle(pAsyncInfo->finishSuspensionWithContinuationContextMethHnd); @@ -4420,6 +4422,8 @@ void MethodContext::repGetAsyncInfo(CORINFO_ASYNC_INFO* pAsyncInfoOut) pAsyncInfoOut->captureContextsMethHnd = (CORINFO_METHOD_HANDLE)value.captureContextsMethHnd; pAsyncInfoOut->restoreContextsMethHnd = (CORINFO_METHOD_HANDLE)value.restoreContextsMethHnd; pAsyncInfoOut->restoreContextsOnSuspensionMethHnd = (CORINFO_METHOD_HANDLE)value.restoreContextsOnSuspensionMethHnd; + pAsyncInfoOut->restoreInlinedFrameContextsMethHnd = (CORINFO_METHOD_HANDLE)value.restoreInlinedFrameContextsMethHnd; + pAsyncInfoOut->captureInlinedFrameTransitionMethHnd = (CORINFO_METHOD_HANDLE)value.captureInlinedFrameTransitionMethHnd; pAsyncInfoOut->finishSuspensionNoContinuationContextMethHnd = (CORINFO_METHOD_HANDLE)value.finishSuspensionNoContinuationContextMethHnd; pAsyncInfoOut->finishSuspensionWithContinuationContextMethHnd = (CORINFO_METHOD_HANDLE)value.finishSuspensionWithContinuationContextMethHnd; DEBUG_REP(dmpGetAsyncInfo(0, value)); @@ -4485,6 +4489,73 @@ CORINFO_METHOD_HANDLE MethodContext::repGetAwaitReturnCall(CORINFO_METHOD_HANDLE return (CORINFO_METHOD_HANDLE)result.methodHnd; } +void MethodContext::recGetAwaitAwaiterInContinuationCall(CORINFO_METHOD_HANDLE callerHnd, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg, + CORINFO_METHOD_HANDLE methHnd) +{ + if (GetAwaitAwaiterInContinuationCall == nullptr) + { + GetAwaitAwaiterInContinuationCall = + new LightWeightMap(); + } + + Agnostic_GetAwaitAwaiterInContinuationCall key; + ZeroMemory(&key, sizeof(key)); + key.callerHnd = CastHandle(callerHnd); + key.callSig = + SpmiRecordsHelper::StoreAgnostic_CORINFO_SIG_INFO(*callSig, GetAwaitAwaiterInContinuationCall, SigInstHandleMap); + key.isUnsafe = isUnsafe; + + Agnostic_GetAwaitReturnCallResult value; + ZeroMemory(&value, sizeof(value)); + value.methodHnd = CastHandle(methHnd); + value.contextHandle = CastHandle(*contextHandle); + value.instArg = SpmiRecordsHelper::StoreAgnostic_CORINFO_LOOKUP(instArg); + + GetAwaitAwaiterInContinuationCall->Add(key, value); + DEBUG_REC(dmpGetAwaitAwaiterInContinuationCall(key, value)); +} + +void MethodContext::dmpGetAwaitAwaiterInContinuationCall( + const Agnostic_GetAwaitAwaiterInContinuationCall& key, + Agnostic_GetAwaitReturnCallResult& value) +{ + printf("GetAwaitAwaiterInContinuationCall caller-%016" PRIX64 " sig-%s unsafe-%u " + "methodHnd-%016" PRIX64 " contextHandle-%016" PRIX64 " instArg %s", + key.callerHnd, + SpmiDumpHelper::DumpAgnostic_CORINFO_SIG_INFO( + key.callSig, GetAwaitAwaiterInContinuationCall, SigInstHandleMap).c_str(), + key.isUnsafe, + value.methodHnd, + value.contextHandle, + SpmiDumpHelper::DumpAgnostic_CORINFO_LOOKUP(value.instArg).c_str()); +} + +CORINFO_METHOD_HANDLE MethodContext::repGetAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHnd, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) +{ + Agnostic_GetAwaitAwaiterInContinuationCall key; + ZeroMemory(&key, sizeof(key)); + key.callerHnd = CastHandle(callerHnd); + key.callSig = SpmiRecordsHelper::RestoreAgnostic_CORINFO_SIG_INFO( + *callSig, GetAwaitAwaiterInContinuationCall, SigInstHandleMap); + key.isUnsafe = isUnsafe; + + Agnostic_GetAwaitReturnCallResult value = + LookupByKeyOrMissNoMessage(GetAwaitAwaiterInContinuationCall, key); + DEBUG_REP(dmpGetAwaitAwaiterInContinuationCall(key, value)); + *contextHandle = (CORINFO_CONTEXT_HANDLE)value.contextHandle; + *instArg = SpmiRecordsHelper::RestoreCORINFO_LOOKUP(value.instArg); + return (CORINFO_METHOD_HANDLE)value.methodHnd; +} + void MethodContext::recGetGSCookie(GSCookie* pCookieVal, GSCookie** ppCookieVal) { if (GetGSCookie == nullptr) diff --git a/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h b/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h index 32eb7aef721b6d..7648249e5a2377 100644 --- a/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h +++ b/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h @@ -567,6 +567,20 @@ class MethodContext void dmpGetAwaitReturnCall(DWORDLONG key, Agnostic_GetAwaitReturnCallResult& value); CORINFO_METHOD_HANDLE repGetAwaitReturnCall(CORINFO_METHOD_HANDLE callerHnd, CORINFO_CONTEXT_HANDLE* contextHandle, CORINFO_LOOKUP* instArg); + void recGetAwaitAwaiterInContinuationCall(CORINFO_METHOD_HANDLE callerHnd, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg, + CORINFO_METHOD_HANDLE methHnd); + void dmpGetAwaitAwaiterInContinuationCall(const Agnostic_GetAwaitAwaiterInContinuationCall& key, + Agnostic_GetAwaitReturnCallResult& value); + CORINFO_METHOD_HANDLE repGetAwaitAwaiterInContinuationCall(CORINFO_METHOD_HANDLE callerHnd, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg); + void recGetWasmWellKnownGlobals(const CORINFO_WASM_WELLKNOWN_GLOBALS* pBaseGlobals); void dmpGetWasmWellKnownGlobals(DWORD key, const Agnostic_CORINFO_WASM_WELLKNOWN_GLOBALS& value); void repGetWasmWellKnownGlobals(CORINFO_WASM_WELLKNOWN_GLOBALS* pWellKnownGlobalsOut); @@ -1228,7 +1242,8 @@ enum mcPackets Packet_GetAwaitReturnCall = 238, Packet_GetAddressAlignment = 239, Packet_GetWasmWellKnownGlobals = 240, - Packet_CanValueClassInstancePointerEscape = 241, + Packet_GetAwaitAwaiterInContinuationCall = 241, + Packet_CanValueClassInstancePointerEscape = 242, }; void SetDebugDumpVariables(); diff --git a/src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp b/src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp index 9a708cd3f7b420..f1d9cfad2fad54 100644 --- a/src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp +++ b/src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp @@ -1404,6 +1404,20 @@ CORINFO_METHOD_HANDLE interceptor_ICJI::getAwaitReturnCall(CORINFO_METHOD_HANDLE return result; } +CORINFO_METHOD_HANDLE interceptor_ICJI::getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) +{ + mc->cr->AddCall("getAwaitAwaiterInContinuationCall"); + CORINFO_METHOD_HANDLE result = original_ICorJitInfo->getAwaitAwaiterInContinuationCall( + callerHandle, callSig, isUnsafe, contextHandle, instArg); + mc->recGetAwaitAwaiterInContinuationCall(callerHandle, callSig, isUnsafe, contextHandle, instArg, result); + return result; +} + /*********************************************************************************/ // // Diagnostic methods diff --git a/src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp b/src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp index 931dad3bb9cce0..14a88356f7a280 100644 --- a/src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp +++ b/src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp @@ -985,6 +985,17 @@ CORINFO_METHOD_HANDLE interceptor_ICJI::getAwaitReturnCall( return original_ICorJitInfo->getAwaitReturnCall(callerHandle, contextHandle, instArg); } +CORINFO_METHOD_HANDLE interceptor_ICJI::getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) +{ + mcs->AddCall("getAwaitAwaiterInContinuationCall"); + return original_ICorJitInfo->getAwaitAwaiterInContinuationCall(callerHandle, callSig, isUnsafe, contextHandle, instArg); +} + mdMethodDef interceptor_ICJI::getMethodDefFromMethod( CORINFO_METHOD_HANDLE hMethod) { diff --git a/src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp b/src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp index 023658229bbf74..2d6199fae28c6a 100644 --- a/src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp +++ b/src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp @@ -863,6 +863,16 @@ CORINFO_METHOD_HANDLE interceptor_ICJI::getAwaitReturnCall( return original_ICorJitInfo->getAwaitReturnCall(callerHandle, contextHandle, instArg); } +CORINFO_METHOD_HANDLE interceptor_ICJI::getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) +{ + return original_ICorJitInfo->getAwaitAwaiterInContinuationCall(callerHandle, callSig, isUnsafe, contextHandle, instArg); +} + mdMethodDef interceptor_ICJI::getMethodDefFromMethod( CORINFO_METHOD_HANDLE hMethod) { diff --git a/src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp b/src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp index e5d65f8e046f0a..15b4c8d0d0c398 100644 --- a/src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp +++ b/src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp @@ -1214,6 +1214,18 @@ CORINFO_METHOD_HANDLE MyICJI::getAwaitReturnCall(CORINFO_METHOD_HANDLE callerHan return jitInstance->mc->repGetAwaitReturnCall(callerHandle, contextHandle, instArg); } +CORINFO_METHOD_HANDLE MyICJI::getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) +{ + jitInstance->mc->cr->AddCall("getAwaitAwaiterInContinuationCall"); + return jitInstance->mc->repGetAwaitAwaiterInContinuationCall( + callerHandle, callSig, isUnsafe, contextHandle, instArg); +} + /*********************************************************************************/ // // Diagnostic methods diff --git a/src/coreclr/vm/corelib.h b/src/coreclr/vm/corelib.h index 84f6ff9bf88230..774cd84cd32d97 100644 --- a/src/coreclr/vm/corelib.h +++ b/src/coreclr/vm/corelib.h @@ -725,11 +725,15 @@ DEFINE_METHOD(ASYNC_HELPERS, TRANSPARENT_AWAIT_TASK, Transpa DEFINE_METHOD(ASYNC_HELPERS, TRANSPARENT_AWAIT_VALUETASK, TransparentAwait, SM_ValueTask_RetVoid) DEFINE_METHOD(ASYNC_HELPERS, TRANSPARENT_AWAIT_TASK_OF_T, TransparentAwait, GM_TaskOfT_RetT) DEFINE_METHOD(ASYNC_HELPERS, TRANSPARENT_AWAIT_VALUETASK_OF_T, TransparentAwait, GM_ValueTaskOfT_RetT) +DEFINE_METHOD(ASYNC_HELPERS, AWAIT_AWAITER_IN_CONTINUATION, AwaitAwaiterInContinuation, NoSig) +DEFINE_METHOD(ASYNC_HELPERS, UNSAFE_AWAIT_AWAITER_IN_CONTINUATION, UnsafeAwaitAwaiterInContinuation, NoSig) DEFINE_METHOD(ASYNC_HELPERS, CAPTURE_EXECUTION_CONTEXT, CaptureExecutionContext, NoSig) DEFINE_METHOD(ASYNC_HELPERS, CAPTURE_CONTINUATION_CONTEXT, CaptureContinuationContext, NoSig) DEFINE_METHOD(ASYNC_HELPERS, CAPTURE_CONTEXTS, CaptureContexts, NoSig) DEFINE_METHOD(ASYNC_HELPERS, RESTORE_CONTEXTS, RestoreContexts, NoSig) DEFINE_METHOD(ASYNC_HELPERS, RESTORE_CONTEXTS_ON_SUSPENSION, RestoreContextsOnSuspension, NoSig) +DEFINE_METHOD(ASYNC_HELPERS, RESTORE_INLINED_FRAME_CONTEXTS, RestoreInlinedFrameContexts, NoSig) +DEFINE_METHOD(ASYNC_HELPERS, CAPTURE_INLINED_FRAME_TRANSITION, CaptureInlinedFrameTransition, NoSig) DEFINE_METHOD(ASYNC_HELPERS, FINISH_SUSPENSION_NO_CONTINUATION_CONTEXT, FinishSuspensionNoContinuationContext, NoSig) DEFINE_METHOD(ASYNC_HELPERS, FINISH_SUSPENSION_WITH_CONTINUATION_CONTEXT, FinishSuspensionWithContinuationContext, NoSig) DEFINE_METHOD(ASYNC_HELPERS, ASYNC_CALL_CONTINUATION, AsyncCallContinuation, NoSig) diff --git a/src/coreclr/vm/jitinterface.cpp b/src/coreclr/vm/jitinterface.cpp index c8ffc31b0d3583..1a1f9fbb290e0a 100644 --- a/src/coreclr/vm/jitinterface.cpp +++ b/src/coreclr/vm/jitinterface.cpp @@ -10445,6 +10445,8 @@ void CEEInfo::getAsyncInfo(CORINFO_ASYNC_INFO* pAsyncInfoOut) pAsyncInfoOut->captureContextsMethHnd = CORINFO_METHOD_HANDLE(CoreLibBinder::GetMethod(METHOD__ASYNC_HELPERS__CAPTURE_CONTEXTS)); pAsyncInfoOut->restoreContextsMethHnd = CORINFO_METHOD_HANDLE(CoreLibBinder::GetMethod(METHOD__ASYNC_HELPERS__RESTORE_CONTEXTS)); pAsyncInfoOut->restoreContextsOnSuspensionMethHnd = CORINFO_METHOD_HANDLE(CoreLibBinder::GetMethod(METHOD__ASYNC_HELPERS__RESTORE_CONTEXTS_ON_SUSPENSION)); + pAsyncInfoOut->restoreInlinedFrameContextsMethHnd = CORINFO_METHOD_HANDLE(CoreLibBinder::GetMethod(METHOD__ASYNC_HELPERS__RESTORE_INLINED_FRAME_CONTEXTS)); + pAsyncInfoOut->captureInlinedFrameTransitionMethHnd = CORINFO_METHOD_HANDLE(CoreLibBinder::GetMethod(METHOD__ASYNC_HELPERS__CAPTURE_INLINED_FRAME_TRANSITION)); pAsyncInfoOut->finishSuspensionNoContinuationContextMethHnd = CORINFO_METHOD_HANDLE(CoreLibBinder::GetMethod(METHOD__ASYNC_HELPERS__FINISH_SUSPENSION_NO_CONTINUATION_CONTEXT)); pAsyncInfoOut->finishSuspensionWithContinuationContextMethHnd = CORINFO_METHOD_HANDLE(CoreLibBinder::GetMethod(METHOD__ASYNC_HELPERS__FINISH_SUSPENSION_WITH_CONTINUATION_CONTEXT)); @@ -10530,6 +10532,60 @@ CORINFO_METHOD_HANDLE CEEInfo::getAwaitReturnCall(CORINFO_METHOD_HANDLE callerHa return CORINFO_METHOD_HANDLE(pMD); } +CORINFO_METHOD_HANDLE CEEInfo::getAwaitAwaiterInContinuationCall( + CORINFO_METHOD_HANDLE callerHandle, + CORINFO_SIG_INFO* callSig, + bool isUnsafe, + CORINFO_CONTEXT_HANDLE* contextHandle, + CORINFO_LOOKUP* instArg) +{ + CONTRACTL { + THROWS; + GC_TRIGGERS; + MODE_PREEMPTIVE; + } CONTRACTL_END; + + MethodDesc* pMD = NULL; + + JIT_TO_EE_TRANSITION(); + + _ASSERTE(callSig->sigInst.methInstCount == 1); + TypeHandle awaiterType(callSig->sigInst.methInst[0]); + + instArg->lookupKind.needsRuntimeLookup = false; + instArg->constLookup.accessType = IAT_VALUE; + instArg->constLookup.addr = NULL; + + MethodDesc* pCallerMD = GetMethod(callerHandle); + MethodDesc* pTypicalAwaitMD = CoreLibBinder::GetMethod( + isUnsafe ? METHOD__ASYNC_HELPERS__UNSAFE_AWAIT_AWAITER_IN_CONTINUATION + : METHOD__ASYNC_HELPERS__AWAIT_AWAITER_IN_CONTINUATION); + pMD = MethodDesc::FindOrCreateAssociatedMethodDesc(pTypicalAwaitMD, pTypicalAwaitMD->GetMethodTable(), FALSE, + Instantiation(&awaiterType, 1), TRUE); + + MethodDesc* pInliningContext = pMD; + if (pMD->RequiresInstArg()) + { + if (awaiterType.IsCanonicalSubtype()) + { + ComputeRuntimeLookupForAwaitAwaiterInContinuationCall(pCallerMD, pTypicalAwaitMD, callSig, instArg); + } + else + { + MethodDesc* pContext = MethodDesc::FindOrCreateAssociatedMethodDesc( + pTypicalAwaitMD, pTypicalAwaitMD->GetMethodTable(), FALSE, Instantiation(&awaiterType, 1), FALSE); + instArg->constLookup.addr = pContext; + pInliningContext = pContext; + } + } + + *contextHandle = MAKE_METHODCONTEXT(pInliningContext); + + EE_TO_JIT_TRANSITION(); + + return CORINFO_METHOD_HANDLE(pMD); +} + // Compute the runtime lookup for the instantiation argument for an // AsyncHelpers.TransparentAwait call, to be used for wrapping a return value // from pCallerMD for its runtime async version. @@ -10594,6 +10650,65 @@ void CEEInfo::ComputeRuntimeLookupForAwaitCall(MethodDesc* pCallerMD, MethodDesc FinishComputeRuntimeLookup(sigBuilder, pCallerMD, lookup); } +void CEEInfo::ComputeRuntimeLookupForAwaitAwaiterInContinuationCall( + MethodDesc* pCallerMD, + MethodDesc* pTypicalAwaitMD, + CORINFO_SIG_INFO* callSig, + CORINFO_LOOKUP* lookup) +{ + lookup->lookupKind.needsRuntimeLookup = true; + + CORINFO_RUNTIME_LOOKUP* rlookup = &lookup->runtimeLookup; + rlookup->signature = NULL; + rlookup->indirectFirstOffset = 0; + rlookup->indirectSecondOffset = 0; + rlookup->sizeOffset = CORINFO_NO_SIZE_CHECK; + rlookup->indirections = CORINFO_USEHELPER; + + if (pCallerMD->RequiresInstMethodDescArg()) + { + lookup->lookupKind.runtimeLookupKind = CORINFO_LOOKUP_METHODPARAM; + rlookup->helper = CORINFO_HELP_RUNTIMEHANDLE_METHOD; + } + else if (pCallerMD->RequiresInstMethodTableArg()) + { + lookup->lookupKind.runtimeLookupKind = CORINFO_LOOKUP_CLASSPARAM; + rlookup->helper = CORINFO_HELP_RUNTIMEHANDLE_CLASS; + } + else + { + lookup->lookupKind.runtimeLookupKind = CORINFO_LOOKUP_THISOBJ; + rlookup->helper = CORINFO_HELP_RUNTIMEHANDLE_CLASS; + } + + SigBuilder sigBuilder; + sigBuilder.AppendData(MethodDescSlot); + if (lookup->lookupKind.runtimeLookupKind != CORINFO_LOOKUP_METHODPARAM) + { + sigBuilder.AppendData(pCallerMD->GetMethodTable()->GetNumDicts() - 1); + } + + sigBuilder.AppendElementType(ELEMENT_TYPE_INTERNAL); + sigBuilder.AppendPointer(pTypicalAwaitMD->GetMethodTable()); + sigBuilder.AppendData(ENCODE_METHOD_SIG_MethodInstantiation | ENCODE_METHOD_SIG_InstantiatingStub); + sigBuilder.AppendElementType(ELEMENT_TYPE_INTERNAL); + sigBuilder.AppendPointer(pTypicalAwaitMD->GetMethodTable()); + sigBuilder.AppendData(RidFromToken(pTypicalAwaitMD->GetMemberDef())); + sigBuilder.AppendData(1); + + _ASSERTE(callSig->pSig != NULL); + SigPointer instantiation(callSig->pSig, callSig->cbSig); + BYTE callingConvention; + IfFailThrow(instantiation.GetByte(&callingConvention)); + _ASSERTE(callingConvention == IMAGE_CEE_CS_CALLCONV_GENERICINST); + uint32_t numArgs; + IfFailThrow(instantiation.GetData(&numArgs)); + _ASSERTE(numArgs == 1); + instantiation.ConvertToInternalExactlyOne(GetModule(callSig->scope), NULL, &sigBuilder); + + FinishComputeRuntimeLookup(sigBuilder, pCallerMD, lookup); +} + static MethodTable* getContinuationType( size_t dataSize, bool* objRefs, diff --git a/src/coreclr/vm/jitinterface.h b/src/coreclr/vm/jitinterface.h index d84a9315fc0049..755b96707db08d 100644 --- a/src/coreclr/vm/jitinterface.h +++ b/src/coreclr/vm/jitinterface.h @@ -413,6 +413,11 @@ class CEEInfo : public ICorJitInfo CORINFO_LOOKUP* pResultLookup); void ComputeRuntimeLookupForAwaitCall(MethodDesc* pCallerMD, MethodDesc* pTypicalAwaitMD, CORINFO_LOOKUP* lookup); + void ComputeRuntimeLookupForAwaitAwaiterInContinuationCall( + MethodDesc* pCallerMD, + MethodDesc* pTypicalAwaitMD, + CORINFO_SIG_INFO* callSig, + CORINFO_LOOKUP* lookup); #if defined(FEATURE_GDBJIT) CalledMethod * GetCalledMethods() { return m_pCalledMethods; } diff --git a/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs b/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs index 7e11855f9b82d0..9e638aa9204141 100644 --- a/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs +++ b/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs @@ -36,6 +36,29 @@ public static unsafe void AwaitAwaiter(TAwaiter awaiter) where TAwaite AsyncSuspend(sentinelContinuation); } + [BypassReadyToRun] + [MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.Async)] + [StackTraceHidden] + private static unsafe void AwaitAwaiterInContinuation(int offset) + where TAwaiter : INotifyCompletion + { + ref RuntimeAsyncAwaitState state = ref t_runtimeAsyncAwaitState; + Continuation? sentinelContinuation = state.SentinelContinuation ??= new Continuation(); + state.StackState->AwaiterContinuation = &AwaiterOnCompletedFromContinuation; + state.StackState->AwaiterOffset = offset; + state.CaptureContexts(); + AsyncSuspend(sentinelContinuation); + } + + private static void AwaiterOnCompletedFromContinuation( + Continuation headContinuation, int offset, Action continuation) + where TAwaiter : INotifyCompletion + { + ref byte data = ref RuntimeHelpers.GetRawData(headContinuation); + TAwaiter awaiter = Unsafe.As(ref Unsafe.Add(ref data, offset)); + awaiter.OnCompleted(continuation); + } + // Must be NoInlining because we use AsyncSuspend to manufacture an explicit suspension point. // It will not capture/restore any local state that is live across it. @@ -70,6 +93,35 @@ public static unsafe void UnsafeAwaitAwaiter(TAwaiter awaiter) where T AsyncSuspend(sentinelContinuation); } + [BypassReadyToRun] + [MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.Async)] + [StackTraceHidden] + private static unsafe void UnsafeAwaitAwaiterInContinuation(int offset) + where TAwaiter : ICriticalNotifyCompletion + { + ref RuntimeAsyncAwaitState state = ref t_runtimeAsyncAwaitState; + Continuation? sentinelContinuation = state.SentinelContinuation ??= new Continuation(); + state.StackState->AwaiterContinuation = &UnsafeAwaiterOnCompletedFromContinuation; + state.StackState->AwaiterOffset = offset; + state.CaptureContexts(); + AsyncSuspend(sentinelContinuation); + } + + private static void UnsafeAwaiterOnCompletedFromContinuation( + Continuation headContinuation, int offset, Action continuation) + where TAwaiter : ICriticalNotifyCompletion + { + if (typeof(TAwaiter) == typeof(YieldAwaitable.YieldAwaiter)) + { + RuntimeAsyncYielder.Instance.UnsafeOnCompleted(continuation); + return; + } + + ref byte data = ref RuntimeHelpers.GetRawData(headContinuation); + TAwaiter awaiter = Unsafe.As(ref Unsafe.Add(ref data, offset)); + awaiter.UnsafeOnCompleted(continuation); + } + /// /// Awaits the specified and returns its result, throwing any exception produced by the task. /// diff --git a/src/tests/JIT/Regression/JitBlue/GitHub_130437/GitHub_130437.csproj b/src/tests/JIT/Regression/JitBlue/GitHub_130437/GitHub_130437.csproj index 8f66634434cd1a..f07af3a622402e 100644 --- a/src/tests/JIT/Regression/JitBlue/GitHub_130437/GitHub_130437.csproj +++ b/src/tests/JIT/Regression/JitBlue/GitHub_130437/GitHub_130437.csproj @@ -1,4 +1,4 @@ - + $(RepoRoot)src/coreclr/.nuget/Microsoft.NET.Sdk.IL/Microsoft.NET.Sdk.IL.targets.template True diff --git a/src/tests/async/awaitingnoasyncgeneric/awaitingnoasyncgeneric.csproj b/src/tests/async/awaitingnoasyncgeneric/awaitingnoasyncgeneric.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/awaitingnoasyncgeneric/awaitingnoasyncgeneric.csproj +++ b/src/tests/async/awaitingnoasyncgeneric/awaitingnoasyncgeneric.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/awaitingnotasync/awaitingnotasync.csproj b/src/tests/async/awaitingnotasync/awaitingnotasync.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/awaitingnotasync/awaitingnotasync.csproj +++ b/src/tests/async/awaitingnotasync/awaitingnotasync.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/byref-param/byref-param.csproj b/src/tests/async/byref-param/byref-param.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/byref-param/byref-param.csproj +++ b/src/tests/async/byref-param/byref-param.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/cancellation/cancellation.csproj b/src/tests/async/cancellation/cancellation.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/cancellation/cancellation.csproj +++ b/src/tests/async/cancellation/cancellation.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/collectible-alc/collectible-alc.csproj b/src/tests/async/collectible-alc/collectible-alc.csproj index 5bc9c9b10779eb..e65e440150c631 100644 --- a/src/tests/async/collectible-alc/collectible-alc.csproj +++ b/src/tests/async/collectible-alc/collectible-alc.csproj @@ -1,4 +1,4 @@ - + True diff --git a/src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs b/src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs new file mode 100644 index 00000000000000..6946cb6975745b --- /dev/null +++ b/src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs @@ -0,0 +1,85 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Xunit; + +public class CustomStructAwaiters +{ + private static int s_safeAwaiterValue; + private static int s_unsafeAwaiterValue; + + [Fact] + public static void TestEntryPoint() + { + Run().Wait(); + } + + private static async Task Run() + { + Volatile.Write(ref s_safeAwaiterValue, 0); + await new SafeAwaitable(42); + Assert.Equal(42, Volatile.Read(ref s_safeAwaiterValue)); + + Volatile.Write(ref s_unsafeAwaiterValue, 0); + await new UnsafeAwaitable(43); + Assert.Equal(43, Volatile.Read(ref s_unsafeAwaiterValue)); + } + + private readonly struct SafeAwaitable + { + private readonly int _value; + + public SafeAwaitable(int value) => _value = value; + + public SafeAwaiter GetAwaiter() => new SafeAwaiter(_value); + + public readonly struct SafeAwaiter : INotifyCompletion + { + private readonly int _value; + + public SafeAwaiter(int value) => _value = value; + + public bool IsCompleted => false; + + public void OnCompleted(Action continuation) + { + Volatile.Write(ref s_safeAwaiterValue, _value); + ThreadPool.QueueUserWorkItem(static state => ((Action)state!).Invoke(), continuation); + } + + public void GetResult() => Assert.Equal(42, _value); + } + } + + private readonly struct UnsafeAwaitable + { + private readonly int _value; + + public UnsafeAwaitable(int value) => _value = value; + + public UnsafeAwaiter GetAwaiter() => new UnsafeAwaiter(_value); + + public readonly struct UnsafeAwaiter : ICriticalNotifyCompletion + { + private readonly int _value; + + public UnsafeAwaiter(int value) => _value = value; + + public bool IsCompleted => false; + + public void OnCompleted(Action continuation) => throw new InvalidOperationException(); + + public void UnsafeOnCompleted(Action continuation) + { + Volatile.Write(ref s_unsafeAwaiterValue, _value); + ThreadPool.UnsafeQueueUserWorkItem(static state => ((Action)state!).Invoke(), continuation); + } + + public void GetResult() => Assert.Equal(43, _value); + } + } +} diff --git a/src/tests/async/custom-struct-awaiters/custom-struct-awaiters.csproj b/src/tests/async/custom-struct-awaiters/custom-struct-awaiters.csproj new file mode 100644 index 00000000000000..197767e2c4e249 --- /dev/null +++ b/src/tests/async/custom-struct-awaiters/custom-struct-awaiters.csproj @@ -0,0 +1,5 @@ + + + + + diff --git a/src/tests/async/eh-microbench/eh-microbench.csproj b/src/tests/async/eh-microbench/eh-microbench.csproj index e0c0b671ef1829..6650af6b67607c 100644 --- a/src/tests/async/eh-microbench/eh-microbench.csproj +++ b/src/tests/async/eh-microbench/eh-microbench.csproj @@ -1,4 +1,4 @@ - + BuildOnly diff --git a/src/tests/async/execution-context/execution-context.csproj b/src/tests/async/execution-context/execution-context.csproj index 644025b102673e..3a8b32c0ae764d 100644 --- a/src/tests/async/execution-context/execution-context.csproj +++ b/src/tests/async/execution-context/execution-context.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/gc-roots-scan/gc-roots-scan.csproj b/src/tests/async/gc-roots-scan/gc-roots-scan.csproj index 638c237d91983d..21cc479969f131 100644 --- a/src/tests/async/gc-roots-scan/gc-roots-scan.csproj +++ b/src/tests/async/gc-roots-scan/gc-roots-scan.csproj @@ -1,4 +1,4 @@ - + BuildOnly diff --git a/src/tests/async/implement/implement.csproj b/src/tests/async/implement/implement.csproj index 644025b102673e..3a8b32c0ae764d 100644 --- a/src/tests/async/implement/implement.csproj +++ b/src/tests/async/implement/implement.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/inlined-frame-contexts/inlined-frame-contexts.cs b/src/tests/async/inlined-frame-contexts/inlined-frame-contexts.cs new file mode 100644 index 00000000000000..0e078d81193d99 --- /dev/null +++ b/src/tests/async/inlined-frame-contexts/inlined-frame-contexts.cs @@ -0,0 +1,196 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Xunit; + +// Behaviors that must be preserved when runtime async calls are inlined. These are the +// worked examples from docs/design/coreclr/jit/runtime-async-inlining.md. +// +// Inlining removes both a call site and a callee, and with them the context handling the +// async infrastructure would otherwise have performed at each frame boundary. These tests +// pin down the observable results of that handling, so they must produce the same answers +// whether or not the calls are inlined. +// +// The inner frames are marked AggressiveInlining so that the inlining actually happens +// rather than being left to the profitability heuristic, which rejects these callees. +public class Async2InlinedFrameContexts +{ + private sealed class NamedContext : SynchronizationContext + { + public readonly string Name; + + public NamedContext(string name) => Name = name; + + public override void Post(SendOrPostCallback d, object? state) + { + SynchronizationContext currentCtx = Current; + SetSynchronizationContext(this); + try + { + d(state); + } + finally + { + SetSynchronizationContext(currentCtx); + } + } + } + + // Always suspends and resumes on a thread pool thread, so resumption never happens on + // whatever context the awaiting frames were running in. + private struct AlwaysThreadPoolAwaitable : INotifyCompletion + { + public bool IsCompleted => false; + + public void OnCompleted(Action continuation) + { + ThreadPool.QueueUserWorkItem(_ => continuation()); + } + + public void GetResult() + { + } + + public AlwaysThreadPoolAwaitable GetAwaiter() => this; + } + + private static readonly NamedContext s_syncContext1 = new NamedContext("ctx1"); + private static readonly NamedContext s_syncContext2 = new NamedContext("ctx2"); + + private static string CurrentContextName => (SynchronizationContext.Current as NamedContext)?.Name ?? "none"; + + // Each frame sets its own synchronization context before awaiting, so each frame's + // continuation captures a different one. After the innermost await resumes on the + // thread pool, every frame must be back on the context it captured by the time it + // observes it again. + private static string s_fooContextAfterAwait = ""; + private static string s_barContextAfterAwait = ""; + + private static async Task ContextFoo() + { + SynchronizationContext.SetSynchronizationContext(s_syncContext1); + await ContextBar(); + s_fooContextAfterAwait = CurrentContextName; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static async Task ContextBar() + { + SynchronizationContext.SetSynchronizationContext(s_syncContext2); + await ContextBaz(); + s_barContextAfterAwait = CurrentContextName; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static async Task ContextBaz() + { + await new AlwaysThreadPoolAwaitable(); + } + + [Fact] + public static void SynchronizationContextIsRestoredPerFrame() + { + SynchronizationContext original = SynchronizationContext.Current; + try + { + ContextFoo().GetAwaiter().GetResult(); + } + finally + { + SynchronizationContext.SetSynchronizationContext(original); + } + + // Bar awaited Baz having set ctx2, so Bar resumes on ctx2. Foo awaited Bar having + // set ctx1, so Foo resumes on ctx1. + Assert.Equal("ctx2", s_barContextAfterAwait); + Assert.Equal("ctx1", s_fooContextAfterAwait); + } + + private static readonly AsyncLocal s_local = new AsyncLocal(); + + private static int s_bazLocalAfterAwait; + private static int s_barLocalAfterAwait; + private static int s_fooLocalAfterAwait; + + private static async Task LocalFoo() + { + s_local.Value = 1; + await LocalBar(); + s_fooLocalAfterAwait = s_local.Value; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static async Task LocalBar() + { + s_local.Value = 2; + await LocalBaz(); + s_barLocalAfterAwait = s_local.Value; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static async Task LocalBaz() + { + s_local.Value = 3; + await new AlwaysThreadPoolAwaitable(); + s_bazLocalAfterAwait = s_local.Value; + } + + [Fact] + public static void ExecutionContextIsRestoredPerFrame() + { + LocalFoo().GetAwaiter().GetResult(); + + // Each frame sees the AsyncLocal value it set, because every frame boundary + // restores the ExecutionContext that frame captured. + Assert.Equal(3, s_bazLocalAfterAwait); + Assert.Equal(2, s_barLocalAfterAwait); + Assert.Equal(1, s_fooLocalAfterAwait); + } + + // The same, but with the awaits inside loops so the frames suspend and resume + // repeatedly. A frame's captured contexts must survive later suspensions of the + // frames nested inside it. + private static async Task LoopFoo(int iterations) + { + SynchronizationContext.SetSynchronizationContext(s_syncContext1); + s_local.Value = 1; + + for (int i = 0; i < iterations; i++) + { + await LoopBar(iterations); + Assert.Equal("ctx1", CurrentContextName); + Assert.Equal(1, s_local.Value); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static async Task LoopBar(int iterations) + { + SynchronizationContext.SetSynchronizationContext(s_syncContext2); + s_local.Value = 2; + + for (int i = 0; i < iterations; i++) + { + await new AlwaysThreadPoolAwaitable(); + Assert.Equal(2, s_local.Value); + } + } + + [Fact] + public static void ContextsSurviveRepeatedSuspensions() + { + SynchronizationContext original = SynchronizationContext.Current; + try + { + LoopFoo(4).GetAwaiter().GetResult(); + } + finally + { + SynchronizationContext.SetSynchronizationContext(original); + } + } +} diff --git a/src/tests/async/inlined-frame-contexts/inlined-frame-contexts.csproj b/src/tests/async/inlined-frame-contexts/inlined-frame-contexts.csproj new file mode 100644 index 00000000000000..5ca3f694332957 --- /dev/null +++ b/src/tests/async/inlined-frame-contexts/inlined-frame-contexts.csproj @@ -0,0 +1,14 @@ + + + + true + + + + + + + + + diff --git a/src/tests/async/inst-unbox-thunks/inst-unbox-thunks.csproj b/src/tests/async/inst-unbox-thunks/inst-unbox-thunks.csproj index 8e9fed2b02ad47..d06e5df35dd4a2 100644 --- a/src/tests/async/inst-unbox-thunks/inst-unbox-thunks.csproj +++ b/src/tests/async/inst-unbox-thunks/inst-unbox-thunks.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/mincallcost-microbench/mincallcost-microbench.csproj b/src/tests/async/mincallcost-microbench/mincallcost-microbench.csproj index e0c0b671ef1829..6650af6b67607c 100644 --- a/src/tests/async/mincallcost-microbench/mincallcost-microbench.csproj +++ b/src/tests/async/mincallcost-microbench/mincallcost-microbench.csproj @@ -1,4 +1,4 @@ - + BuildOnly diff --git a/src/tests/async/object/object.csproj b/src/tests/async/object/object.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/object/object.csproj +++ b/src/tests/async/object/object.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/objects-captured/objects-captured.csproj b/src/tests/async/objects-captured/objects-captured.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/objects-captured/objects-captured.csproj +++ b/src/tests/async/objects-captured/objects-captured.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/osr-inlined-contexts/osr-inlined-contexts.cs b/src/tests/async/osr-inlined-contexts/osr-inlined-contexts.cs new file mode 100644 index 00000000000000..c6af045972fd4b --- /dev/null +++ b/src/tests/async/osr-inlined-contexts/osr-inlined-contexts.cs @@ -0,0 +1,125 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Xunit; + +// Coverage for runtime async inlining in methods that are rejitted via OSR. +// +// An OSR method is entered with the tier 0 method's continuation, whose layout differs +// from the one the OSR method computes. Reading an inlined frame's state off it would be +// wrong, so this pins down that inlining async callees into an OSR method still behaves: +// an inlined callee's contexts are its own, and resuming inside one keeps every frame's +// contexts intact. +public class Async2OsrInlinedContexts +{ + private sealed class MarkerContext : SynchronizationContext + { + // Re-establish this context around the callback, as a real UI-style context would. + // The default implementation queues to the thread pool without doing so, which + // would leave the continuation with no current context at all. + public override void Post(SendOrPostCallback d, object? state) + { + SynchronizationContext currentCtx = Current; + SetSynchronizationContext(this); + try + { + d(state); + } + finally + { + SetSynchronizationContext(currentCtx); + } + } + } + + // Enough iterations to get the enclosing method rejitted via OSR. + private const int Iterations = 100_000; + + private static int s_sideEffect; + + // Async, and so has its own context save and restore, but never suspends, so it is + // inlinable into the loop below. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static async Task BumpAsync(int value) + { + s_sideEffect += value; + } + + private static async Task LoopWithInlinedCallee() + { + int total = 0; + for (int i = 0; i < Iterations; i++) + { + await BumpAsync(1); + total += i; + } + + return total; + } + + [Fact] + public static void InlinedCalleeDoesNotClobberContextsInOsrMethod() + { + SynchronizationContext original = SynchronizationContext.Current; + MarkerContext marker = new MarkerContext(); + try + { + SynchronizationContext.SetSynchronizationContext(marker); + + LoopWithInlinedCallee().GetAwaiter().GetResult(); + + // The inlined callee has its own context save and restore, which must not + // disturb the caller's. + Assert.Same(marker, SynchronizationContext.Current); + } + finally + { + SynchronizationContext.SetSynchronizationContext(original); + } + } + + private static readonly AsyncLocal s_local = new AsyncLocal(); + + // Same, but the callee suspends, so the loop also exercises resuming inside an inlined + // frame of an OSR method. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static async Task YieldAsync() + { + await Task.Yield(); + } + + private static async Task LoopWithSuspendingInlinedCallee() + { + SynchronizationContext.SetSynchronizationContext(new MarkerContext()); + s_local.Value = 7; + + int total = 0; + for (int i = 0; i < 2_000; i++) + { + await YieldAsync(); + Assert.IsType(SynchronizationContext.Current); + Assert.Equal(7, s_local.Value); + total += i; + } + + return total; + } + + [Fact] + public static void ResumingInsideInlinedFrameOfOsrMethodKeepsContexts() + { + SynchronizationContext original = SynchronizationContext.Current; + try + { + LoopWithSuspendingInlinedCallee().GetAwaiter().GetResult(); + } + finally + { + SynchronizationContext.SetSynchronizationContext(original); + } + } +} diff --git a/src/tests/async/osr-inlined-contexts/osr-inlined-contexts.csproj b/src/tests/async/osr-inlined-contexts/osr-inlined-contexts.csproj new file mode 100644 index 00000000000000..8525b085aae062 --- /dev/null +++ b/src/tests/async/osr-inlined-contexts/osr-inlined-contexts.csproj @@ -0,0 +1,18 @@ + + + + true + True + + + + + + + + + + + + diff --git a/src/tests/async/override/override.csproj b/src/tests/async/override/override.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/override/override.csproj +++ b/src/tests/async/override/override.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/pgo/pgo.csproj b/src/tests/async/pgo/pgo.csproj index ff94a80928aef7..7c7111c8cd7ee1 100644 --- a/src/tests/async/pgo/pgo.csproj +++ b/src/tests/async/pgo/pgo.csproj @@ -1,6 +1,6 @@ - + - + \ No newline at end of file diff --git a/src/tests/async/pinvoke/pinvoke.csproj b/src/tests/async/pinvoke/pinvoke.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/pinvoke/pinvoke.csproj +++ b/src/tests/async/pinvoke/pinvoke.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/regression/125042.csproj b/src/tests/async/regression/125042.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/regression/125042.csproj +++ b/src/tests/async/regression/125042.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/regression/125615.csproj b/src/tests/async/regression/125615.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/regression/125615.csproj +++ b/src/tests/async/regression/125615.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/regression/125805.csproj b/src/tests/async/regression/125805.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/regression/125805.csproj +++ b/src/tests/async/regression/125805.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/regression/128566.csproj b/src/tests/async/regression/128566.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/regression/128566.csproj +++ b/src/tests/async/regression/128566.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/regression/managed-thread-id.csproj b/src/tests/async/regression/managed-thread-id.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/regression/managed-thread-id.csproj +++ b/src/tests/async/regression/managed-thread-id.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/returns/returns.csproj b/src/tests/async/returns/returns.csproj index ff94a80928aef7..7548a677889c52 100644 --- a/src/tests/async/returns/returns.csproj +++ b/src/tests/async/returns/returns.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/shared-generic/shared-generic.csproj b/src/tests/async/shared-generic/shared-generic.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/shared-generic/shared-generic.csproj +++ b/src/tests/async/shared-generic/shared-generic.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/staticvirtual/staticvirtual.csproj b/src/tests/async/staticvirtual/staticvirtual.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/staticvirtual/staticvirtual.csproj +++ b/src/tests/async/staticvirtual/staticvirtual.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/struct/struct.cs b/src/tests/async/struct/struct.cs index b8d8ab53b14077..df393687e5216e 100644 --- a/src/tests/async/struct/struct.cs +++ b/src/tests/async/struct/struct.cs @@ -32,7 +32,6 @@ private static async Task Async2() AssertEqual(100, s.Value); } - [MethodImpl(MethodImplOptions.NoInlining)] private static void AssertEqual(int expected, int val) { diff --git a/src/tests/async/struct/struct.csproj b/src/tests/async/struct/struct.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/struct/struct.csproj +++ b/src/tests/async/struct/struct.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/synchronization-context/synchronization-context.csproj b/src/tests/async/synchronization-context/synchronization-context.csproj index 644025b102673e..3a8b32c0ae764d 100644 --- a/src/tests/async/synchronization-context/synchronization-context.csproj +++ b/src/tests/async/synchronization-context/synchronization-context.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/valuetask-source/valuetask-source-stub.csproj b/src/tests/async/valuetask-source/valuetask-source-stub.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/valuetask-source/valuetask-source-stub.csproj +++ b/src/tests/async/valuetask-source/valuetask-source-stub.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/valuetask-source/valuetask-source.csproj b/src/tests/async/valuetask-source/valuetask-source.csproj index 7e637235662d05..c2c78be9d50708 100644 --- a/src/tests/async/valuetask-source/valuetask-source.csproj +++ b/src/tests/async/valuetask-source/valuetask-source.csproj @@ -1,4 +1,4 @@ - + true diff --git a/src/tests/async/valuetask/valuetask.csproj b/src/tests/async/valuetask/valuetask.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/valuetask/valuetask.csproj +++ b/src/tests/async/valuetask/valuetask.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/varying-yields/varying-yields-task.csproj b/src/tests/async/varying-yields/varying-yields-task.csproj index a063851b97b392..b80a06405ee432 100644 --- a/src/tests/async/varying-yields/varying-yields-task.csproj +++ b/src/tests/async/varying-yields/varying-yields-task.csproj @@ -1,4 +1,4 @@ - + ASYNC1_TASK;$(DefineConstants) diff --git a/src/tests/async/varying-yields/varying-yields-valuetask.csproj b/src/tests/async/varying-yields/varying-yields-valuetask.csproj index 7679b339a8019b..9e82183ae9f70b 100644 --- a/src/tests/async/varying-yields/varying-yields-valuetask.csproj +++ b/src/tests/async/varying-yields/varying-yields-valuetask.csproj @@ -1,4 +1,4 @@ - + ASYNC1_VALUETASK;$(DefineConstants) diff --git a/src/tests/async/varying-yields/varying-yields.csproj b/src/tests/async/varying-yields/varying-yields.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/varying-yields/varying-yields.csproj +++ b/src/tests/async/varying-yields/varying-yields.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/void/void.csproj b/src/tests/async/void/void.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/void/void.csproj +++ b/src/tests/async/void/void.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/with-yields/with-yields.csproj b/src/tests/async/with-yields/with-yields.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/with-yields/with-yields.csproj +++ b/src/tests/async/with-yields/with-yields.csproj @@ -1,4 +1,4 @@ - + diff --git a/src/tests/async/without-yields/without-yields.csproj b/src/tests/async/without-yields/without-yields.csproj index 3fc50cde4b3443..197767e2c4e249 100644 --- a/src/tests/async/without-yields/without-yields.csproj +++ b/src/tests/async/without-yields/without-yields.csproj @@ -1,4 +1,4 @@ - +