Cache reference timestamps behind an IVsAsyncFileChangeEx2 watcher for snapshot reuse

docs/ide/file-watching.md

@@ -0,0 +1,66 @@
+# File watching in FSharp.Editor
+
+`vsintegration/src/FSharp.Editor/LanguageService/FileChangeWatcher.fs` is the F# counterpart of
+Roslyn's `FileChangeWatcher` and `ReferenceFileChangeTracker` (both internal to
+`Microsoft.VisualStudio.LanguageServices` and not exposed through `ExternalAccess.FSharp`).
+
+## What the workspace already gives us
+
+Not every on-disk change needs this watcher. A `-r:` that the Roslyn workspace holds as a
+`MetadataReference` is already watched by Roslyn: `ProjectSystemProjectFactory` advises every
+reference path, and when one changes it swaps the reference on the solution, which bumps
+`Project.Version`. `FSharpProjectOptionsReactor` sees that version through `isProjectInvalidated`,
+recomputes, and calls `InvalidateConfiguration` — measured in VS, that path wins the race against
+a watcher subscribed to the same file, because Roslyn batches over 500 ms where this tracker
+additionally debounces for 2 s.
+
+So a second subscription to the same reference set buys nothing. What the workspace does *not*
+cover is everything it has no document or reference for, and every stat FCS still performs
+internally:
+
+- `#load` sources of a script: not documents, not references, invisible to the workspace.
+- `IsReferencesInvalidated` on the incremental builder, which stats every reference on every
+  request.
+- `ReferencesOnDisk` on snapshot reuse, which did the same per comparison — the consumer below.
+
+## Shape
+
+- **Service.** `IVsAsyncFileChangeEx2`, obtained asynchronously; nothing ever blocks on the UI
+  thread. Callbacks arrive through `IVsFreeThreadedFileChangeEvents2` and stay on background
+  threads.
+- **Batching.** Subscribe/unsubscribe operations go through a single-consumer queue with a 500 ms
+  window (Roslyn's empirical value for solution open/close). Consecutive operations of the same
+  kind, and for file watches the same sink, are coalesced into one service call.
+- **Directory watches.** A context starts with recursive `.dll` watches on the places reference
+  assemblies live: `DOTNET_ROOT/packs` and the machine-wide `dotnet/packs`, the .NET Framework
+  reference assemblies, and the NuGet cache (`NUGET_PACKAGES` or `~/.nuget/packages`). A file
+  under one of them costs no cookie of its own. Roslyn does not watch the NuGet cache; a consumer
+  that watches every `-r:` uniformly wants it, since package assemblies are the bulk of them and
+  the alternative is a per-file advise for each.
+- **Per-file watches.** Paths outside those directories (project outputs, loose assemblies) get
+  an individual advise, ref-counted across consumers by `FSharpReferenceChangeTracker`. The
+  tracker also keeps the last-write stamp of each watched path (see Consumers).
+- **Debounce.** A rebuild writes a temp file and renames it, producing several notifications; the
+  tracker fires one callback per path after 2 s of quiet.
+
+## Consumers
+
+**Snapshot reference stamps.** `FSharpProjectOptionsReactor` watches the `-r:` set of every
+project it computes options for, diffed on recompute so an unchanged set touches nothing. The
+tracker keeps each path's last-write stamp inside its watch entry and drops it on the raw change
+notification, before the debounce. The `ReferencesOnDisk` guard in `createProjectSnapshot` reads
+stamps through `IReferenceStamps`, so the comparison that runs for every new `Project` instance is
+a dictionary read per reference instead of a stat. A path nobody watches is stat'd directly. A
+mismatch against the snapshot's own stamps (FCS stats when it builds a snapshot) drops the
+project's stamps, so a missed notification costs one re-stat pass rather than a rebuild per
+`Project` instance. The reactor watch exists for these stamps, not to invalidate the FCS build —
+Roslyn already does that, as the previous section says.
+
+Still to come:
+
+1. Scripts: watch `#load` sources so an edit outside the editor drops the cached options for
+   that document.
+2. `FSharpProjectSnapshot.FromOptions` stats every `-r:` when a snapshot is built from scratch;
+   an overload taking host-supplied stamps lets it read the same cache.
+3. A reference-change notification for the incremental builder on the FCS side, the analogue of
+   `useChangeNotifications` for sources, so `IsReferencesInvalidated` stops stat'ing at all.

docs/release-notes/.FSharp.Compiler.Service/11.0.200.md

@@ -1,5 +1,6 @@
 ### Added
 
+* `Internal.Utilities.Library`: `String.StartsWithOrdinalIgnoreCase` extension, the `StartsWith` sibling of `EndsWithOrdinalIgnoreCase`. ([PR #20457](https://github.com/dotnet/fsharp/pull/20457))
 * F# Interactive gains a JSON-RPC server mode, `--fsi-server-jsonrpc:<pipe name>`, in which a host submits interactions over a named pipe and receives structured results — diagnostics with positions, escaping exceptions, the values each interaction bound, and the session's own process id — instead of recovering them by looking for a `SERVER-PROMPT>` marker in the output text. Program output continues to flow through the redirected console streams. The pipe admits only the user running the session; `--fsi-server-client-pid:<pid>` names the host process whose exit ends the session. `FsiEvaluationSession` exposes both options as `JsonRpcServerPipeName` and `JsonRpcClientProcessId`. The mode is part of the .NET fsi only. ([PR #20396](https://github.com/dotnet/fsharp/pull/20396))
 
 ### Fixed

docs/release-notes/.VisualStudio/18.vNext.md

@@ -4,6 +4,7 @@
 * Expand `<inheritdoc/>` in IDE tooltips, completion, and signature help, inheriting XML documentation from base classes, interfaces, overridden members, and constructors. ([Issue #19175](https://github.com/dotnet/fsharp/issues/19175), [PR #19188](https://github.com/dotnet/fsharp/pull/19188))
 
 ### Fixed
+* Transparent compiler snapshot reuse no longer stats every `-r:` reference on each project change; reference timestamps are cached and invalidated by `IVsAsyncFileChangeEx2` notifications. ([PR #20457](https://github.com/dotnet/fsharp/pull/20457))
 
 * Go To Definition no longer blocks the UI thread with a bare `Task.Wait`: the synchronous `IFSharpGoToDefinitionService` call now waits through the cancellable threaded-wait dialog, and the editor's `TaskCompletionSource` bridges run their continuations on the thread pool instead of inline on whichever thread finished the check, so repeated F12 on a large solution no longer starves semantic classification and other main-thread work. ([PR #20482](https://github.com/dotnet/fsharp/pull/20482))
 * Peek Definition on an F# symbol whose definition lives in metadata no longer deadlocks Visual Studio. Peek holds the main thread in `JoinableTaskFactory.Run` without pumping messages while it asks the language service for the definition, and generating the metadata document needs that same thread; Peek now stops at definitions that already have a document, and Go To Definition, which owns the wait it makes, still opens the generated one. ([PR #20503](https://github.com/dotnet/fsharp/pull/20503))
@@ -12,6 +13,7 @@
 * Fixed Find All References crash when F# project contains non-F# files like `.cshtml`. ([Issue #16394](https://github.com/dotnet/fsharp/issues/16394), [PR #19252](https://github.com/dotnet/fsharp/pull/19252))
 * Avoid using `cancellableTask` in `DocumentCache`; the editor cache now uses direct `CancellationToken`-aware `task` wrappers, avoiding the background `Task.Run` offload and a larger wrapper closure from the `cancellableTask` builder. ([Issue #20268](https://github.com/dotnet/fsharp/issues/20268))
 * Cache document diagnostics by version stamp, so an unchanged document is not reanalyzed on every crawler pass. ([Issue #20120](https://github.com/dotnet/fsharp/issues/20120), [PR #20121](https://github.com/dotnet/fsharp/pull/20121))
+* Watch on-disk `-r:` reference assemblies via `IVsAsyncFileChangeEx2`, so F# project options are invalidated when a referenced assembly is rebuilt instead of waiting for a timestamp poll. ([PR #20457](https://github.com/dotnet/fsharp/pull/20457))
 * Find All References for external DLL symbols now only searches projects that reference the specific assembly. ([Issue #10227](https://github.com/dotnet/fsharp/issues/10227), [PR #19252](https://github.com/dotnet/fsharp/pull/19252))
 * Improve static compilation of state machines. ([PR #19297](https://github.com/dotnet/fsharp/pull/19297))
 * Make Alt+F1 (momentary toggle) work for inlay hints. ([PR #19421](https://github.com/dotnet/fsharp/pull/19421))

src/Compiler/Utilities/illib.fs

@@ -97,6 +97,9 @@ module internal PervasiveAutoOpens =
         member inline x.StartsWithOrdinal(value: string) =
             x.StartsWith(value, StringComparison.Ordinal)
 
+        member inline x.StartsWithOrdinalIgnoreCase(value: string) =
+            x.StartsWith(value, StringComparison.OrdinalIgnoreCase)
+
         member inline x.EndsWithOrdinal(value: string) =
             x.EndsWith(value, StringComparison.Ordinal)
 

src/Compiler/Utilities/illib.fsi

@@ -58,6 +58,8 @@ module internal PervasiveAutoOpens =
 
         member inline StartsWithOrdinal: value: string -> bool
 
+        member inline StartsWithOrdinalIgnoreCase: value: string -> bool
+
         member inline EndsWithOrdinal: value: string -> bool
 
         member inline EndsWithOrdinalIgnoreCase: value: string -> bool

vsintegration/src/FSharp.Editor/FSharp.Editor.fsproj

@@ -55,6 +55,7 @@
     <Compile Include="LanguageService\IProjectSite.fs" />
     <Compile Include="LanguageService\ProvideFSharpVersionRegistrationAttribute.fs" />
     <Compile Include="LanguageService\MetadataAsSource.fs" />
+    <Compile Include="LanguageService\FileChangeWatcher.fs" />
     <Compile Include="LanguageService\FSharpProjectOptionsManager.fs" />
     <Compile Include="LanguageService\IFSharpWorkspaceService.fs" />
     <Compile Include="LanguageService\SingleFileWorkspaceMap.fs" />

vsintegration/src/FSharp.Editor/LanguageService/FSharpProjectOptionsManager.fs

@@ -8,20 +8,21 @@ open System.Collections.Concurrent
 open System.Collections.Immutable
 open System.IO
 open System.Linq
-open Microsoft.CodeAnalysis
+open System.Runtime.CompilerServices
+open System.Threading
+open System.Windows
 open FSharp.Compiler
 open FSharp.Compiler.CodeAnalysis
+open FSharp.Compiler.Text
+open Microsoft.CodeAnalysis
+open Microsoft.VisualStudio
 open Microsoft.VisualStudio.FSharp.Editor
-open System.Threading
 open Microsoft.VisualStudio.FSharp.Interactive.Session
-open System.Runtime.CompilerServices
-open CancellableTasks
 open Microsoft.VisualStudio.FSharp.Editor.Extensions
-open System.Windows
-open Microsoft.VisualStudio
-open FSharp.Compiler.Text
 open Microsoft.VisualStudio.TextManager.Interop
 
+open CancellableTasks
+
 #nowarn "57"
 
 [<AutoOpen>]
@@ -116,7 +117,7 @@ type private FSharpProjectOptionsMessage =
     | ClearSingleFileOptionsCache of DocumentId
 
 [<Sealed>]
-type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
+type private FSharpProjectOptionsReactor(checker: FSharpChecker, fileChangeWatcher: IFSharpFileChangeWatcher) =
     let cancellationTokenSource = new CancellationTokenSource()
 
     // Store command line options
@@ -137,6 +138,51 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
 
     let scriptUpdatedEvent = Event<FSharpProjectOptions>()
 
+    // The '-r:' set of each project is watched for the reference stamps the snapshot-reuse
+    // guard reads; invalidating the FCS build on a change is Roslyn's job (it swaps the
+    // MetadataReference and bumps Project.Version, which reaches tryComputeOptions).
+    let referenceChangeTracker =
+        new FSharpReferenceChangeTracker(fileChangeWatcher, ignore)
+
+    let referenceWatches = ConcurrentDictionary<ProjectId, HashSet<string>>()
+
+    let referencePaths (projectOptions: FSharpProjectOptions) =
+        let paths = HashSet<string>(StringComparer.OrdinalIgnoreCase)
+
+        for option in projectOptions.OtherOptions do
+            if option.StartsWith("-r:", StringComparison.Ordinal) then
+                paths.Add(option.Substring "-r:".Length) |> ignore
+
+        paths
+
+    let watchReferenceFiles (projectId: ProjectId) (projectOptions: FSharpProjectOptions) =
+        let paths = referencePaths projectOptions
+
+        match referenceWatches.TryGetValue projectId with
+        | true, previous ->
+            for path in previous do
+                if not (paths.Contains path) then
+                    referenceChangeTracker.StopWatchingReference path
+
+            for path in paths do
+                if not (previous.Contains path) then
+                    referenceChangeTracker.StartWatchingReference path
+        | _ ->
+            for path in paths do
+                referenceChangeTracker.StartWatchingReference path
+
+        if paths.Count > 0 then
+            referenceWatches[projectId] <- paths
+        else
+            referenceWatches.TryRemove projectId |> ignore
+
+    let clearReferenceWatches (projectId: ProjectId) =
+        match referenceWatches.TryRemove projectId with
+        | true, paths ->
+            for path in paths do
+                referenceChangeTracker.StopWatchingReference path
+        | _ -> ()
+
     let createPEReference (referencedProject: Project) (comp: Compilation) =
         let projectId = referencedProject.Id
 
@@ -410,7 +456,9 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
 
                             if not (Seq.isEmpty projectsToClearCache) then
                                 projectsToClearCache
-                                |> Seq.iter (fun pair -> cache.TryRemove pair.Key |> ignore)
+                                |> Seq.iter (fun pair ->
+                                    cache.TryRemove pair.Key |> ignore
+                                    clearReferenceWatches pair.Key)
 
                                 let options =
                                     projectsToClearCache
@@ -430,6 +478,7 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
                             let parsingOptions, _ = checker.GetParsingOptionsFromProjectOptions(projectOptions)
 
                             cache.[projectId] <- struct (project, parsingOptions, projectOptions)
+                            watchReferenceFiles projectId projectOptions
 
                             return ValueSome struct (parsingOptions, projectOptions)
 
@@ -516,6 +565,7 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
                     | _ -> ()
 
                     legacyProjectSites.TryRemove(projectId) |> ignore
+                    clearReferenceWatches projectId
                 | FSharpProjectOptionsMessage.ClearSingleFileOptionsCache(documentId) ->
                     match singleFileCache.TryRemove(documentId) with
                     | true, (_, _, _, projectOptions, subscription) ->
@@ -558,18 +608,24 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
         singleFileCache.Clear()
         lastSuccessfulCompilations.Clear()
 
+        for projectId in referenceWatches.Keys |> Array.ofSeq do
+            clearReferenceWatches projectId
+
     member _.ScriptUpdated = scriptUpdatedEvent.Publish
 
+    member _.ReferenceStamps = referenceChangeTracker :> IReferenceStamps
+
     interface IDisposable with
         member _.Dispose() =
+            referenceChangeTracker.Dispose()
             cancellationTokenSource.Cancel()
             cancellationTokenSource.Dispose()
-            (agent :> IDisposable).Dispose()
+            agent.Dispose()
 
 /// Manages mappings of Roslyn workspace Projects/Documents to FCS.
-type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Workspace) =
+type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Workspace, fileChangeWatcher: IFSharpFileChangeWatcher) =
 
-    let reactor = new FSharpProjectOptionsReactor(checker)
+    let reactor = new FSharpProjectOptionsReactor(checker, fileChangeWatcher)
 
     do
         // We need to listen to this event for lifecycle purposes.
@@ -638,4 +694,6 @@ type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Wor
 
     member _.ClearAllCaches() = reactor.ClearAllCaches()
 
+    member _.ReferenceStamps = reactor.ReferenceStamps
+
     member _.Checker = checker