feat/incremental-index

doc/decisions/0006-import-the-workspace-on-start-and-re-import-on-save.md

@@ -25,7 +25,9 @@ Chosen option: **full import on server start, then file-level re-import on `didS
 
 A `didChange` (keystroke-level) event does **not** trigger a re-import. Only saves do.
 
-This decision covers only the **OFT workspace index**: spec items and coverage links across the whole project. That is what `Oft.importItems()` rebuilds. It is genuinely expensive to redo on every keystroke. It does **not** mean the server ignores unsaved edits. The current text of every open document is still tracked in memory. It is updated on every `didChange` (`req~live-document-buffer~1`). So requests that only need the text of the file being edited see live content. That covers hover, definition, semantic tokens and especially completion, which almost always runs on text the user has not saved yet. Only cross-file concerns that depend on the *index* lag behind until the next save. An example is whether a given ID still exists workspace-wide.
+This decision covers only the **OFT workspace index**: spec items and coverage links across the whole project. That is what `Oft.importItems()` rebuilds. It is genuinely expensive to redo on every keystroke. It does **not** mean the server ignores unsaved edits. The current text of every open document is still tracked in memory. It is updated on every `didChange` (`req~live-document-buffer~2`). So requests that only need the text of the file being edited see live content. That covers hover, definition, semantic tokens and especially completion, which almost always runs on text the user has not saved yet. Only cross-file concerns that depend on the *index* lag behind until the next save. An example is whether a given ID still exists workspace-wide.
+
+The rebuild itself reads open documents from that buffer rather than from disk (`req~index-reads-open-documents~1`). This decision fixes **when** the index is rebuilt, not which version of a file it is built from. Reading disk for a file the editor holds open would produce a state that exists nowhere, mixing saved and unsaved files.
 
 **Excluding build output** Handing the workspace root straight to `Oft.importItems()` also indexed build output. The indexer now does its own file walk. It prunes hidden paths and the common build output names `target`, `build`, `out`, `dist` and `node_modules` at every depth, then hands the surviving files to OFT. Files without a matching importer are skipped by OFT itself, so passing single files is safe. A `.oftignore` file in the workspace root adds further glob patterns.
 
@@ -39,7 +41,7 @@ This decision covers only the **OFT workspace index**: spec items and coverage l
 * Good, because a large workspace no longer freezes the editor while it is indexed. The price is that requests before the first import finishes are answered from an empty index.
 * Good, because separating "live document text" from "workspace index" keeps the expensive part save-gated while the cheap part stays live. Completion would be unusable if it only saw saved content.
 * Neutral, because the *index* lags behind unsaved edits. A newly typed spec item ID does not resolve workspace-wide until saved. That matches most build-tool-backed language servers.
-* Bad, because a large workspace re-imports everything on every save, not just the changed file. Worth revisiting if profiling later shows this is too slow.
+* Neutral, because a rebuild covers the whole workspace. It no longer re-parses all of it: only changed files are imported, see [ADR 0014](0014-import-only-changed-files-when-rebuilding-the-index.md).
 * Bad, because the debounce adds up to 300 ms of deliberate delay before the index catches up.
 
 ### Confirmation
@@ -48,4 +50,4 @@ Integration tests verify that after a file save, the next definition request ret
 
 ## More Information
 
-A possible follow-up is file-scoped re-import: re-parse only the saved file and merge it into the existing index. That needs a better understanding of OFT's import pipeline than we have now.
+File-scoped re-import was the follow-up named here and is decided in [ADR 0014](0014-import-only-changed-files-when-rebuilding-the-index.md).

doc/decisions/0014-import-only-changed-files-when-rebuilding-the-index.md

@@ -0,0 +1,48 @@
+---
+status: accepted
+date: 2026-09-07
+decision-makers: Felix Gorke
+---
+
+# Import Only Changed Files When Rebuilding the Index
+`adr~import-only-changed-files-when-rebuilding-the-index~1`
+
+Needs: impl
+
+## Context and Problem Statement
+
+Every save and every file change outside the editor rebuilds the whole index ([ADR 0006](0006-import-the-workspace-on-start-and-re-import-on-save.md)), which re-parses every file in the workspace. Measuring shows:
+
+| workspace | walk | import | link | build |
+| --- | --- | --- | --- | --- |
+| this repository | 12 ms | 52 ms | 2 ms | 2 ms |
+| synthetic, 6000 files | 44 ms | 1170 ms | 4 ms | 6 ms |
+
+Parsing is the rebuild. Linking, which is the part that has to see the whole workspace, costs almost nothing.
+
+## Considered Options
+
+* Keep re-parsing everything
+* Cache the imported items per file and re-import only what changed
+* Update the linked graph incrementally as well
+
+## Decision Outcome
+
+Chosen option: **cache the imported items per file, re-import only changed files, link everything**. The expensive phase becomes proportional to what the user touched, while the phase that needs global knowledge stays a full pass and therefore stays correct by construction.
+
+A file counts as unchanged when its size and its modification time match the ones it had at the last import. Both come from the directory walk, which reads them anyway, so the check costs no extra file access. Hashing the content instead would be exact but was measured at 686 ms for those 6000 files, which is most of what the cache saves.
+
+A file the editor holds open is never cached and always imported again. Its text is the buffer, not the file, and the file says nothing about it (`req~index-reads-open-documents~1`).
+
+Updating the linked graph incrementally was rejected: a link status such as `ORPHANED` or `AMBIGUOUS` depends on items in other files, so a change would have to be propagated across the graph for 4 ms of savings.
+
+### Consequences
+
+* Good, because a rebuild after one edit no longer scales with the size of the workspace. Measured on 6000 files: 1253 ms for a full build, 73 ms for a rebuild.
+* Good, because changes made outside the editor are picked up like any other, by their timestamp. A branch switch invalidates exactly the files it rewrote.
+* Good, because the cache cannot drift into a different result: the items of an unchanged file are the ones it produced, and linking sees all of them.
+* Bad, because an edit that restores both size and modification time is invisible to the rebuild. That takes deliberate effort, or a filesystem whose timestamps are coarser than the interval between two writes of equal length.
+
+### Confirmation
+
+Integration tests for different cases.

doc/demo/example/src/CoffeeMaker.java

@@ -15,7 +15,8 @@ private void grindBeans() {
     private void heatWater() {
     }
 
-    // Intentional defect: this tag references revision 1, the design item is at revision 2.
+    // Intentional defect: this tag references revision 1, the design item is at
+    // revision 2.
     // [impl->dsn~heat-water~1]
     private void keepWarm() {
     }

doc/spec/requirements.md

@@ -96,12 +96,26 @@ Files changed outside the editor also trigger a reindex.
 Needs: impl, utest
 
 ## Live Document Buffer
-req~live-document-buffer~1
+req~live-document-buffer~2
 
-The server keeps the current in-memory text of every open document. It updates that text on `textDocument/didOpen` and `textDocument/didChange`. Requests that work on document text read from this buffer while the document is open. Those requests are hover, go to definition, find references, semantic tokens and completion. When the document is not open, they fall back to the file on disk. This buffer is separate from the OFT workspace index, which is only rebuilt on save (`req~index-refresh-on-file-change~1`).
+The server keeps the current in-memory text of every open document. It updates that text on `textDocument/didOpen` and `textDocument/didChange`. Requests that work on document text read from this buffer while the document is open. Those requests are hover, go to definition, find references, semantic tokens and completion. When the document is not open, they fall back to the file on disk.
 
 Needs: impl, utest, itest
 
+## Index Reads Open Documents
+req~index-reads-open-documents~1
+
+When the index is rebuilt, a file the editor holds open is read from the document buffer rather than from disk. Every other file is read from disk as before.
+
+Needs: impl, itest
+
+## Index Imports Only Changed Files
+req~index-imports-changed-files-only~1
+
+A rebuild imports only the files that changed since the last one and takes the items of every other file from what that run produced. A file counts as unchanged when its size and its modification time are the ones it had then. A file the editor holds open is always imported again, because its text is not the one on disk, and it is not cached either.
+
+Needs: impl, itest
+
 ## Highlight Specification Item Definitions
 req~highlight-specification-item~1
 

src/main/java/org/itsallcode/openfasttrace/lsp/OftLanguageServer.java

@@ -2,6 +2,7 @@
 
 import java.net.URI;
 import java.nio.file.Path;
+import java.util.Map;
 import java.util.List;
 import java.util.concurrent.CompletableFuture;
 
@@ -68,7 +69,6 @@ public CompletableFuture<InitializeResult> initialize(final InitializeParams par
 
         final var syncOptions = new TextDocumentSyncOptions();
         syncOptions.setOpenClose(true);
-        // [impl->req~live-document-buffer~1]
         syncOptions.setChange(TextDocumentSyncKind.Full);
         syncOptions.setSave(new SaveOptions(false));
 
@@ -162,11 +162,15 @@ private void buildInitialIndex() {
     }
 
     private void rebuildIndex() {
+        rebuildIndex(textDocumentService.openDocuments());
+    }
+
+    private void rebuildIndex(final Map<Path, String> openDocuments) {
         if (rootUri == null) {
             return;
         }
         final Path workspaceRoot = Path.of(URI.create(rootUri));
-        final OftWorkspaceIndex index = indexer.buildIndex(workspaceRoot);
+        final OftWorkspaceIndex index = indexer.buildIndex(workspaceRoot, openDocuments);
         textDocumentService.updateIndex(index);
         workspaceService.updateIndex(index);
         Logger.info("Workspace index ready: " + index.specItemCount() + " item(s)");

src/main/java/org/itsallcode/openfasttrace/lsp/OftTextDocumentService.java

@@ -122,6 +122,14 @@ private void publishWorkspaceDiagnostics() {
                 .forEach(workspaceDiagnosticUris::add);
     }
 
+    // [impl->req~index-reads-open-documents~1]
+    Map<Path, String> openDocuments() {
+        final Map<Path, String> documents = new LinkedHashMap<>();
+        openDocumentBuffers.forEach((uri, lines) -> LocationConverter.toPath(uri)
+                .ifPresent(file -> documents.put(file, String.join("\n", lines))));
+        return documents;
+    }
+
     private boolean isOpenFile(final String uri) {
         final String key = LocationConverter.toFileKey(uri);
         return openUris.stream().anyMatch(open -> LocationConverter.toFileKey(open).equals(key));
@@ -604,7 +612,7 @@ public void didOpen(final DidOpenTextDocumentParams params) {
         publishDiagnostics(uri);
     }
 
-    // [impl->req~live-document-buffer~1]
+    // [impl->req~live-document-buffer~2]
     @Override
     public void didChange(final DidChangeTextDocumentParams params) {
         final String uri = params.getTextDocument().getUri();