From 559aa3acfff0339733eb68eb2ff769282683fffa Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 26 Sep 2026 07:04:05 +0800 Subject: [PATCH 01/31] docs(agents): the issue #23 analysis, the fix plan for 0.0.5 and its implementation split; #24 is the register of upstream defects The analysis of issue #23 (Windows, MSVC, LTO: clangd's module scan fails on commands without -c) with the Windows CI measurements, the hello-project investigation (a half-typed import saved to disk spins clangd past WA-CLANGD-001), and the fix plan F1-F18 with its decisions D1-D5 and the split into work packages for one 0.0.5 pull request. The contributing skill names issue #24 as the single register of upstream defects. --- .agents/docs/2026-09-26-issue-23-fix-plan.md | 426 ++++++++++++++++++ .../2026-09-26-issue-23-lto-module-scan.md | 116 +++++ .agents/skills/mcppls-contributing/SKILL.md | 19 + 3 files changed, 561 insertions(+) create mode 100644 .agents/docs/2026-09-26-issue-23-fix-plan.md create mode 100644 .agents/docs/2026-09-26-issue-23-lto-module-scan.md diff --git a/.agents/docs/2026-09-26-issue-23-fix-plan.md b/.agents/docs/2026-09-26-issue-23-fix-plan.md new file mode 100644 index 0000000..347482c --- /dev/null +++ b/.agents/docs/2026-09-26-issue-23-fix-plan.md @@ -0,0 +1,426 @@ +# mcppls 修复与优化方案：issue #23（Windows/LTO）与 hello 项目实测（编辑中的卡死与重启） + +状态：方案第 3 版（D1–D5 已定；新增 F18 导出与脱敏；D4 的防频繁策略；自我 review）；§7 为 0.0.5 的实施计划（一个 PR 全部实现） · 2026-09-26 · 基于 `main` 04b186a（0.0.4） + +依据： + +- 分析报告 `.agents/docs/2026-09-26-issue-23-lto-module-scan.md`（含 GalTranslPP 的 Windows CI 实测）； +- hello 项目实测：用户 VS Code 会话日志 `~/.cache/mcppls/logs/server-20260925-215311.673-2d10.log`，以及在副本上逐字输入、 + 模拟自动保存、按线程采样 CPU 的复现； +- 上游问题统一登记在 issue #24（本文引用其中的 `UP-`）。**F 编号保持稳定**：#24 的评论引用了 F1、F3、F4、F5、F7、F11、 + F12、F16。 + +## 0. 摘要 + +### 0.1 两条问题链 + +**A. issue #23（Windows/MSVC + LTO）**：给 clangd 的命令没有 `-c` 却带 `-flto` → 模块扫描全部失败、不建 BMI（UP-11）→ Windows +clangd 在无 BMI 的模块单元上崩溃（UP-12）→ mcppls 隔离错了文件、临时模型阶段的崩溃先耗掉额度 → clangd 被判不可用。另有独立的 +上游崩溃（UP-13）和 BOM 识别缺陷（M2）。 + +**B. hello 项目（Linux，`files.autoSave: afterDelay`）**：自动保存把写了一半的 `import hello.` 写到磁盘 → WA-CLANGD-001 只改写 +LSP 文本、clangd 却从磁盘解析前置模块（UP-14）→ 该文件的 worker 进入 UP-01 死循环（实测单核满载，改回缓冲区也停不下）→ +该文件的全部 clangd 请求（含关键字补全、hover）排队超时；同时写到一半的模块名让计划来回变（stand-in 增删、数据库 13↔14 条）→ +每变一次重启一次（另一次来自切换工具链）→ 10 分钟 3 次后被上限锁死，之后空转的 clangd 无法再被恢复。另有诊断吵闹与位置偏移（UP-15）、未保存的新 import 不构建（UP-14）等体验问题。 + +### 0.2 条目总表（按主题） + +| 主题 | # | 修什么 | 来源 | 优先级 | 估算 | +|---|---|---|---|---|---| +| **正确性** | F1 | 给 clangd 的每条命令都带 `-c` | #23 根因（UP-11/12） | P0 | 1 天 | +| | F2 | 扫描器跳过 UTF-8 BOM | #23 附带（M2） | P0 | 0.5 天 | +| | F16 | WA-CLANGD-001 的磁盘漏洞：磁盘上的半截 import 让 clangd 死循环 | hello：提示消失、请求超时的根因（UP-01×UP-14） | P0 | 1–1.5 天 | +| | F5 | 兜底扫描排除包管理器目录 | #23 CI（M5） | P2 | 0.5 天 | +| **引擎恢复** | F3 | 按 clangd 崩溃上下文精确隔离，记录退出码 | #23（M3；UP-12/13 止损） | P1 | 1 天 | +| | F13 | 编辑中的计划变化防抖；不为仍在输入的 import 建 stand-in；重启合并 | hello（3 次重启） | P1 | 1.5–2 天 | +| | F14 | 重启分预算、达到上限后退避，不再锁死 10 分钟 | hello（被锁死） | P1 | 1 天 | +| | F4 | producer 回答前 clangd 等待；模型切换时清零崩溃记账 | #23（M4） | P1 | 2–3 天 | +| **编辑体验** | F11 | 未保存缓冲区里新加的 import：not found 改写为"保存后可用" | UP-14 | P1 | 1 天 | +| | F12 | 漏 `;` 的错误移回本行；可选的诊断防抖 | UP-15 | P2 | 0.5 天 | +| | F9 | `import ` 空格后自动弹出模块列表（仅 import 行） | hello | P2 | 0.5–1 天 | +| | F15 | 模块语法关键字补全由 mcppls 提供 | hello | P2 | 0.5–1 天 | +| | F8 | 声明前的 `export` 与 `export module` 高亮一致 | 用户提问 | P2 | 0.5 天 | +| **可观察性** | F17 | 常开日志环形缓冲、事故快照、计划差异、workaround 前提守卫、状态栏给原因 | 本次排查成本 | P1 | 2–3 天 | +| | F18 | 一键导出问题包（环境、日志、事故、崩溃、引擎数据库），统一脱敏（用户名、主目录、主机名、密钥），校验不泄漏才写出 | 用户要求 | P1 | 2.5–3 天 | +| | F6 | 扫描失败汇总成 issue；崩溃/扫描行不被限流 | #23 | P1 | 1–1.5 天 | +| **其他** | F7 | macOS 部署目标降到 11.0 | UP-M1（mcpp#685 已修） | P2 | 0.5 天 | +| | F10 | mcpp#699 落地后显示失败成员的诊断 | UP-M2/M3 | 后续 | 0.5 天 | +| **验证** | V | Windows（GalTranslPP CI）+ Linux（hello/自动保存夹具）验收 | 全部 | — | 0.5–1 天 | + +合计（不含 F10）约 **18–23 人日**。 + +### 0.3 版本划分建议 + +| 版本 | 目标 | 内容 | 估算 | +|---|---|---|---| +| **0.0.5 "稳"** | issue #23 关闭；hello 场景不再卡死、不再被锁死；出事能看清原因并一键导出 | F1、F2、F16、F3、F13、F14、F6、F7、F17 的 1–2 项（环形缓冲 + 事故快照）、F18 | 约 12–14 天（另加验证 V） | +| **0.0.6 "顺"** | 编辑体验 | F4、F11、F9 + F15、F12、F8、F5、F17 的 3–6 项 | 约 6–9 天 | +| 之后 | 跟随上游 | F10（mcpp#699）；UP-01 的上游处理与（必要时）自建 clangd（F16.3，D2：推迟） | — | + +### 0.4 依赖关系 + +- **F3 → F14 → F17 → F18**：F14 的分预算需要 F3 给出的崩溃归因；F17 的事故快照复用 F3 的崩溃上下文与 F14 的原因分类；F18 把 + F17 的事故目录、日志与报告打包并脱敏（没有 F17 时 F18 仍可导出日志、报告与环境，只是少了事故现场）。 +- **F13 ↔ F16**：两者都在"来自正在编辑的文档的变化"上做判断（F13 防抖计划变化，F16 按磁盘内容主动隔离），共用"文档最近编辑时间 + + 磁盘内容检查"的状态，一起实现。 +- **F4 ↔ F14**：都改崩溃/重启记账（F4 的"模型切换清零"与 F14 的"分预算"），一起设计。 +- **F9 ↔ F15**：都改补全路由（空格触发只交给 mcppls 引擎；关键字补全由 mcppls 提供并与 clangd 结果合并），一起实现。 +- **与 mcpp 无依赖**：F7 只需 mcpp ≥ 2026.9.24.1（已发布）；F10 等 mcpp#699。 + +## 1. 正确性 + +### F1. 给 clangd 的每条命令都带 `-c`（P0） + +**问题**：`src/normalize/gnu.cpp:20,49` 的 `ALWAYS_STRIP_EXACT` 删掉 `-c` 后无处补回；`translate_msvc`（`src/normalize/msvc.cpp`） +的输出同样没有；`semantic.cpp:86` 加的 `-c` 也会被 `translate_gnu` 删掉。最终阶段是 Link，windows-msvc + LTO 下驱动报错，clangd +的模块扫描失败（UP-11），不建 BMI，进而在 Windows 上触发 UP-12。 + +**改法**： + +1. `translate_gnu`、`translate_msvc` 仍删除构建命令自带的 `-c`、`--precompile`，但在输出末尾**补一个 `-c`**（已有则不重复）； + `kit_arguments` 同样补。 +2. 派生条目自动继承：项目单元（`plan.cpp:473`）、游离文件（`:296`）、prime（`:499-501`）、std（`:523-547`）、stand-in + （`:577-581`）。 +3. **不删 `-flto` 等链接参数**：`-c` 一次关掉所有只在链接阶段做的驱动检查，不需维护链接参数清单，也不改变项目的 LTO（#23 报告者 + 明确要求）。 + +**为什么安全**：clangd 建 AST/BMI 时 `createInvocation` 插入 `-fsyntax-only`，与 `-c` 并存时取更早的阶段，行为不变；CMake、mcpp +自己写的构建数据库本来就带 `-c`。 + +**测试**：`tests/test_normalize.cpp:70,114` 改为"恰好一个 `-c`"；新增 windows-msvc + `-flto` 的 GNU、clang-cl 与 kit 三种输入； +plan 级测试覆盖各派生条目；conformance：Windows CI 的 `mcpp-llvm-msvc-lto`，以及 Linux 可跑的 compdb 夹具（本机已证明该驱动检查 +在 Linux 上同样触发）。 + +### F2. 扫描器跳过 UTF-8 BOM（P0） + +**问题**：`src/project/scan.cpp` 从偏移 0 开始词法分析，BOM + `export module X;` 开头的文件认不出模块声明。所有文本扫描都走 +`project::scan_source`（计划 `plan.cpp:288`、自带模块引擎 `engine/native/index.cpp:71`、AI review、查询视图）， +`engine/native/exports.cpp` 另有一份镜像的声明解析。本机复现：可信模型下 mcppls 引擎误报 `module 'm' not found`；兜底模型下还会 +漏掉 `-x c++-module`。 + +**改法**：`Lexer` 构造时若以 `EF BB BF` 开头则从偏移 3 开始，偏移仍按原文计算（语义 token、诊断位置不变）；`exports.cpp` 同样处理。 + +**测试**：BOM + `export module`/`module;`/`module m;`/`import` 四种；语法 token 列号与无 BOM 时一致；conformance `mcpp-bom`。 + +### F16. WA-CLANGD-001 的磁盘漏洞（P0） + +**实测（hello 副本，0.0.4，`Worker:main.cpp` 在 20 秒内的 CPU tick）**：`import hello.` 只在缓冲区 → 2；自动保存写盘后 → 约 2000； +缓冲区改回 `import hello.greet;` 而磁盘仍是 `import hello.` → 约 2000。WA-CLANGD-001 只改写 LSP 文本，而 clangd 从磁盘解析前置模块 +（UP-14），于是进入 UP-01 的死循环，只有重启能解开。这是 hello 场景"只有 `import hello.` 有提示""停久了请求超时"的根因； +会话里计入的 3 次重启另有来源（两次计划抖动见 F13，一次切换工具链），达到上限后，因空转而请求的重启又被拦下（22:02:25）， +clangd 从此停在空转里（F14）。 + +**改法**： + +1. **磁盘侧检测 + 主动隔离**：`didSave` 与文件监视时，对数据库内的文件用 WA-CLANGD-001 的同一判定检查磁盘内容；命中时在 clangd + 重建该文件的 preamble 之前把它交给 mcppls 自己的引擎（隔离），磁盘内容变回合法后交还。不等空转、不走重启、不耗额度。 +2. **兜底**：若已在空转（`Worker:` 持续满载且磁盘内容命中），立即重启 clangd 并隔离该文件到磁盘恢复为止，原因写明，不计入 + 计划重启额度。 +3. **根治（D2 已定：推迟）**：本期只做 1、2 的规避；上游处理（bisect `6dcfc17b1b`、提交、申请 23.1.x backport）推迟，已在 #24 + 的 UP-01 标记为 deferred；若迟迟没有 backport，再自建 23.1.x + cherry-pick 作为后备。UP-14 若在上游修复（扫描改用含缓冲区 + 的 `TFS`），磁盘路径也随之消失。 + +**测试**：conformance——逐字输入、在 `import hello.` 处写盘 + `didSave`；断言无满载 worker、无重启、其他文件照常应答，改成完整名字 +并保存后该文件回到 clangd。 + +### F5. 兜底扫描排除包管理器目录（P2） + +`src/project/infer.cpp:21` 的 `SKIPPED_DIRECTORIES` 加入 `vcpkg_installed`、`vcpkg`、`.conan`/`.conan2`、`.xmake`、`.cache`、 +`.git`，并跳过含 `vcpkg.json` 的子项目根。#23 的 CI 中 producer 失败时 349 条里有 166 条来自 `vcpkg_installed/`，产生伪 +`ambiguous-module`。测试：`test_project.cpp` 增加对应用例。 + +## 2. 引擎恢复 + +### F3. 按 clangd 的崩溃上下文精确隔离，记录退出码（P1） + +**问题**：`handle_closed_`（`src/engine/clangd.cpp:1528-1575`）以"未答请求 + 退出前 10 秒碰过的文件"为嫌疑；Windows 实测中 clangd +报 `NormalJsonTranslator.Core.cpp`，mcppls 却隔离了 `GPPDefines.ixx`，前两次崩溃无嫌疑。 + +**改法**：在 stderr 转发处（`clangd.cpp:1066-1100`，与 `parse_module_failure` 并列）解析 +`Signalled during AST worker action: <动作>` / `Filename:` / `Exception Code:`（或 POSIX 信号）；本代进程有崩溃上下文时只隔离 +那个文件，没有时才回退启发式；`engine-exit` 事件与报告增加 `exitCode`（`Connection::exit_code()` 已有）、`crashFile`、 +`crashAction`；issue 文案写明"clangd crashed while building `<文件>`"。 + +**测试**：解析器单测（Windows/POSIX、多行、夹杂日志）；conformance 用替身 clangd 打印崩溃上下文后退出。 + +### F13. 编辑中的计划变化不让 clangd 抖（P1） + +**实测（用户会话日志）**：21:54 输入 `import hello…`，自动保存把 `import h` 写盘，clangd 报 `Failed to build module h`，`h` 是合法名 +字，生成 stand-in，4 秒后撤掉 → 重启 1；22:01 新建 `src/test/test.cppm`，先与 `src/test.cppm` 重复声明 `hello.test`、后为 +`hello.test.`，mcpp emit 两次失败，数据库 13↔14 条 → 重启 3。 + +**改法**： + +1. 来自正在编辑的文档的 import/模块声明引起的计划变化（stand-in 增删、补位条目、provider 变化），在该文档停止修改约 2 秒后才应用； + 自动保存不算"稳定"。 +2. 不为最近编辑过的文档里的 import 生成 stand-in（H3 从"非法名字"扩大到"仍在输入"）。 +3. 计划驱动的重启合并为最多每 N 秒一次、总用最新计划；clangd 自己重读数据库（约 5 秒）能解决的不重启。 +4. producer 在编辑中途失败时保持上一份计划，打开文件的补位条目也不来回增删。 + +**测试**：conformance——逐字输入 `import hello.test.a;`，中途模拟自动保存；断言无 stand-in、无 `engine-restart`。 + +### F14. 重启上限不再把 clangd 锁死（P1） + +**实测**：22:01:24 起 `not restarting clangd again: already 3 restarts in the last 10 minutes`，22:02–22:04 clangd 对全部请求 +不应答约 90 次，`main.cpp` 被隔离。 + +**改法**：重启按原因分预算（计划驱动 / 崩溃与卡住）；模型的工具链或 profile 变化（如用户切换工具链）引起的重启不计入任何预算；达到上限后按 1、2、4、8 分钟退避，到点即可恢复卡住的 +clangd；受限期间状态栏给出原因与"Restart clangd"动作。 + +**测试**：预算与退避单测；conformance：连续触发计划重启后 clangd 仍能在退避后恢复。 + +### F4. producer 回答前 clangd 等待；模型切换时清零崩溃记账（P1） + +**问题**：`start_inferred_load()`（`workspace.cpp` ~770）在 producer 回答前用语义套件的临时模型（Windows 上是 `x86_64-w64-mingw32`， +无项目 include 目录）喂给 clangd。#23 CI：E1 的 5 次退出中 3 次发生在前 18 秒；E4（关 LTO）可信阶段本身正常，仍因这一阶段的崩溃 +`engine-restart-capped`。 + +**改法**：识别出构建系统（level 3）且 producer 正在运行时，clangd 等待（打开的文件进"等待数据库"队列，由 mcppls 引擎回答），上限为 +producer 超时（默认 60 秒）；超时或失败后才用临时/兜底计划。模型来源变化（`inferred` → `producer`）时清零崩溃计数、释放因临时模型进入的 +隔离、重置重启门。没有构建系统的项目保持现状。 + +**取舍**：等待期间只有模块级功能（GalTranslPP 温启动 producer 约 24 秒）；换来可信模型到达时 clangd 不带"前科"。 + +**测试**：mockmcpp 延迟回答（沿用 `mcpp-emit-hang`），断言回答前无 `engine-start`；"模型切换清零"单测。 + +## 3. 编辑体验 + +### F11. 未保存缓冲区里新加的 import（UP-14，P1） + +**实测**：未保存时补全出 `import a.b;`，`Module 'a.b' not found` 45 秒以上不消失；保存后 1–8 秒消失（hello 与 cmp 两个项目）。 + +**改法**：clangd 对打开且有未保存修改的文档报 `Module 'X' not found`，而 X 在计划里有提供者、这行 import 只在缓冲区不在磁盘时，替换为 +信息级诊断"module 'X' is in the project; clangd loads it once the file is saved"；X 不在计划里时保留原错误。其间 X 相关的模块级请求 +由 mcppls 引擎回答。上游修复建议已写入 UP-14。 + +**测试**：增量输入不保存，断言无该错误、出现信息；保存后信息消失。 + +### F12. 输入 import 时的诊断位置与闪烁（UP-15，P2） + +**实测**：输入 `import hello` 时 `Import directive must end with a ';'` 标在两行之后的 `auto main()` 上；逐字输入时每约 150 ms 一条 +（`Unknown type name 'i'`、`Module 'h' not found`……）。 + +**改法**：该诊断落在第 N 行、且其上方最近的非空行是未以 `;` 结束的 `import`/`export import`/`module` 指令时，把范围移到那一行末尾 +（登记为新的 `WA-CLANGD-`）；可选：对正在编辑的文档把引擎诊断推迟到最后一次修改后约 400 ms，保存或停顿后照常发布。 + +### F9 + F15. 补全：空格触发与模块语法关键字（P2） + +**实测（LSP 层）**：模块名补全（`import hello.`、`import hello.test.`、`import a`、`import :`）瞬时可用；关键字补全来自 clangd +（`i` 起即含 `import name;`，`e` 起含 `export`，`isIncomplete` 正确透传）；clangd 该文件卡住时关键字补全随之消失；`triggerCharacters` +是 clangd 的 `. < > : " / *`，没有空格；clangd 不提供 `export module`、`export import` 组合。 + +**改法**： + +- F9：`triggerCharacters` 加空格；空格触发的补全只在行前缀为 `[export] import ` 时交给 mcppls 引擎，其余立即回空、**不转发 clangd**。 + **避免频繁处理（D4）**，由外到内四层： + 1. **编辑器侧拦截（VS Code）**：插件的 language client middleware 在发请求前检查 `triggerCharacter === ' '` 与当前行文本 + （`document.lineAt`），不匹配就直接返回空，**普通代码里的空格不产生任何 IPC**。 + 2. **服务端门槛**：路由收到空格触发的请求时，只取该文档当前行到光标的文本（文档已在内存）做一次前缀判断 + （`^\s*(export\s+)?import\s$`：关键字后恰好一个空格、之后没有别的字符），不匹配立即回 `{isIncomplete:false, items:[]}`； + 不转发 clangd、不查索引、不写 info 日志（只计数）。 + 3. **结果缓存**：模块名候选按计划代次缓存，匹配时直接复制，不随每次按键重建。 + 4. **按客户端启用**：只对已知能优雅处理空结果的客户端（`clientInfo.name` 为 VS Code / Cursor）声明空格触发；Neovim、Zed、 + CLion 默认不声明，可用初始化选项 `mcppls.completion.triggerOnSpace` 打开。VS Code 另有 F15 的补充：接受 `import` 关键字后 + 随即弹出模块列表（补全项带 `editor.action.triggerSuggest`），不依赖空格触发。 + 报告（F17）记录空格触发的请求数、通过门槛的次数与耗时，用来确认开销可以忽略。 +- F15：mcppls 引擎在行首上下文提供 `import`、`export import`、`module;`（仅文件开头）、`export module`、`module`（实现单元）、 + `module :private;`（仅接口单元），与 clangd 结果合并去重；接受 import 类关键字后接着弹出模块名列表。 + +**测试**：`completion-contains` 覆盖各前缀、空格触发的两种行、clangd 不可用时仍有关键字。 + +### F8. 声明前的 `export` 与 `export module` 高亮一致（P2） + +**实测**：`module;`、`export module`、`export import`、`import`、`module :private;` 的关键字都有 `keyword` 语义 token；声明前的 +`export`（`export namespace`、`export {`、`export int f()`）服务端不发，VS Code 内置 C++ 语法也不给作用域，显示为普通文本。 + +**改法**：`scan_syntax_tokens` 对模块单元里声明开头的 `export` 发 `keyword`；注入语法增加对应规则。 + +## 4. 可观察性 + +### F17. 一次复现就能看清原因（P1；0.0.5 先做 1–2） + +这次排查的代价：用户的 info 级日志（487 行）只有"重启了""没应答"，没有哪个文件、哪个线程、clangd 看到的是缓冲区还是磁盘、计划为什么 +变；每个关键事实都要另起环境复现才得到。 + +1. **常开的 clangd 日志环形缓冲**：clangd 以 `--log=info` 运行（D3 已定），最近 N 行（例如 4000 行 / 4 MB）只留在内存；出事时整段 + 落盘并记入报告；需要更细时由事故触发临时切到 verbose 再采一段。 +2. **事故快照**（崩溃/卡住/隔离/重启/达到上限各一份，`/incidents/<时间>-<类型>/`）：原因链；涉及文件的缓冲区版本与磁盘内容差异 + （只记涉及的行）；引擎数据库中的条目；`fileStatus` 时间线；按线程 CPU（Linux `/proc//task`，Windows `GetThreadTimes`）； + 退出码与崩溃上下文（F3）；最近的计划差异。 +3. **计划差异日志**：列出增删改的条目及原因；`units are compiled with other arguments` 写出哪个单元、哪个参数。 +4. **workaround 前提守卫**：每个 `WA-CLANGD-` 声明前提（WA-001：clangd 只看到 LSP 文本），运行时检测到前提被破坏即记事故（本次即 + "磁盘内容命中判定"）。 +5. **状态栏写原因、给动作**："clangd 卡在 `main.cpp`：磁盘上的 `import hello.` 未写完（UP-01）"，附"查看事故 / 导出问题包（F18）/ 重启 clangd"。 +6. **conformance 覆盖自动保存路径**（本次漏洞就是这样漏过的）。 +7. **事故目录保留**：最近 20 次或 7 天，超出即删，避免无限增长。 + +### F18. 一键导出问题包，统一脱敏（P1） + +**为什么**：现在出了问题，用户能给的只有 `Collect Report`（一份 JSON，最近 300 行日志，提示"含本机路径，请自行编辑"）。本次排查 +真正用到的信息——多次会话的日志、clangd 自己的输出、线程 CPU、缓冲区与磁盘差异、工具链与编辑器环境、崩溃退出码——散在各处， +要靠开发者另起环境复现。用户也不该手工删除用户名和路径。 + +**入口**（同一个实现，服务端 C++，所有编辑器共用）： + +- VS Code：命令 **"C++ Modules: Export Diagnostic Bundle"**；F17 的事故通知、状态栏受限提示、`Collect Report` 的结果页都提供它。 +- 其他编辑器：`workspace/executeCommand` `mcppls.exportBundle`。 +- 命令行：`mcppls report --bundle [--root ] [--settle N] [--hide-project-paths] [--no-source-excerpts] + [--include-dumps] [--no-redact]`；CI（如 GalTranslPP 的验证 workflow）直接把问题包作为 artifact 上传。 + +**内容**（zip，默认写到 `/bundles/mcppls-bundle-<时间>.zip`，完成后给出"在文件夹中显示 / 复制路径"；**从不自动上传**）： + +| 文件 | 内容 | +|---|---| +| `manifest.json` | 格式版本、生成时间、包含了什么、应用了哪些脱敏规则及各自命中次数（**不含原值**）、各文件大小与哈希 | +| `report.json` | 当前的 `cxxModules/report`（引擎状态、计划、事件、工具运行、workaround） | +| `environment.json` | 操作系统/内核/架构、CPU 与内存、区域与代码页（Windows ACP）、编辑器名与版本、插件版本、其他 C++ 插件的安装与启用状态、mcppls 设置、payload 版本（`payload.json`）、clangd / mcpp / xlings 版本、工具链探测结果、环境变量**白名单**（`MCPP_*`、`XLINGS_*`、`LANG`/`LC_*`、`PATH`） | +| `logs/` | 最近 3 次会话或 24 小时内的服务端日志（超出按"头 + 尾"截断）；插件自己的输出通道日志 | +| `incidents/` | F17 的事故快照：clangd 日志片段、线程 CPU、fileStatus 时间线、缓冲区/磁盘差异、崩溃上下文与退出码 | +| `engine/` | 引擎数据库（`compile_commands.json`）、计划摘要、模块图 | +| `dumps/` | 仅 `--include-dumps` 或界面勾选时：Windows minidump（体积大，且含内存内容） | + +**不包含**：源代码全文。事故里只带与问题直接相关的行（例如 `import hello.`），可用 `--no-source-excerpts` 关掉；项目路径默认按 +主目录规则替换为 `~/…`，`--hide-project-paths` 再替换为 `/…`。 + +**脱敏**（对包内每个文本文件执行，JSON 与纯文本都适用）： + +1. **主目录 → `~`**：覆盖 POSIX 路径、Windows 路径（`\` 与 `/`、JSON 转义的 `\\`、盘符大小写）、8.3 短名（`C:\Users\RUNNER~1`）、 + `file://` URI 及其百分号编码（`%3A`）、WSL 的 `/mnt/c/Users/<名>`。 +2. **用户名 → ``、主机名 → ``**：只在路径与已知字段（环境变量值、URI 的 authority）中替换，避免把普通单词误改。 +3. **密钥 → ``**：环境变量只收白名单；键名像 `api_key`/`token`/`secret`/`password`/`authorization` 的值；已知前缀 + （`ghp_`、`github_pat_`、`sk-`、`xoxb-`）；邮箱；编译参数里名字像密钥的 `-D=`；AI 网关相关设置的值。 +4. **一致映射**：同一原值总是换成同一占位符，路径仍可比较；映射表本身不进包。 +5. **写出前自检，失败即不写**：打包完成后在所有文件里搜索原始的主目录、用户名、主机名（上述各种写法）；只要还有残留，就不生成 + 问题包，并报告是哪个文件、哪条规则漏了。 +6. `--no-redact` 只给本地排查用，界面上不提供。 + +**同步改动**：`Collect Report` 默认也走同一套脱敏（它本来就是为贴到 issue 准备的），界面文案从"请自行编辑"改为"已替换用户名与路径， +可在导出前预览"。 + +**测试**：脱敏单测覆盖上面每种写法（Linux 与 Windows、JSON 转义、URI 编码、8.3 短名、WSL）；自检"残留即失败"；conformance： +在一个用户名较长且出现在各类路径里的临时 HOME 下导出，断言包里任何文件都不含原用户名与主目录、manifest 与实际内容一致、 +大小在上限内（例如 25 MB）。 + +### F6. 扫描失败自解释（P1） + +解析 `Scanning modules dependencies for <文件> failed: <原因>` 汇总为 issue（次数、前几个文件、第一条原因；驱动错误时措辞为 +"clangd rejected the compile command for module scanning: `<原因>`"——#23 的报告者因此能直接看到 `LTO requires -fuse-ld=lld`）； +崩溃上下文、扫描失败、`Failed to build module` 行不受 `LineLimiter` 限流；报告增加 `scanFailures`。 + +## 5. 其他 + +- **F7（UP-M1）**：`.github/versions.env` 的 `MCPP_VERSION` 升到 ≥ 2026.9.24.1，`mcpp.toml` 的 `[build]` 加 + `macos_deployment_target = "11.0"`，用 `llvm-objdump --macho --private-headers` 验证 `minos 11.0`。 +- **F10（mcpp#699 落地后）**：`parse_database_envelope`（`src/spec/discovery.cpp:62`）只要有 `data` 就接受、不看退出码，部分数据库 + 会被直接使用（前提：mcpp 保持 `kindVersion = 1`）；届时只需把失败成员的诊断显示为 issue。 +- **V（验证）**： + - Windows：Sunrisepeak/GalTranslPP PR #1 的 workflow 加 `workflow_dispatch` 输入，改用 mcppls 某次 CI 的 win32-x64 payload；验收： + E1 可信阶段 0 次 LTO 扫描失败、hover 有内容（含跨模块）、隔离的都是 clangd 报出的文件、不因临时阶段被判不可用、BOM 文件无误报。 + - Linux：把本次的逐字输入脚本（`import hello.` 写盘 A/B、`import hello.test.a;` 未保存、线程 CPU 采样）固化为 conformance 夹具； + 验收：无满载 worker、无计划驱动的重启风暴、关键字补全在任何时候都可用。 + - 问题包（F18）：两个验证环境都用 `mcppls report --bundle` 产出问题包并作为 artifact 上传；验收：包含事故快照与多次会话日志，且任何 + 文件都不含 runner 的用户名与主目录（Windows `C:\Users\runneradmin`、8.3 短名 `RUNNER~1`，Linux 的 `$HOME`）。 + - 验证结束后关闭 GalTranslPP PR #1。 + +## 6. 决策与自我 review + +### 6.1 已定的决策（2026-09-26） + +| # | 决策 | 结论 | +|---|---|---| +| D1 | F4：producer 回答前 clangd 等待（最长 60 秒，只剩模块级功能） | 做，只对识别出构建系统的项目 | +| D2 | F16.3：UP-01 的根治是否捆绑自建 clangd | 本期只做 F16.1/F16.2 的规避；上游处理推迟，已在 #24 的 UP-01 标记（待账号权限恢复后写入，见 §6.4）；自建作为后备 | +| D3 | F17.1：常开的 clangd 日志级别 | `--log=info` 进环形缓冲，出事后临时提到 verbose 再采一段 | +| D4 | F9：空格触发如何避免频繁处理 | 四层：编辑器侧拦截（零 IPC）→ 服务端前缀门槛（不转发 clangd）→ 结果按计划代次缓存 → 只对已知客户端声明；并计数验证开销 | +| D5 | F15：`export module ` 后是否建议名字 | 不建议名字，只补关键字 | +| — | 新增 F18：问题包导出与脱敏 | 做，P1，进 0.0.5 | + +### 6.2 这一版做了什么 + +- 写入 D1–D5；F9 增加 D4 的四层策略；F16.3、F17.1 按决策改写。 +- 新增 F18，并把 F17 的事故通知、状态栏提示、`Collect Report` 都接到它；事故目录加保留上限（F17.7）。 +- 重算：合计 18–23 人日；0.0.5 约 12–14 天（新增 F18）；依赖链改为 F3 → F14 → F17 → F18。 +- 更正：F16 是"只有 `import hello.` 有提示、停久了超时"的根因，但不是"重启 3 次"的根因（那是 F13 的计划抖动加一次切换工具链）； + §0.1、F16、§6.3 已按日志改写。F14 的"用户操作不计"改为以模型的工具链/profile 变化为判据。F18 的失败处理统一为"残留即失败"。 + +### 6.3 自我 review：检查过的点 + +- **编号与引用**：#24 引用的 F1、F3、F4、F5、F7、F11、F12、F16 都在，含义未变；新增的 F17、F18 未被外部引用。 +- **优先级一致**：P0 只有"会让功能完全失效"的三项（F1、F2、F16）；P1 是"会反复失效或无法定位"的（引擎恢复、F11、F17、F18、F6）； + 体验项为 P2。 +- **版本边界**：0.0.5 覆盖 issue #23 的关闭条件（F1、F2、F3、F6）与 hello 场景的根因与锁死（F16、F13、F14），并能导出现场（F17 部分、 + F18）；F4 放 0.0.6：hello 场景的 3 次重启里两次是计划抖动（F13 消除）、一次是切换工具链（F14 不再计入）；F4 针对的是 Windows 冷启动 + 临时阶段的崩溃，F14 的分预算已能防止它把 clangd 锁死，F3 能正确归因，因此可以晚一个版本。 +- **互相冲突的地方**：F13 的 2 秒防抖与 F11 的信息级诊断配合，不会出现"合法 import 被报错"；F9 的服务端门槛与 F15 的关键字补全 + 走同一条补全路由，一起实现避免两次改动；F18 的脱敏同时用于 `Collect Report`，避免两套规则。 +- **隐私**：问题包默认不含源代码全文、不含 minidump、不含非白名单环境变量；"残留即失败"的自检保证不会带着用户名写出。 + +### 6.4 剩余风险与待办 + +1. **F16 只覆盖了实测到的读盘路径**：后台索引同样读磁盘，可能在其他线程上遇到 UP-01。实现前先验证（写盘 `import hello.` 后观察 + `ground-worker-*` 线程）；若也会卡住，F16.1 需把该文件暂时移出后台索引，或在命中时推迟转发文件变化。 +2. **F1 改变所有命令**：clangd 模块缓存按命令哈希，升级后首次打开会重建全部 BMI（一次性），更新说明写明。 +3. **F13 的 2 秒防抖**让合法的新 import 晚约 2 秒生效；由 F11 的信息级诊断兜住体验。 +4. **F14 的退避可能掩盖崩溃循环**：用 F3 的归因区分，崩溃循环仍走现有上限。 +5. **Windows 的 UP-13** 在 0.0.5 只能止损，根因等取到调用栈（#24 待办）。 +6. **F18 的脱敏漏网**：用户名很短或是常见词时，只在路径与已知字段替换，其他位置可能残留；由"残留即失败"兜底：界面上只提供 + "隐藏项目路径后重试"，不提供"仍然导出"；确需原样内容时只能在命令行用 `--no-redact`，且只用于本地排查。 +7. **时序敏感的测试**（F13、F14、F16）：夹具用事件驱动而非固定 sleep，避免 CI 不稳定。 +8. **issue #24 的待写入更新**：当前 gh 账号 `speak-agent` 对仓库只有 pull 权限，UP-01 的"D2：上游处理推迟"说明与索引中 UP-01、UP-14 + 两行的更新已在本地准备好，需要切回有权限的账号（或给该账号授权）后写入。 + +## 7. 实施计划：0.0.5 一个 PR 做完（2026-09-26 定） + +范围调整：§0.3 的两个版本合为 **0.0.5 一次发布、一个 PR**（与 0.0.4 的 #22 同样的形态）；F1–F9、F11–F18 全做，F10 等 +mcpp#699（仍 open），F16.3 按 D2 推迟。 + +### 7.1 多角度的约束（每个工作包都按这 8 条自查） + +| 角度 | 约束 | +|---|---| +| 架构 | 新能力放进独立模块，引擎只通过 `Host` 与工作区交互：崩溃/扫描行解析在 `engine.clangd.process`；重启预算在 `engine.clangd.guard`（`RestartGate` 改为按原因分预算 + 退避）；日志环形缓冲与事故记录是引擎无关的新模块（`Host::record_incident`，默认空实现，与 `record_event` 同形）；问题包与脱敏是新目录 `src/bundle/`，只读缓存目录与报告，不反向依赖引擎 | +| 稳定性 | 任何新逻辑都不阻塞事件循环（问题包在后台线程生成、以事件回到循环）；预算/退避/防抖都有上限；脱敏失败即不写出；事故目录有保留上限；时序测试用事件驱动 | +| 优雅简洁 | F1 一条规则（补 `-c`）替代链接参数清单；F18 一套脱敏同时服务问题包与 Collect Report；F9/F15 走同一条补全路由；每个 workaround 仍在注册表里（F12 是 `WA-CLANGD-006`） | +| 用户体验 | 不卡死、不锁死；状态栏说出原因并给出动作（导出问题包 / 重启 clangd）；import 行空格即出模块列表、关键字补全任何时候可用；`export` 高亮一致；未保存的新 import 是信息而非错误 | +| 兼容性 | 非 VS Code 客户端：空格触发默认不声明、可用初始化选项打开；新命令都走标准 `workspace/executeCommand`；报告与状态只增字段（S3 附加）；clangd 日志 info 行仍只进 debug 级日志，不增加默认日志量 | +| 跨平台 | 崩溃上下文同时认 Windows（`Exception Code`）与 POSIX（信号）；线程 CPU 在 Linux 读 `/proc`，其他平台退化为进程 CPU；脱敏覆盖 Windows 路径、8.3 短名、`file://` 编码、WSL；zip 写出器纯 C++、无平台依赖；F7 macOS 11 | +| 一致性 | 事件名 `engine-*`、报告字段 camelCase、文档中英同步、S3 规范 + traceability + 夹具一起改、#24 与 workaround 注册表对应 | +| 无感升级 | 设置默认值不变；新设置可选；缓存目录只新增 `incidents/`、`bundles/`；F1 让 clangd 命令哈希变化 → 首次打开重建 BMI 一次（CHANGELOG 写明）；不需要迁移 | + +### 7.2 工作包与依赖 + +| 包 | 内容 | 主要文件 | 依赖 | 执行 | +|---|---|---|---|---| +| **A** | F1、F2、F5、F8 | `normalize/{gnu,msvc,semantic}.cpp`、`project/{scan,infer}.cpp`、`engine/native/exports.cpp`、VS Code 语法 | 无 | 并行（独立 worktree） | +| **C** | F9、F15 | `orchestrator/{routing,workspace(路由入口)}.cpp`、`engine/native{,/index}.cpp`、`server/session.cpp`（初始化选项）、VS Code middleware | 无 | 并行 | +| **D** | F18 + Collect Report 脱敏 + VS Code 两个命令（导出问题包、重启 clangd） | 新 `src/bundle/`、`cli/{options,commands}.cpp`、`server/session.cpp`（`mcppls.exportBundle`）、VS Code | 与 B 的契约（7.3） | 并行 | +| **B** | F3、F6、F14、F4、F13、F16、F11、F12、F17（1–7） | `engine/clangd{,/process,/guard,/workarounds}.cpp`、`orchestrator/workspace.cpp`、新事故模块 | 内部顺序：F3/F6 → F17.1–2 → F14 → F4 → F13+F16 → F11/F12 → F17.3–7 | 主线 | +| **E** | F7 | `.github/versions.env`、`mcpp.toml` | 无 | 主线 | +| **F** | 版本 0.0.5、CHANGELOG、docs（中英）、S3 规范/schema/traceability、design.md、本文实施记录 | — | A–E 合入后 | 主线 | +| **V** | Linux conformance 全量 + 新夹具；Windows：GalTranslPP 探针用本 PR 的 payload；问题包验收 | — | F 之后 | 主线 | + +合入顺序：A、C、D 各自在 worktree 分支上完成并通过单测，按 A → C → D 合进 `release/0.0.5`（冲突只可能在 +`session.cpp`、`workspace.cpp` 路由入口、`conformance.cpp`、`traceability.json`、VS Code `package.json`，均为追加），然后 B、E、F、V。 + +### 7.3 B ↔ D 的契约（问题包读什么） + +- 事故目录：`<工作区缓存>/incidents/-/`，内含 `incident.json`（kind、时间、根目录、原因链、 + 涉及文件与其缓冲区/磁盘差异的相关行、fileStatus 时间线、线程 CPU、退出码与崩溃上下文、最近计划差异、重启历史）与 + `clangd.log`（环形缓冲落盘）。保留最近 20 个或 7 天。 +- 服务端日志：`<缓存>/logs/server-*.log`（`open_log_file`）；工作区缓存目录在报告的 `roots[].cacheDirectory`。 +- 服务端命令：`mcppls.exportBundle`（D）、`mcppls.restartClangd`（B：绕过预算重启一次，记为用户操作、不计入预算）；VS Code 命令 + `mcppls.exportDiagnosticBundle`、`mcppls.restartClangd`（D）。状态 issue 的 `command` 可指向这两个（B）。 + +### 7.4 完成标准 + +单测（dev + release profile）、`devtools check all`、`validate.py`、全部 conformance（含新夹具）本地通过；PR 的 CI 三平台全绿； +自我 review（含一次独立 review agent）后无遗留问题；squash 合入；Release 工作流发布 0.0.5；下载发布产物、校验哈希、本地装 VSIX 与 +payload 验证；最后报告产物目录。 + +## 不做的事 + +- 不删除 `-flto` 或其他链接参数（F1）。 +- 不为 UP-13 做规避性改写：只止损，取栈与上游报告在 issue #24 跟踪。 +- 不改写用户的文件（F16 只检测磁盘内容，不修改）。 +- 问题包不自动上传，不含源代码全文与非白名单环境变量（F18）。 +- mcpp 侧的 UP-M2/UP-M3 已合并提交为 mcpp-community/mcpp#699，在 issue #24 跟踪，本方案只做 F10 的适配。 diff --git a/.agents/docs/2026-09-26-issue-23-lto-module-scan.md b/.agents/docs/2026-09-26-issue-23-lto-module-scan.md new file mode 100644 index 0000000..ee7c1bd --- /dev/null +++ b/.agents/docs/2026-09-26-issue-23-lto-module-scan.md @@ -0,0 +1,116 @@ +# Issue #23 分析：Windows/MSVC + LTO 下 clangd 模块扫描失败、clangd 反复退出 + +状态：分析完成（未改代码）· 2026-09-26 · `main` 04b186a（0.0.4）· 捆绑 clangd 23.1.0（clangd/clangd 官方 Windows +发布版）· issue： + +证据来源： + +- 本机（Linux x64）：Clang 22 驱动、`clang-scan-deps`、捆绑 clangd 23.1.0，对照 llvm-project 23.1.0 源码。 +- **Windows 实测**：Sunrisepeak/GalTranslPP 临时 PR #1 的 workflow（windows-2025，VS 18 / MSVC 14.51，mcpp 2026.9.25.1， + LLVM 22.1.8，Qt 6.11.1，vcpkg 22 个 port），跑 mcppls 0.0.4 的 win32-x64 payload。第 4 轮（run 36178100611）复现了 + 报告者的环境：mcpp producer 成功，引擎数据库 **227 条、227 条带 `-flto`、0 条带 `-c`**（报告者也是 227 条）。 + 第 3 轮（run 36164741493）因 CI 的 Qt 不全（`lupdate.exe` 依赖 `Qt6Qml.dll`）producer 失败，意外覆盖了兜底模型路径。 + +## 结论 + +| # | 问题 | 归属 | 结论 | 证据 | +|---|---|---|---|---| +| **M1** | clangd 数据库去掉 `-c`、保留 `-flto`：windows-msvc 目标下驱动报 `LTO requires -fuse-ld=lld`，clangd 的 P1689 扫描失败，**不建任何 BMI** | mcppls | **根因，已确认** | 本机复现 + Windows 实测：可信模型阶段扫描失败 12 次**全部**是这条；210 次 hover 全空 | +| **U1** | clangd 23.1.0（Windows）在"没有 BMI、import 解析不了"的模块单元上 Build AST 时崩溃（`0x80000003`） | clangd 上游，**由 M1 触发** | 已确认；修 M1 即消除 | `GPPDefines.ixx`：原样命令 `--check` 崩溃（E3a），加 `-c` 后 exit 0（E3b）；编辑器会话 20 秒内崩在它上面（E3c） | +| **U2** | clangd 23.1.0（Windows）在 `NormalJsonTranslator.Core.cpp` 等文件上 Build AST 崩溃，**与 `-c`、LTO 无关** | clangd 上游 | 已确认，根因未定位 | 加 `-c`、BMI 正常、跨模块 hover 正常时仍崩（E3d）；Linux 上同形状不崩 | +| **M2** | 扫描器不认 UTF-8 BOM：文件以 BOM + `export module X;` 开头时认不出模块接口 | mcppls | 已确认，**新发现** | 本机：可信模型下 mcppls 模块引擎误报 `module 'm' not found`，去掉 BOM 即消失；兜底模型下还漏掉 `-x c++-module`（CI：3 个 `.ixx`）。GalTranslPP 中 `GPPDefines`、`ProgressBar`、`TerminalController` 受影响 | +| **M3** | 崩溃嫌疑判断不准：按"退出前 10 秒碰过的文件"判定，而 clangd 已打印崩溃文件 | mcppls | 已确认 | E1：clangd 报 `NormalJsonTranslator.Core.cpp`，mcppls 隔离 `GPPDefines.ixx`；前两次崩溃嫌疑为空。**报告者"`GPPDefines.ixx` 被隔离"很可能同属误判** | +| **M4** | producer 尚未回答时的临时模型（`firstOrigin: inferred`）用机器上找到的 MinGW（`x86_64-w64-mingw32`）、不带项目 include 目录喂给 clangd；这一阶段的崩溃耗掉重启额度 | mcppls | 已确认 | E1：5 次退出中 3 次发生在前 18 秒的临时阶段；E4（关 LTO）可信阶段本身正常，仍因临时阶段的崩溃出现 `engine-restart-capped` | +| **M5** | producer 失败时的兜底扫描把 `vcpkg_installed/` 当项目源码（349 条中 166 条），产生伪 `ambiguous-module`（`proxy.v4`） | mcppls | 已确认，影响小 | 第 3 轮 | + +一句话：**报告者看到的"模块全部不可用 + clangd 反复退出 + `GPPDefines.ixx` 被隔离"，是 M1（没有 BMI）→ U1（Windows clangd 在 +无 BMI 的模块单元上崩溃）→ M3（隔离错文件）叠加 M4（临时阶段的崩溃先耗掉额度）的结果**；U2 是独立的上游崩溃；M2 是同一项目 +暴露的独立 bug。 + +## 1. M1：命令是怎么变成这样的、为什么只坏在扫描上 + +- `src/normalize/plan.cpp:220-236`：单元参数来自 `options_arguments`（`semantic.cpp:86` 会加 `-c`）或构建命令，再经 + `translate_gnu`/`translate_msvc`。 +- `src/normalize/gnu.cpp:20,49`：`ALWAYS_STRIP_EXACT` 删掉 `-c`，此后无处补回；`-flto`、`-O3`、`-g` 原样保留。 + `src/normalize/msvc.cpp:33-40` 放行所有 `-f*`，输出同样无 `-c`。所有派生条目（单元 `:473`、游离文件 `:296`、prime + `:499-501`、std `:523-547`、stand-in `:577-581`）都继承。`tests/test_normalize.cpp:70,114` 断言输出无 `-c`——首个版本 + 起就是这样，0.0.3/0.0.4 都受影响，不是回归。kit 路径（`semantic_subset`）不受影响。 +- 驱动检查（`clang/lib/Driver/Driver.cpp:4348-4364`）只在 `FinalPhase == Link`、windows-msvc、开 LTO、`-fuse-ld` 非 lld + 时报 `err_drv_lto_without_lld`。 +- clangd 建 AST/BMI 走 `createInvocation`（`clang/lib/Driver/CreateInvocationFromArgs.cpp:51`），会插入 `-fsyntax-only`， + 不受影响（Windows 日志里 cc1 为 `-O3 -fsyntax-only -flto=full`）；而模块扫描 + （`clangd/ProjectModules.cpp:213-240` → `DependencyScanningTool.cpp:121` 的 `containsError()`）用原命令，直接失败。 + `CompoundProjectModules::getRequiredModules` 直接用扫描结果 → 空 → 不建 BMI；`MODULE_HINTS` 也要扫描确认，同样失败。 + +本机复现（`-###` 不执行任何步骤）： + +| 命令形状（`clang++ -x c++-module a.ixx -###`） | 退出码 | +|---|---| +| `--target=x86_64-pc-windows-msvc -flto` / `-flto=thin` | 1，`LTO requires -fuse-ld=lld` | +| 同上加 `-c`，或加 `-fuse-ld=lld` | 0 | +| `--target=x86_64-linux-gnu -flto` | 0 | + +## 2. Windows 实测（第 4 轮，报告者环境） + +| 实验 | 设置 | 结果 | +|---|---|---| +| E0 | `mcppls report`，不开文件 | 在引擎开始干活前结束，无异常（快照意义有限） | +| E1 | `mcppls serve` 编辑器会话 15 分钟（GPPDefines.ixx、ApiTool.ixx、GPPDefines.cpp、NormalJsonTranslator.Core.cpp、GPPCLI.cpp） | 可信阶段扫描失败 12 次全为 LTO；clangd 90 秒内退出 5 次后被判不可用（`state: unavailable`）；**210 次 hover 全空** | +| E2 | `mcppls check GPPDefines.ixx` | 扫描 LTO 失败 1 次 | +| E3a | clangd `--check GPPDefines.ixx`，mcppls 的数据库原样 | **崩溃 `0x80000003`** | +| E3b | 同上，数据库每条加 `-c` | **exit 0，无扫描失败，无崩溃** | +| E3c | clangd 单独会话，原样数据库 | 第一轮请求前即崩在 `GPPDefines.ixx` | +| E3d | clangd 单独会话，加 `-c` | 无扫描失败；跨模块 hover 生效（`string` → `provided by `）；约 100 秒后崩在 `NormalJsonTranslator.Core.cpp`（U2） | +| E4 | E1 但默认 profile 改为 `fast-release`（无 LTO） | 可信阶段无扫描失败、无崩溃；48 次 hover 有内容（6 次跨模块）；但临时阶段崩溃 2 次、随后 `engine-restart-capped`、文件被隔离与"隔离后仍在处理"的重启（M3/M4） | + +E1 的时间线（mcppls 事件 × clangd 崩溃上下文）： + +| 时间 | 模型阶段 | clangd 报的崩溃文件 | mcppls 的嫌疑 | +|---|---|---|---| +| 19:15:23 | 临时（MinGW，无 include） | `GPPCLI.cpp` | 无 | +| 19:15:27 | 临时 | `GPPCLI.cpp` | 无 | +| 19:15:38 | 临时 | `NormalJsonTranslator.Core.cpp` | **`GPPDefines.ixx`**（错） | +| 19:15:55 | 可信（MSVC + `-flto`） | `ApiTool.ixx` | `ApiTool.ixx` | +| 19:16:45 | 可信 | `NormalJsonTranslator.Core.cpp` | `NormalJsonTranslator.Core.cpp` | + +没有拿到崩溃栈：clangd 只打印了 `Signalled during AST worker action: Build AST` 与 `Exception Code: 0x80000003`，未写 +minidump，Application 事件日志为空。`0x80000003` 在 LLVM 的 Windows 构建中通常意味着内部 `abort()`（LLVM 把 SIGABRT 转成 +陷阱以进入崩溃处理），日志中没有断言或 `LLVM ERROR` 文本。 + +## 3. M2：BOM + +`src/project/scan.cpp` 没有跳过 UTF-8 BOM。第一行是 `module;` 的文件不受影响（后面的 `export module` 仍被识别），第一行就是 +`export module X;` 的文件会被判为非模块。可信模型下 clangd 的数据库用 mcpp 给的 `ide.role`/`provides`（mcpp 构建数据库自带 +这两个字段），所以不影响给 clangd 的命令；但 mcppls 自己的模块引擎按文本扫描，会误报。兜底模型下还会漏掉 `-x c++-module`， +clang 把 `.ixx` 当成链接输入（`'linker' input unused`）。 + +本机复现：一个 mcpp 项目，`src/m.cppm` 写成 BOM + `export module m;`，`mcppls check src/main.cpp` 输出 +`module main.cpp:1: module 'm' not found`；去掉 BOM 后无此行。 + +## 4. 修复建议（未实施） + +| 优先级 | 项 | 做法 | 测试 | +|---|---|---|---| +| P0 | **M1** | 保留删除构建自带的 `-c`/`--precompile`，但在 `translate_gnu`、`translate_msvc`（与 `kit_arguments`）输出末尾补一个 `-c`；所有派生条目随之修正。不删 `-flto`：`-c` 一次关掉所有链接阶段检查，且不动项目的 LTO | `test_normalize.cpp:70,114` 改为断言恰好一个 `-c`；新增 windows-msvc + `-flto` 用例；一个 Linux 可跑的 clangd 级 fixture（本机已证明 windows-msvc 目标在 Linux 上也复现） | +| P0 | **M2** | 扫描器（及模块引擎的文本扫描）跳过开头的 UTF-8 BOM | 单测：BOM + `export module`、BOM + `module;` | +| P1 | **M3** | 解析 clangd 的崩溃上下文（`Signalled during AST worker action …` 后的 `Filename:`），以它为唯一嫌疑；没有上下文时才回退到现有启发式 | 用 mock clangd 输出崩溃上下文的 conformance fixture | +| P1 | **M4** | producer 尚未回答的 mcpp 项目：临时模型不把 MinGW 等"猜来的"工具链喂给 clangd（或至少读取 `mcpp.toml` 的 `[toolchain]` 与 `include_dirs`）；模型从临时切到可信时清零崩溃计数与隔离 | fixture：producer 延迟回答 + 临时阶段崩溃不导致 `engine-restart-capped` | +| P2 | **M5** | 兜底扫描排除包管理器目录（`vcpkg_installed`、`.conan*`、`node_modules`、`target` 等） | 单测 | +| P2 | 诊断 | `engine-exit` 事件与报告记录 clangd 退出码（`Connection::exit_code()` 已有）与崩溃文件；把 `Scanning modules dependencies for X failed: ` 汇总成 issue；崩溃上下文行不受 `LineLimiter` 限流 | — | + +预期效果（按 Windows 实测推断）：M1 修复后可信阶段扫描恢复、BMI 可建、U1 不再触发（E3b/E3d 已证明）；剩下 U2 这类上游崩溃， +由 M3 精确隔离到单个文件、M4 不让临时阶段耗掉额度，clangd 对其余文件保持可用。 + +## 5. 上游（clangd） + +- U1 与 U2 都是 clangd 23.1.0 Windows 官方发布版在 Build AST 时的崩溃，应向 llvm-project 报告，前提是一个最小复现和调用栈。 +- 下一步：在 GalTranslPP 的 CI 上用 Windows SDK 的 `cdb.exe` 挂 clangd 取栈；并用 LLVM 22.1.8 自带的 clangd 对照，判断是否为 + 23.x 回归。U1 可从 `GPPDefines.ixx`（只有 import、无 include）开始缩小。 + +## 6. 给报告者的回复要点 + +- 根因已在 Windows 上复现并定位（M1），下一个版本修复；**无需**关闭 LTO 或加 `-fuse-ld=lld`。 +- `GPPDefines.ixx` 被隔离很可能是误判（M3），与 clangd 在无 BMI 时的崩溃（U1）有关；修复 M1 后这类崩溃在实测中消失。 +- BOM 开头的模块接口（`GPPDefines.ixx` 等）目前会被 mcppls 自身误判（M2），同版本修复。 +- 可选：在 VS Code 设置 `"mcppls.trace.server": "verbose"` 后复现一次并附上日志，便于确认 `NormalJsonTranslator.Core.cpp` + 一类的独立崩溃（U2）在其机器上是否同样出现。 diff --git a/.agents/skills/mcppls-contributing/SKILL.md b/.agents/skills/mcppls-contributing/SKILL.md index 992852b..a6a47ac 100644 --- a/.agents/skills/mcppls-contributing/SKILL.md +++ b/.agents/skills/mcppls-contributing/SKILL.md @@ -55,6 +55,25 @@ Conformance needs a payload; `CONTRIBUTING.md` has the exact command and how to - **The version lives in `mcpp.toml`** and is written everywhere else by `mcpp run -p devtools -- version --set`, never by hand. +## Upstream defects (clangd, mcpp, …) + +**Issue #24 is the single register of upstream defects** (pinned, English). Its body is only an +index; **each comment is one problem**, `UP-` (clangd/LLVM) or `UP-M` (mcpp): symptom, +affected versions, upstream status, what mcppls does, when that can go, evidence, TODO. Read it +before calling something "a clangd bug", and before working around one. + +- **Found a new one?** Post one comment in that shape (`unfiled` is a status) and add its row to + the index — then decide what mcppls does. +- **Compensating in code?** It is a registered workaround: an entry in + `src/engine/clangd/workarounds.cpp` (`WA-CLANGD-`: upstream, evidence, `removeWhen`, a canary + where one can exist), and its #24 comment names the ID. A limit mcppls deliberately does not work around + goes in `.agents/docs/design.md` §7 and in #24. +- **Filed or fixed upstream?** Put the link in that comment and the index. Bumping the bundled clangd + (`packaging/payload.lock.json`) means walking #24: run the canaries, remove what they say is gone, + update the comments. +- **Not upstream:** a defect in how mcppls drives an upstream tool (for example the arguments it + generates) is an mcppls bug, fixed here; #24 records only the upstream side of it. + ## Commit messages Lowercase `type(scope): a sentence that says what is now true`, not an imperative. The body is From aec618aa7412328acf0be2217ab5f8c014c6040d Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 26 Sep 2026 07:06:39 +0800 Subject: [PATCH 02/31] build(macos): the server runs on macOS 11 again, built with mcpp 2026.9.26.1 The macOS server said minos 14.0 although it imports only three libSystem symbols from macOS 10.12, because mcpp ignored macos_deployment_target when cross-compiling from Linux (mcpp#685, UP-M1 in #24). mcpp 2026.9.24.1 honours it; CI now builds with 2026.9.26.1 and the target is 11.0, the same as the bundled clangd's (F7). --- .github/versions.env | 2 +- mcpp.toml | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/versions.env b/.github/versions.env index 072a8f9..8885b51 100644 --- a/.github/versions.env +++ b/.github/versions.env @@ -4,6 +4,6 @@ # # These are build inputs, not the product's version: that one lives in mcpp.toml and is checked # everywhere else by `mcppls-devtools version --check`. -MCPP_VERSION=2026.9.21.3 +MCPP_VERSION=2026.9.26.1 LLVM_VERSION=22.1.8 XLINGS_VERSION=v2026.8.17.2 diff --git a/mcpp.toml b/mcpp.toml index 2dd7198..661827d 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -42,6 +42,9 @@ repo = "https://github.com/Sunrisepeak/mcpp-language-server" [build] sources = ["src/**/*.cppm", "src/**/*.cpp"] +# The server imports three libSystem symbols, all of them macOS 10.12+; clangd in the payload is +# built for 11.0. Honoured when cross-compiling from Linux since mcpp 2026.9.24.1 (mcpp#685). +macos_deployment_target = "11.0" # Every executable shares every module under src/. [targets.mcppls] From 4ef4d3c35f3aa226bbb9986e1c8414ff600cb279 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 26 Sep 2026 07:19:07 +0800 Subject: [PATCH 03/31] fix(normalize): every command clangd is given stops before linking, so a windows-msvc build with -flto has its modules built again Issue #23: the engine database dropped the build's own -c with the other build-only arguments and never put one back, while it kept link arguments such as -flto. Without -c the driver plans a link, and for the windows-msvc target a check made only for the link step, "LTO requires -fuse-ld=lld", made clangd's P1689 module scan fail for every unit: no module was built, imports reported as not found, and hovers across modules answered nothing, while each file's own AST looked fine. translate_gnu, translate_msvc and kit_arguments now end with exactly one -c (COMPILE_ONLY), before -x c++-module, so every entry derived from them carries it too: project units, files opened outside the model, prime units, the standard library's units and stand-ins. clangd's own -fsyntax-only still decides what it builds, and -flto stays: the project's LTO is not the engine's business. The fixture compdb-lto-msvc writes the compile_commands.json of such a project (clang++ --target=x86_64-pc-windows-msvc -flto, with --no-default-config so no configuration file picks lld) and runs on Linux, where the driver raises the same error. With the previous server its hover and diagnostics checks fail and clangd logs the scan failure six times; with this one they pass and nothing fails to scan. --- .github/workflows/ci.yml | 2 +- conformance/README.md | 1 + .../fixtures/compdb-lto-msvc/scenario.json | 40 +++++++++ .../fixtures/compdb-lto-msvc/src/answer.cppm | 3 + .../fixtures/compdb-lto-msvc/src/main.cpp | 5 ++ src/bin/conformance.cpp | 31 ++++++- src/normalize/gnu.cpp | 2 + src/normalize/gnu.cppm | 14 +++- src/normalize/msvc.cpp | 2 + tests/test_normalize.cpp | 82 ++++++++++++++++++- 10 files changed, 175 insertions(+), 7 deletions(-) create mode 100644 conformance/fixtures/compdb-lto-msvc/scenario.json create mode 100644 conformance/fixtures/compdb-lto-msvc/src/answer.cppm create mode 100644 conformance/fixtures/compdb-lto-msvc/src/main.cpp diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7c47220..badc87e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -277,7 +277,7 @@ jobs: - platform: linux-x64 part: 1 of 2 os: ubuntu-24.04 - fixtures: mcpp-split mcpp-all-cppm verify-changes mingw mcpp-split-gcc mcpp-watch multi-root mcpp-llvm mcpp-watch@polling mcpp-emit s1-two-sets failure-at-base@vscode generated-module-negotiated typing-import typing-import-spin workaround-canaries + fixtures: mcpp-split mcpp-all-cppm verify-changes mingw mcpp-split-gcc mcpp-watch multi-root mcpp-llvm mcpp-watch@polling mcpp-emit s1-two-sets compdb-lto-msvc failure-at-base@vscode generated-module-negotiated typing-import typing-import-spin workaround-canaries - platform: linux-x64 part: 2 of 2 os: ubuntu-24.04 diff --git a/conformance/README.md b/conformance/README.md index 5651073..ff79464 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -46,6 +46,7 @@ checks fail at once with that reason instead of each waiting out its timeout. | `cmake-clang-cl` | CMake 4.4 modules built by clang-cl (P6), the first CMake that scans clang-cl module sources | | `compdb-clangxx-msvc-std` | clang++ for the MSVC ABI with `import std`, built by the fixture's own script (P5) | | `compdb-clang-cl-std` | clang-cl with `import std`, built by the fixture's own script (P6) | +| `compdb-lto-msvc` | Issue #23: a `compile_commands.json` of clang++ for the MSVC ABI with `-flto` (the runner's `prepare compdb-lto-msvc`, no configuration file choosing lld). Every command the server gives clangd carries `-c`, so clangd's module scan does not stop at the driver's `LTO requires -fuse-ld=lld`: the module is built, the importer's import resolves and a hover across it answers. The driver raises that for the windows-msvc target on any host, so this runs on Linux | | `mcpp-msvc` | mcpp with `msvc@system` and `import std`, described by mcpp's own `emit build-database`: level 3, a test across sets, the workspace unchanged | | `mcpp-llvm-msvc` | mcpp's default Windows toolchain, LLVM for `x86_64-windows-msvc`, with the MSVC STL (P5), described the same way | | `inferred-msvc` | Loose module sources on a machine with Visual Studio: MSVC STL semantics without a build system (design 9.3, D27) | diff --git a/conformance/fixtures/compdb-lto-msvc/scenario.json b/conformance/fixtures/compdb-lto-msvc/scenario.json new file mode 100644 index 0000000..cbe70e6 --- /dev/null +++ b/conformance/fixtures/compdb-lto-msvc/scenario.json @@ -0,0 +1,40 @@ +{ + "name": "compdb-lto-msvc", + "description": "Issue #23, fix plan F1: a compile_commands.json of a project built with LTO for the MSVC ABI (clang++ --target=x86_64-pc-windows-msvc -flto -c, written by the runner's `prepare compdb-lto-msvc`). Every command the server gives clangd must carry -c: without it clangd's module dependency scan fails on the driver's `LTO requires -fuse-ld=lld`, no module is built, and the importer reports its import as not found and hovers across it answer nothing. The driver raises that for the windows-msvc target on any host, so this runs on Linux.", + "prepare": [ + [ + "{conformance}", + "prepare", + "compdb-lto-msvc", + "{env:CONFORMANCE_CLANGXX|clang++}" + ] + ], + "checks": [ + { + "id": "S1", + "kind": "status", + "source": "compile-commands", + "profile-kind": "build-toolchain" + }, + { + "id": "L1-the-module-is-built", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 3, + 12 + ], + "expect": "answer_of_everything", + "timeout": 60 + }, + { + "id": "L2-the-import-resolves", + "kind": "diagnostics-empty", + "file": "src/main.cpp" + }, + { + "id": "W1", + "kind": "workspace-unchanged" + } + ] +} diff --git a/conformance/fixtures/compdb-lto-msvc/src/answer.cppm b/conformance/fixtures/compdb-lto-msvc/src/answer.cppm new file mode 100644 index 0000000..2a393fb --- /dev/null +++ b/conformance/fixtures/compdb-lto-msvc/src/answer.cppm @@ -0,0 +1,3 @@ +export module answer; + +export int answer_of_everything() { return 42; } diff --git a/conformance/fixtures/compdb-lto-msvc/src/main.cpp b/conformance/fixtures/compdb-lto-msvc/src/main.cpp new file mode 100644 index 0000000..7eb63a7 --- /dev/null +++ b/conformance/fixtures/compdb-lto-msvc/src/main.cpp @@ -0,0 +1,5 @@ +import answer; + +int main() { + return answer_of_everything() == 42 ? 0 : 1; +} diff --git a/src/bin/conformance.cpp b/src/bin/conformance.cpp index f628e1b..61c0cdc 100644 --- a/src/bin/conformance.cpp +++ b/src/bin/conformance.cpp @@ -2220,6 +2220,34 @@ int prepare_generated_module_compdb(const std::string& compiler) { return 0; } +// compdb-lto-msvc (issue #23, fix plan F1): the compile_commands.json of a project built with LTO for the +// MSVC ABI, as CMake writes it for `-flto` with clang++ on Windows, with `compiler` (or clang++ on PATH) +// as the driver. Nothing is compiled: the fixture is about the commands the server gives clangd, whose +// module scan failed on `LTO requires -fuse-ld=lld` when they carried `-flto` and no `-c`. The driver +// raises that for the windows-msvc target on any host, so the fixture runs on Linux. `--no-default-config` +// makes the driver the one of the LLVM Windows installer, with no configuration file: an LLVM that +// carries one choosing lld (as mcpp's does) would not plan the link that fails. +int prepare_compdb_lto_msvc(const std::string& compiler) { + const std::string root { fs::current_directory() }; + auto clangxx = on_path(compiler.empty() ? std::string { "clang++" } : compiler); + if (!clangxx) { + say("compdb-lto-msvc: {} is not on PATH", compiler); + return 1; + } + Json database = Json::array(); + for (const std::string_view relative : { "src/answer.cppm", "src/main.cpp" }) { + const std::string source { native(base::join_path(root, relative)) }; + database.push_back(Json { { "directory", native(root) }, { "file", source }, + { "arguments", Json::array({ *clangxx, "--no-default-config", "--target=x86_64-pc-windows-msvc", "-std=c++23", + "-flto", "-O2", "-c", source, "-o", source + ".obj" }) } }); + } + if (auto written = fs::write_file(base::join_path(root, "compile_commands.json"), database.dump(2)); !written) { + say("compdb-lto-msvc: {}", written.error().message); + return 1; + } + return 0; +} + // real-project plan RP2.1: a second, newer mock mcpp under a fixture's isolated HOME // (`"isolate-home": true`), at the path producer negotiation searches // (`mcppls::project::other_mcpp_executables`, `xim-x-mcpp//bin/mcpp`), so a fixture whose @@ -2404,7 +2432,8 @@ int prepare(const std::string& kind, const std::string& argument) { if (kind == "compdb-clangxx-msvc-std") return prepare_compdb_msvc_std(false); if (kind == "generated-module-old-mcpp") return prepare_generated_module_compdb(argument); if (kind == "clangd-cannot-load") return prepare_clangd_cannot_load(); - say("prepare: unknown fixture kind {} (s1-two-sets, payload-corrupt, producer-candidate, failure-at-base, compdb-clang-cl-std, compdb-clangxx-msvc-std, generated-module-old-mcpp, clangd-cannot-load)", kind); + if (kind == "compdb-lto-msvc") return prepare_compdb_lto_msvc(argument); + say("prepare: unknown fixture kind {} (s1-two-sets, payload-corrupt, producer-candidate, failure-at-base, compdb-clang-cl-std, compdb-clangxx-msvc-std, generated-module-old-mcpp, clangd-cannot-load, compdb-lto-msvc)", kind); return 2; } diff --git a/src/normalize/gnu.cpp b/src/normalize/gnu.cpp index 66f4efe..f44e4d2 100644 --- a/src/normalize/gnu.cpp +++ b/src/normalize/gnu.cpp @@ -69,6 +69,7 @@ std::vector translate_gnu(const GnuInput& input) { if (input.facts != nullptr && input.facts->toolchain.target.find("windows-msvc") != std::string::npos) { for (auto& argument : windows_msvc_arguments(*input.facts, out, input.noAlignedAllocationWithMsvcStl)) out.push_back(std::move(argument)); } + out.emplace_back(COMPILE_ONLY); if (input.importable) { out.emplace_back("-x"); out.emplace_back("c++-module"); @@ -157,6 +158,7 @@ std::vector kit_arguments(const spec::Kit& kit, std::string_view la out.emplace_back("-isysroot"); out.emplace_back(macosSdk); } + out.emplace_back(COMPILE_ONLY); return out; } diff --git a/src/normalize/gnu.cppm b/src/normalize/gnu.cppm index bceb78e..8446a2d 100644 --- a/src/normalize/gnu.cppm +++ b/src/normalize/gnu.cppm @@ -16,6 +16,14 @@ inline constexpr std::string_view ENGINE_CLANG_DRIVER { "clang++" }; // the translated arguments carry no -x. inline constexpr std::string_view ENGINE_CLANG_C_DRIVER { "clang" }; +// Every engine command stops before linking (issue #23, fix plan F1). The build's own `-c` goes with +// the rest of what only the build needs, and this one is added once: without it the driver plans a +// link, and a check made only for the link step -- `LTO requires -fuse-ld=lld` for windows-msvc with +// `-flto` -- makes clangd's module dependency scan fail, so no module is built. clangd's own +// `-fsyntax-only` takes precedence over it, so what clangd reads is unchanged; link arguments such as +// `-flto` stay, since they mean nothing once no link is planned. +inline constexpr std::string_view COMPILE_ONLY { "-c" }; + struct GnuInput { std::span arguments; // full compile command, argv[0] first, response files expanded std::string source; // absolute path of the unit's source @@ -26,7 +34,8 @@ struct GnuInput { }; // Arguments without argv[0], output, dependency files, BMI and scanning flags, -// the source file or -x; for GCC with target, standard library and installation made explicit. +// the source file or -x, and with one COMPILE_ONLY; for GCC with target, standard library and +// installation made explicit. std::vector translate_gnu(const GnuInput& input); // For a toolchain that targets the MSVC ABI: target, emulated cl version, the Visual // Studio toolset and Windows SDK, and, when `noAlignedAllocation`, aligned allocation off with the @@ -36,7 +45,8 @@ std::vector windows_msvc_arguments(const toolchain::ToolchainFacts& bool noAlignedAllocation = true); // Semantic arguments any dialect can keep: include paths, macros, standard, forced includes. std::vector semantic_subset(std::span arguments); -// Base arguments for a kit: target, standard, the kit's own arguments, include directories, sysroot. +// Base arguments for a kit: target, standard, the kit's own arguments, include directories, sysroot, +// and COMPILE_ONLY. std::vector kit_arguments(const spec::Kit& kit, std::string_view languageStandard, std::string_view macosSdk); // The -std= value in arguments, or empty. std::string language_standard_of(std::span arguments); diff --git a/src/normalize/msvc.cpp b/src/normalize/msvc.cpp index cb9b8cc..e7562b0 100644 --- a/src/normalize/msvc.cpp +++ b/src/normalize/msvc.cpp @@ -142,6 +142,8 @@ std::vector translate_msvc(const MsvcInput& input) { if (input.facts != nullptr) { for (auto& argument : windows_msvc_arguments(*input.facts, out, input.noAlignedAllocationWithMsvcStl)) out.push_back(std::move(argument)); } + // The engine's clang++ spelling of /c, whatever the build said (fix plan F1). + out.emplace_back(COMPILE_ONLY); if (input.importable) { out.emplace_back("-x"); out.emplace_back("c++-module"); diff --git a/tests/test_normalize.cpp b/tests/test_normalize.cpp index b3ecb34..1a33d4a 100644 --- a/tests/test_normalize.cpp +++ b/tests/test_normalize.cpp @@ -67,7 +67,8 @@ int main() { const auto out = n::translate_gnu(n::GnuInput { arguments, "/p/src/greet/greet.cppm", "/p", &facts, true }); expect(contains(out, "-std=c++23") && contains(out, "-O0") && contains(out, "--sysroot=/opt/subos")); expect(!contains(out, "-fmodules") && !contains_prefix(out, "-fmodule-mapper") && !contains_prefix(out, "-B") && !contains_prefix(out, "-fdeps")); - expect(!contains(out, "-c") && !contains(out, "-o") && !contains(out, "obj/greet.o") && !contains(out, "src/greet/greet.cppm")); + expect(std::ranges::count(out, std::string { "-c" }) == 1) << "the build's -c is replaced by the engine's own, once: " << std::format("{}", out); + expect(!contains(out, "-o") && !contains(out, "obj/greet.o") && !contains(out, "src/greet/greet.cppm")); expect(!contains(out, "-MD") && !contains(out, "-MF") && !contains(out, "x.d")); expect(contains(out, "--no-default-config") && contains(out, "--target=x86_64-linux-gnu") && contains(out, "-stdlib=libstdc++")); expect(contains(out, "--gcc-install-dir=/opt/gcc/lib/gcc/x86_64-linux-gnu/16.1.0")); @@ -111,7 +112,7 @@ int main() { expect(!contains_prefix(out, "/") && !contains_prefix(out, "-interface") && !contains_prefix(out, "-ifc") && !contains_prefix(out, "-reference")) << std::format("{}", out); expect(!contains(out, "std=CMakeFiles\\__cmake_cxx23.dir\\std.ifc") && !contains(out, "CMakeFiles\\greet.dir\\greet.ifc")); - expect(!contains(out, "C:\\p\\src\\greet.ixx") && !contains(out, "-c") && !contains_prefix(out, "-Zi")); + expect(!contains(out, "C:\\p\\src\\greet.ixx") && std::ranges::count(out, std::string { "-c" }) == 1 && !contains_prefix(out, "-Zi")); expect(contains(out, "--no-default-config") && contains(out, "--target=x86_64-pc-windows-msvc")); expect(contains(out, "-fms-compatibility-version=19.44.35228")); expect(contains(out, "-Xmicrosoft-visualc-tools-root") && contains(out, "C:/VS/VC/Tools/MSVC/14.44.35207")); @@ -163,7 +164,7 @@ int main() { kit.systemIncludeDirectories = { "/kit/include/c++/v1", "/kit/include" }; const auto out = n::kit_arguments(kit, "c++26", ""); const std::vector expected { "--no-default-config", "--target=x86_64-w64-mingw32", "-std=c++26", "-nostdinc++", - "-nostdlibinc", "-isystem", "/kit/include/c++/v1", "-isystem", "/kit/include" }; + "-nostdlibinc", "-isystem", "/kit/include/c++/v1", "-isystem", "/kit/include", "-c" }; expect(out == expected) << std::format("{}", out); const std::vector command { "cl.exe", "/IC:/inc", "/DA=1", "/std:c++20", "-O2", "/FIpch.h", "-I", "x" }; const auto subset = n::semantic_subset(command); @@ -172,6 +173,81 @@ int main() { expect(n::language_standard_of(command) == "c++20"); }; + // Fix plan F1 (issue #23): a windows-msvc command with -flto and without -c makes the driver plan a link, and + // its `LTO requires -fuse-ld=lld` fails clangd's module scan, so no module is built. Every engine command + // carries exactly one -c, and the link arguments stay: the project's LTO is not the engine's business. + "every engine command stops before linking, and keeps -flto"_test = [] { + const auto once = [](const std::vector& out) { return std::ranges::count(out, std::string { "-c" }) == 1; }; + auto clangxx = msvc_facts(s::Family::clang); + clangxx.toolchain.driver = "C:/LLVM/bin/clang++.exe"; + const std::vector gnu { "C:/LLVM/bin/clang++.exe", "--target=x86_64-pc-windows-msvc", "-std=c++23", "-flto", "-O2", + "-c", "-c", "D:/w/src/m.cppm", "-o", "D:/w/obj/m.obj" }; + for (const bool importable : { true, false }) { + const auto out = n::translate_gnu(n::GnuInput { gnu, "D:/w/src/m.cppm", "D:/w", &clangxx, importable }); + expect(once(out) && contains(out, "-flto") && contains(out, "--target=x86_64-pc-windows-msvc")) << std::format("{}", out); + expect(importable ? out.back() == "c++-module" : !contains(out, "c++-module")) << std::format("{}", out); + } + const auto clangCl = msvc_facts(s::Family::clang_cl); + const std::vector cl { "C:/LLVM/bin/clang-cl.exe", "/std:c++latest", "-flto", "/GL", "/MD", "/c", "-c", "C:/p/src/m.ixx", + "/FoC:/p/obj/m.obj" }; + const auto fromCl = n::translate_msvc(n::MsvcInput { cl, "C:/p/src/m.ixx", "C:/p", &clangCl, true }); + expect(once(fromCl) && contains(fromCl, "-flto") && !contains(fromCl, "/c") && !contains_prefix(fromCl, "/")) << std::format("{}", fromCl); + const auto cmsvc = msvc_facts(s::Family::msvc); + const std::vector msvc { "cl.exe", "/std:c++latest", "/GL", "/c", "C:/p/src/main.cpp" }; + expect(once(n::translate_msvc(n::MsvcInput { msvc, "C:/p/src/main.cpp", "C:/p", &cmsvc, false }))); + s::Kit kit; + kit.target = "x86_64-pc-windows-msvc"; + expect(once(n::kit_arguments(kit, "c++23", ""))); + + // The plan: project units, a file the editor opened outside the model, a stand-in, prime units and std all + // come from these arguments, and each carries the one -c. + const std::map sources { + { "/w/src/main.cpp", "import std;\nimport app.core;\nimport missing.module;\nint main() {}\n" }, + { "/w/src/core.cppm", "export module app.core;\nimport std;\n" }, + { "/w/apps/tool.cpp", "import app.core;\nint tool() { return 0; }\n" }, + }; + s::Database database; + database.hasIde = true; + s::Set set; + set.name = "app"; + set.hasIde = true; + set.toolchain = "llvm-22.1.8-x86_64-pc-windows-msvc"; + for (const std::string path : { "/w/src/main.cpp", "/w/src/core.cppm" }) { + s::TranslationUnit unit; + unit.source = path; + unit.workDirectory = "/w"; + unit.arguments = { "clang++", "--target=x86_64-pc-windows-msvc", "-std=c++23", "-flto", "-c", path, "-o", path + ".obj" }; + set.units.push_back(std::move(unit)); + } + database.sets.push_back(set); + std::map> facts { { set.toolchain, clangxx } }; + n::PlanInput input; + input.database = &database; + input.facts = &facts; + input.engineDriverDirectory = "/payload/clangd/bin"; + input.primeDirectory = "/cache/prime"; + input.stubDirectory = "/cache/stubs"; + input.openSources = { "/w/apps/tool.cpp" }; + input.scanner = [&](std::string_view path) { + const auto it = sources.find(std::string { path }); + return it == sources.end() ? p::ScanResult {} : p::scan_source(it->second); + }; + input.metadataReader = [](std::string_view) { + return std::vector { { "std", "C:/VS/VC/Tools/MSVC/14.44.35207/modules/std.ixx", true, {}, {} } }; + }; + const auto plan = n::plan_engine(input); + std::set kinds; + for (const auto& entry : plan.entries) { + expect(once(entry.arguments) && contains(entry.arguments, "-flto")) << entry.file << ": " << std::format("{}", entry.arguments); + if (entry.file.starts_with("/cache/prime/")) kinds.insert("prime"); + else if (entry.file.starts_with("/cache/stubs/")) kinds.insert("stand-in"); + else if (entry.file.ends_with("std.ixx")) kinds.insert("std"); + else if (entry.file == "/w/apps/tool.cpp") kinds.insert("opened"); + else kinds.insert("unit"); + } + expect(kinds == std::set { "opened", "prime", "stand-in", "std", "unit" }) << std::format("{}", kinds); + }; + "a plan resolves, injects std once and leaves out what cannot resolve"_test = [] { const std::string root { mcppls::base::join_path(mcppls::platform::dirs::temp_directory(), std::format("mcppls-test-plan-{}", std::chrono::steady_clock::now().time_since_epoch().count())) }; From 1e3fcb8d07d81d73cb70ef455f66c0b1c734b205 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 26 Sep 2026 07:26:09 +0800 Subject: [PATCH 04/31] feat(completion): a space after import opens the module list, and the module keywords come from mcppls whatever clangd is doing A space is now a completion trigger character for clients that drop the other spaces themselves (VS Code and its forks, by clientInfo) or that ask for it with initializationOptions.completion.triggerOnSpace; false turns it off for anyone. The workspace gates every space-triggered request at its routing entry on the line's own text before the cursor: only `[export] import ` followed by nothing passes, and goes to mcppls's own engine for the module names; anything else is answered at once with an empty list, never reaching clangd, the index or the log. The report counts what that cost (roots[].completion.spaceTrigger). Import completion's module names are kept by the index until a module is declared, dropped or changes role, instead of being rebuilt, each with a scan of every file, on every request. The module-syntax keywords -- import, export import, module;, export module, module and module :private; -- are offered where each can begin a declaration: at global scope, outside comments and literals, each only where it can appear (export import in an interface unit, module; before anything else, the declarations in a file that has none, the private fragment once in a primary interface). clangd offers some of them but not the combined forms, and nothing at all while it is stuck on the file, which is how a spinning clangd took `import` away in the hello project. They are merged into clangd's answer without duplicate labels, replace an error from it, and go out on their own, as an incomplete list, when clangd has not answered within 1.5 s. For VS Code, accepting import or export import opens the module list at once. After `export module ` nothing is suggested (D5). Fix plan 2026-09-26 F9 and F15, decisions D4 and D5. --- src/engine/native/index.cpp | 69 ++++++++-- src/engine/native/index.cppm | 13 ++ src/engine/native/keywords.cpp | 168 +++++++++++++++++++++++++ src/engine/native/keywords.cppm | 27 ++++ src/orchestrator/completion.cpp | 101 +++++++++++++++ src/orchestrator/completion.cppm | 54 ++++++++ src/orchestrator/workspace.cpp | 112 ++++++++++++++++- src/server/session.cpp | 4 + tests/test_completion.cpp | 208 +++++++++++++++++++++++++++++++ 9 files changed, 743 insertions(+), 13 deletions(-) create mode 100644 src/engine/native/keywords.cpp create mode 100644 src/engine/native/keywords.cppm create mode 100644 src/orchestrator/completion.cpp create mode 100644 src/orchestrator/completion.cppm create mode 100644 tests/test_completion.cpp diff --git a/src/engine/native/index.cpp b/src/engine/native/index.cpp index a737d85..fd2f273 100644 --- a/src/engine/native/index.cpp +++ b/src/engine/native/index.cpp @@ -67,17 +67,63 @@ Json make_location(std::string_view path, const base::Range& range) { return Json { { "uri", base::path_to_uri(path) }, { "range", to_json(range) } }; } +namespace { + +// What a file contributes to the module names import completion offers: the name it declares and +// the role it has. Anything else about it can change without changing those. +std::string provided_identity(const project::ScanResult& scan) { + if (!scan.declaration) return {}; + return std::format("{}|{}", declared_name(*scan.declaration), spec::to_string(project::role_of(scan))); +} + +} // namespace + void ModuleIndex::update(std::string_view path, std::string_view text) { - files_[base::path_key(path)] = { std::string { path }, project::scan_source(text) }; + auto scan = project::scan_source(text); + auto& entry = files_[base::path_key(path)]; + if (provided_identity(entry.second) != provided_identity(scan)) structure_changed_(); + entry = { std::string { path }, std::move(scan) }; } void ModuleIndex::remove(std::string_view path) { - if (const auto it = files_.find(base::path_key(path)); it != files_.end()) files_.erase(it); + if (const auto it = files_.find(base::path_key(path)); it != files_.end()) { + if (it->second.second.declaration) structure_changed_(); + files_.erase(it); + } } -void ModuleIndex::clear() { files_.clear(); } +void ModuleIndex::clear() { + files_.clear(); + structure_changed_(); +} + +void ModuleIndex::set_external(std::vector modules) { + external_ = std::move(modules); + structure_changed_(); +} -void ModuleIndex::set_external(std::vector modules) { external_ = std::move(modules); } +void ModuleIndex::structure_changed_() { + ++structure_; + candidates_.reset(); +} + +const std::vector& ModuleIndex::candidates_now_() const { + if (candidates_) return *candidates_; + std::map> byName; // sorted by name, like module_names + for (const auto& [key, entry] : files_) { + const auto& scan = entry.second; + if (!scan.declaration) continue; + const spec::Role role { project::role_of(scan) }; + if (role == spec::Role::module_implementation) continue; + byName.try_emplace(declared_name(*scan.declaration), role_label(role)); // the first provider's role, like providers().front() + } + for (const auto& module : external_) byName.try_emplace(module.name, "module"); + std::vector built; + built.reserve(byName.size()); + for (auto& [name, detail] : byName) built.push_back(Candidate { name, std::move(detail) }); + candidates_ = std::move(built); + return *candidates_; +} void ModuleIndex::set_profile_label(std::string label) { profileLabel_ = std::move(label); } @@ -212,20 +258,19 @@ Json ModuleIndex::completion(std::string_view path, std::string_view text, base: if (partial.starts_with(':')) { if (scan == nullptr || !scan->declaration) return Json { { "isIncomplete", false }, { "items", items } }; const std::string owner { scan->declaration->module }; - for (const auto& name : module_names()) { - if (!name.starts_with(owner + ":")) continue; - std::string label { name.substr(owner.size()) }; + for (const auto& candidate : candidates_now_()) { + if (!candidate.name.starts_with(owner + ":")) continue; + std::string label { candidate.name.substr(owner.size()) }; if (label.starts_with(partial) && !(scan->declaration->partition.size() && label == ":" + scan->declaration->partition)) { add(label, "module partition"); } } } else { const std::string self { scan != nullptr && scan->declaration ? declared_name(*scan->declaration) : std::string {} }; - for (const auto& name : module_names()) { - if (name.find(':') != std::string::npos || name == self) continue; - if (!name.starts_with(partial)) continue; - const auto units = providers(name); - add(name, units.empty() ? std::string_view { "module" } : std::string_view { role_label(units.front().role) }); + for (const auto& candidate : candidates_now_()) { + if (candidate.name.find(':') != std::string::npos || candidate.name == self) continue; + if (!candidate.name.starts_with(partial)) continue; + add(candidate.name, candidate.detail); } } return Json { { "isIncomplete", false }, { "items", items } }; diff --git a/src/engine/native/index.cppm b/src/engine/native/index.cppm index b1a33b3..c06210d 100644 --- a/src/engine/native/index.cppm +++ b/src/engine/native/index.cppm @@ -36,6 +36,17 @@ private: std::map, std::less<>> files_; // path key -> (path, scan) std::vector external_; std::string profileLabel_; + // What import completion offers (fix plan 2026-09-26 F9, D4): every module name with what + // provides it, built once and kept until a module is declared, dropped or changes role -- + // not on every edit, which leaves the declarations as they were. + struct Candidate { + std::string name; // "m" or "m:p" + std::string detail; // the role of the unit that provides it, or where an external module comes from + }; + mutable std::optional> candidates_; + std::uint64_t structure_ { 0 }; + const std::vector& candidates_now_() const; + void structure_changed_(); public: void update(std::string_view path, std::string_view text); @@ -50,6 +61,8 @@ public: std::vector providers(std::string_view name) const; const ExternalModule* external(std::string_view name) const; std::vector module_names() const; + // Advanced whenever the set of declared modules, or the role of a unit declaring one, changes. + std::uint64_t structure_generation() const { return structure_; } std::optional module_at(std::string_view path, base::Position position) const; // Each returns null when the position is not one this index answers for. diff --git a/src/engine/native/keywords.cpp b/src/engine/native/keywords.cpp new file mode 100644 index 0000000..1782c25 --- /dev/null +++ b/src/engine/native/keywords.cpp @@ -0,0 +1,168 @@ +module mcppls.engine.native.keywords; + +import std; +import nlohmann.json; +import mcppls.base.text; +import mcppls.project.scan; + +namespace mcppls::index { + +using Json = nlohmann::json; + +namespace { + +constexpr int COMPLETION_KIND_KEYWORD { 14 }; + +bool is_blank(char c) { return c == ' ' || c == '\t' || c == '\v' || c == '\f' || c == '\r'; } +bool is_word_char(char c) { return base::is_identifier_char(c); } + +struct Keyword { + std::string_view label; + std::string_view insert; + std::string_view detail; + bool opensModules; +}; + +constexpr std::array KEYWORDS { { + { "import", "import ", "import a module", true }, + { "export import", "export import ", "import a module and export it", true }, + { "module;", "module;", "start the global module fragment", false }, + { "export module", "export module ", "declare a module interface unit", false }, + { "module", "module ", "declare a module implementation unit", false }, + { "module :private;", "module :private;", "start the private module fragment", false }, +} }; + +// What the text before a line says about it: whether it is at global scope, and whether anything +// but comments and blanks comes before it. Braces are counted outside comments, string and +// character literals, and preprocessor lines (a macro's braces are not the file's). +struct Before { + int depth { 0 }; + bool inComment { false }; // the line starts inside a block comment + bool anything { false }; // a token or a preprocessor line comes before the line +}; + +Before read_before(std::string_view text) { + Before before; + bool lineStart { true }; + for (std::size_t i { 0 }; i < text.size(); ++i) { + const char c { text[i] }; + if (c == '\n') { + lineStart = true; + continue; + } + if (c == '/' && i + 1 < text.size() && text[i + 1] == '/') { + const std::size_t end { text.find('\n', i) }; + if (end == std::string_view::npos) break; + i = end - 1; + continue; + } + if (c == '/' && i + 1 < text.size() && text[i + 1] == '*') { + const std::size_t end { text.find("*/", i + 2) }; + if (end == std::string_view::npos) { + before.inComment = true; + break; + } + i = end + 1; + continue; + } + if (is_blank(c)) continue; + before.anything = true; + if (c == '#' && lineStart) { + // A preprocessor line, continuation lines included. + std::size_t end { i }; + while (true) { + end = text.find('\n', end); + if (end == std::string_view::npos || end == 0) break; + std::size_t back { end - 1 }; + if (text[back] == '\r' && back > 0) --back; + if (text[back] != '\\') break; + ++end; + } + if (end == std::string_view::npos) break; + i = end - 1; + continue; + } + lineStart = false; + if (c == 'R' && i + 1 < text.size() && text[i + 1] == '"' && (i == 0 || !is_word_char(text[i - 1]) || text[i - 1] == '8' + || text[i - 1] == 'u' || text[i - 1] == 'U' || text[i - 1] == 'L')) { + // A raw string literal: R"delimiter( ... )delimiter" + const std::size_t open { text.find('(', i + 2) }; + if (open == std::string_view::npos) break; + const std::string closing { ")" + std::string { text.substr(i + 2, open - i - 2) } + "\"" }; + const std::size_t end { text.find(closing, open + 1) }; + if (end == std::string_view::npos) break; + i = end + closing.size() - 1; + continue; + } + if (c == '"' || c == '\'') { + // A digit separator (1'000) is no character literal. + if (c == '\'' && i > 0 && std::isxdigit(static_cast(text[i - 1]))) continue; + std::size_t j { i + 1 }; + while (j < text.size() && text[j] != c && text[j] != '\n') j += text[j] == '\\' ? 2 : 1; + i = std::min(j, text.size() - 1); + continue; + } + if (c == '{') ++before.depth; + else if (c == '}' && before.depth > 0) --before.depth; + } + return before; +} + +bool has_private_fragment(std::string_view text) { + return text.find("module :private") != std::string_view::npos || text.find("module:private") != std::string_view::npos; +} + +} // namespace + +Json keyword_completion(std::string_view text, base::Position position, const project::ScanResult* scan, KeywordOptions options) { + const auto offset = base::offset_at(text, position); + if (!offset) return nullptr; + const std::size_t newline { *offset == 0 ? std::string_view::npos : text.rfind('\n', *offset - 1) }; + const std::size_t lineStart { newline == std::string_view::npos ? 0 : newline + 1 }; + // The cursor is at the end of what was typed, not inside a word. + if (*offset < text.size() && is_word_char(text[*offset])) return nullptr; + std::size_t headStart { lineStart }; + while (headStart < *offset && is_blank(text[headStart])) ++headStart; + const std::string_view head { text.substr(headStart, *offset - headStart) }; + // A partial word, or `export`, blanks and a partial word; `export` alone is a partial word too. + std::string typed { head }; + if (!std::ranges::all_of(head, is_word_char)) { + if (!head.starts_with("export") || head.size() == 6 || !is_blank(head[6])) return nullptr; + std::string_view word { head.substr(6) }; + while (!word.empty() && is_blank(word.front())) word.remove_prefix(1); + if (!std::ranges::all_of(word, is_word_char)) return nullptr; + typed = "export " + std::string { word }; + } + if (std::ranges::none_of(KEYWORDS, [&](const Keyword& keyword) { return keyword.label.starts_with(typed); })) return nullptr; + const Before before { read_before(text.substr(0, lineStart)) }; + if (before.inComment || before.depth != 0) return nullptr; + + const auto& declaration = scan != nullptr ? scan->declaration : std::optional {}; + const bool declaredBefore { declaration && declaration->nameRange.start.line < position.line }; + const bool declaredElsewhere { declaration && declaration->nameRange.start.line != position.line }; + const bool interfaceUnit { declaredBefore && declaration->isExported }; + const auto applies = [&](const Keyword& keyword) { + if (keyword.label == "import") return true; + if (keyword.label == "export import") return interfaceUnit; + if (keyword.label == "module;") return !before.anything; + if (keyword.label == "export module" || keyword.label == "module") return !declaredElsewhere; + if (keyword.label == "module :private;") return interfaceUnit && declaration->partition.empty() && !has_private_fragment(text); + return false; + }; + const Json range { { "start", Json { { "line", position.line }, { "character", base::position_at(text, headStart).character } } }, + { "end", Json { { "line", position.line }, { "character", position.character } } } }; + Json items = Json::array(); + for (const auto& keyword : KEYWORDS) { + if (!keyword.label.starts_with(typed) || !applies(keyword)) continue; + Json item { { "label", std::string { keyword.label } }, { "kind", COMPLETION_KIND_KEYWORD }, { "detail", std::string { keyword.detail } }, + { "filterText", std::string { keyword.label } }, { "insertTextFormat", 1 }, + { "textEdit", Json { { "range", range }, { "newText", std::string { keyword.insert } } } } }; + if (keyword.opensModules && options.suggestModulesAfterImport) { + item["command"] = Json { { "title", "Suggest modules" }, { "command", "editor.action.triggerSuggest" } }; + } + items.push_back(std::move(item)); + } + return items; +} + +} // namespace mcppls::index diff --git a/src/engine/native/keywords.cppm b/src/engine/native/keywords.cppm new file mode 100644 index 0000000..e3a9f1a --- /dev/null +++ b/src/engine/native/keywords.cppm @@ -0,0 +1,27 @@ +// Module-syntax keyword completion (fix plan 2026-09-26 F15, decision D5): `import`, +// `export import`, `module;`, `export module`, `module` and `module :private;` where each can +// begin a declaration, from the text alone. clangd offers some of them, but not the combined +// forms, and nothing at all while it is stuck on the file; these come from mcppls whatever +// clangd is doing, and are merged with what clangd answers. +export module mcppls.engine.native.keywords; + +import std; +import nlohmann.json; +import mcppls.base.text; +import mcppls.project.scan; + +export namespace mcppls::index { + +struct KeywordOptions { + // For a client that runs VS Code's commands: accepting `import` or `export import` opens the + // module list at once (editor.action.triggerSuggest), without waiting for a space. + bool suggestModulesAfterImport { false }; +}; + +// The keyword items for `position` in `text`: at the start of a line at global scope (outside any +// braces, comments and literals), with a partial word, or `export` and a partial word, before the +// cursor. `scan` is the file's scan, for which declarations the file already has. An empty array +// when the position is such a place and no keyword fits what was typed; null when it is not. +nlohmann::json keyword_completion(std::string_view text, base::Position position, const project::ScanResult* scan, KeywordOptions options = {}); + +} // namespace mcppls::index diff --git a/src/orchestrator/completion.cpp b/src/orchestrator/completion.cpp new file mode 100644 index 0000000..78a035b --- /dev/null +++ b/src/orchestrator/completion.cpp @@ -0,0 +1,101 @@ +module mcppls.orchestrator.completion; + +import std; +import nlohmann.json; +import mcppls.base.text; +import mcppls.lsp.jsonrpc; + +namespace mcppls::orchestrator::completion { + +namespace { + +bool is_blank(char c) { return c == ' ' || c == '\t' || c == '\v' || c == '\f'; } + +std::size_t skip_blanks(std::string_view& text) { + std::size_t skipped { 0 }; + while (skipped < text.size() && is_blank(text[skipped])) ++skipped; + text.remove_prefix(skipped); + return skipped; +} + +std::string lowercase(std::string_view text) { + std::string lowered { text }; + std::ranges::transform(lowered, lowered.begin(), [](unsigned char c) { return static_cast(std::tolower(c)); }); + return lowered; +} + +} // namespace + +bool is_import_line_prefix(std::string_view linePrefix) { + std::string_view rest { linePrefix }; + skip_blanks(rest); + if (rest.starts_with("export")) { + std::string_view afterExport { rest.substr(6) }; + // `export` and `import` are two words: `exportimport ` is neither. + if (skip_blanks(afterExport) > 0) rest = afterExport; + } + return rest.size() == 7 && rest.starts_with("import") && is_blank(rest[6]); +} + +std::optional line_prefix(std::string_view text, base::Position position) { + const auto offset = base::offset_at(text, position); + if (!offset) return std::nullopt; + const std::size_t newline { *offset == 0 ? std::string_view::npos : text.rfind('\n', *offset - 1) }; + const std::size_t start { newline == std::string_view::npos ? 0 : newline + 1 }; + return text.substr(start, *offset - start); +} + +bool is_space_trigger(const Json& params) { + const Json* context { lsp::find(params, "context") }; + if (context == nullptr || !context->is_object()) return false; + return lsp::int_at(*context, "triggerKind").value_or(0) == 2 && context->value("triggerCharacter", std::string {}) == " "; +} + +bool vscode_like(const Json& clientParams) { + const Json* info { lsp::find(clientParams, "clientInfo") }; + if (info == nullptr || !info->is_object()) return false; + const std::string name { lowercase(info->value("name", std::string {})) }; + if (name.starts_with("visual studio code")) return true; // and "- Insiders" + static constexpr std::array FORKS { "code - oss", "vscodium", "cursor", "windsurf", "trae", "positron" }; + return std::ranges::find(FORKS, name) != FORKS.end(); +} + +bool space_trigger_wanted(const Json& clientParams) { + if (const Json* option = lsp::find_path(clientParams, { "initializationOptions", "completion", "triggerOnSpace" }); option != nullptr && option->is_boolean()) { + return option->get(); + } + return vscode_like(clientParams); +} + +void add_space_trigger(Json& capabilities) { + if (!capabilities.is_object()) return; + Json& provider = capabilities["completionProvider"]; + if (!provider.is_object()) provider = Json::object(); + Json& triggers = provider["triggerCharacters"]; + if (!triggers.is_array()) triggers = Json::array(); + if (std::ranges::find(triggers, Json(" ")) == triggers.end()) triggers.push_back(" "); +} + +Json empty_list() { return Json { { "isIncomplete", false }, { "items", Json::array() } }; } + +Json merge(const Json& engineResult, const Json& keywordItems) { + if (!keywordItems.is_array() || keywordItems.empty()) return engineResult; + Json list = engineResult.is_object() ? engineResult : Json { { "isIncomplete", false }, { "items", engineResult.is_array() ? engineResult : Json::array() } }; + if (!list.contains("items") || !list["items"].is_array()) list["items"] = Json::array(); + if (!list.contains("isIncomplete") || !list["isIncomplete"].is_boolean()) list["isIncomplete"] = false; + Json& items = list["items"]; + std::set> labels; + for (const auto& item : items) { + if (item.is_object()) labels.emplace(base::trim(item.value("label", std::string {}))); + } + for (const auto& keyword : keywordItems) { + if (labels.emplace(base::trim(keyword.value("label", std::string {}))).second) items.push_back(keyword); + } + return list; +} + +Json keywords_only(const Json& keywordItems) { + return Json { { "isIncomplete", true }, { "items", keywordItems.is_array() ? keywordItems : Json::array() } }; +} + +} // namespace mcppls::orchestrator::completion diff --git a/src/orchestrator/completion.cppm b/src/orchestrator/completion.cppm new file mode 100644 index 0000000..2eb0462 --- /dev/null +++ b/src/orchestrator/completion.cppm @@ -0,0 +1,54 @@ +// Completion routing (fix plan 2026-09-26 F9, F15, decisions D4, D5): a space after `import` +// opens the module list, and module-syntax keywords come from mcppls whatever the core engine is +// doing. Pure functions: the workspace calls them on its routing path. +// +// A space is a trigger character only where it cannot cost anything (D4). The editor drops a +// space-triggered request outside an import line before it is sent (VS Code's middleware); the +// server answers one that arrives anyway at once, empty, from the line's own text, without asking +// an engine; the module names are cached by the index; and the space is advertised only to +// clients that do the first step, or that ask for it. +export module mcppls.orchestrator.completion; + +import std; +import nlohmann.json; +import mcppls.base.text; + +export namespace mcppls::orchestrator::completion { + +using Json = nlohmann::json; + +// `^\s*(export\s+)?import\s$`: the text of a line before the cursor is an import directive's +// keyword and exactly one blank after it, with nothing typed after that. +bool is_import_line_prefix(std::string_view linePrefix); + +// The text of the line `position` is on, before it; nullopt when the position is not in `text`. +std::optional line_prefix(std::string_view text, base::Position position); + +// A completion request the client sent because a space was typed (CompletionTriggerKind.TriggerCharacter). +bool is_space_trigger(const Json& params); + +// The client runs this repository's VS Code extension, or one built on the same extension host: +// VS Code and its forks name themselves in clientInfo (Visual Studio Code, Code - OSS, VSCodium, +// Cursor, Windsurf, Trae, Positron). Such a client also runs the extension's middleware. +bool vscode_like(const Json& clientParams); + +// D4 layer 4: whether the server advertises the space as a trigger character to this client: +// initializationOptions.completion.triggerOnSpace when the client gave it, else vscode_like. +bool space_trigger_wanted(const Json& clientParams); + +// Adds " " to capabilities.completionProvider.triggerCharacters. +void add_space_trigger(Json& capabilities); + +// The answer to a space-triggered request that is not on an import line: an empty, complete list. +Json empty_list(); + +// F15: the core engine's completion result (null, CompletionItem[] or CompletionList) with the +// module-syntax keyword items added. A keyword whose label the engine already gave is not added +// twice. `isIncomplete` is the engine's; null stays null when there are no keywords. +Json merge(const Json& engineResult, const Json& keywordItems); + +// Only keywords, as an incomplete list: the core engine did not answer in time, and the client +// asks again as the person types on. +Json keywords_only(const Json& keywordItems); + +} // namespace mcppls::orchestrator::completion diff --git a/src/orchestrator/workspace.cpp b/src/orchestrator/workspace.cpp index 258bace..257249e 100644 --- a/src/orchestrator/workspace.cpp +++ b/src/orchestrator/workspace.cpp @@ -31,6 +31,8 @@ import mcppls.normalize.plan; import mcppls.engine; import mcppls.engine.payload; import mcppls.engine.native.index; +import mcppls.engine.native.keywords; +import mcppls.orchestrator.completion; import mcppls.orchestrator.journal; import mcppls.orchestrator.client; import mcppls.orchestrator.documents; @@ -62,6 +64,11 @@ constexpr std::array BUILD_FILES { "mcpp.toml", "mcpp.lock" constexpr std::array WATCH_POLL_SKIP_DIRECTORIES { "target", "build", "node_modules", "out", "_build", "cmake-build-debug", "cmake-build-release" }; +// F15 (fix plan 2026-09-26): a completion at the start of a declaration waits this long for the core +// engine; past it the module-syntax keywords go out alone, as an incomplete list the client asks +// again for as the person types on. A clangd stuck on the file no longer takes `import` with it. +constexpr std::chrono::milliseconds KEYWORD_PATIENCE { 1500 }; + // Changes to cxxModules/status that keep its state are sent at most this often (S3 4). constexpr std::chrono::milliseconds STATUS_COALESCE { 250 }; // import-hang plan §6: a change from a working state to degraded goes out only once it has lasted this @@ -200,6 +207,16 @@ struct Workspace::Impl final : engine::Host { std::map> answeredBy; }; std::map> requestStats; + // F9 (D4): what space-triggered completion costs. Most never pass the gate and cost one look at a line. + struct SpaceTriggerStats { + std::size_t count { 0 }; + std::size_t passed { 0 }; + std::int64_t maxMicros { 0 }; + std::int64_t totalMicros { 0 }; + }; + SpaceTriggerStats spaceTrigger; + std::size_t keywordsWithoutEngine { 0 }; // F15: keyword completions answered before the core engine did + bool vscodeLike { false }; // the client runs VS Code's commands (completion::vscode_like) // Requests in flight across engines. struct Job { @@ -219,6 +236,10 @@ struct Workspace::Impl final : engine::Host { std::string path; std::string text; Json message; + // F15: a completion's module-syntax keywords, merged into whatever the engines answer, and + // when they go out without the core engine. + Json keywords; + std::optional keywordsBy; }; std::map jobs; std::uint64_t nextJob { 1 }; @@ -451,6 +472,7 @@ struct Workspace::Impl final : engine::Host { consider(statusFlushAt); consider(leaseRenewAt); consider(tokensRefreshAt); + for (const auto& [id, job] : jobs) consider(job.keywordsBy); for (const auto& engine : engines) consider(engine->next_deadline()); return deadline; } @@ -528,6 +550,18 @@ struct Workspace::Impl final : engine::Host { const Document* document { uri.empty() ? nullptr : documents_.find(uri) }; if (document != nullptr) job.text = document->text; job.view = engine::RequestView { job.method, &job.params, job.path, job.text }; + if (job.method == lsp::method::TEXT_DOCUMENT_COMPLETION) { + if (completion::is_space_trigger(job.params)) { + route_space_triggered(jobId); + return; + } + if (document != nullptr && !job.path.empty()) { + if (const auto position = position_of(job.params)) { + job.keywords = index::keyword_completion(job.text, *position, index.scan_of(job.path), + index::KeywordOptions { .suggestModulesAfterImport = vscodeLike }); + } + } + } std::vector candidates; for (const auto& engine : engines) candidates.push_back(engine.get()); @@ -560,9 +594,61 @@ struct Workspace::Impl final : engine::Host { return; } job.answerers = std::move(selection.answerers); + if (job.keywords.is_array() && !job.keywords.empty() && coreEngine != nullptr + && std::ranges::find(job.answerers, coreEngine) != job.answerers.end()) { + job.keywordsBy = job.started + KEYWORD_PATIENCE; + } + ask_next(jobId); + } + + static std::optional position_of(const Json& params) { + const Json* position { lsp::find(params, "position") }; + if (position == nullptr || !position->is_object()) return std::nullopt; + const auto line = lsp::int_at(*position, "line"); + const auto character = lsp::int_at(*position, "character"); + if (!line || !character) return std::nullopt; + return base::Position { static_cast(*line), static_cast(*character) }; + } + + // F9 (D4 layer 2): a completion the client asked for because a space was typed. Only the line + // before the cursor is looked at: anything but `[export] import ` is answered at once, empty, + // without an engine, a lookup or a log line; an import line gets the module names from mcppls's + // own engine, never from the core engine. + void route_space_triggered(std::uint64_t jobId) { + Job& job = jobs.at(jobId); + const auto gateStart = Clock::now(); + std::optional prefix; + if (const auto position = position_of(job.params)) prefix = completion::line_prefix(job.text, *position); + const bool passed { prefix && completion::is_import_line_prefix(*prefix) }; + const std::int64_t micros { std::chrono::duration_cast(Clock::now() - gateStart).count() }; + ++spaceTrigger.count; + spaceTrigger.maxMicros = std::max(spaceTrigger.maxMicros, micros); + spaceTrigger.totalMicros += micros; + if (!passed) { + const Json id = job.clientId; + jobs.erase(jobId); + client.reply(id, completion::empty_list()); + return; + } + ++spaceTrigger.passed; + if (moduleEngine != nullptr && moduleEngine->claims(job.view)) job.answerers = { moduleEngine }; + if (job.answerers.empty()) { + finish_job(jobId, completion::empty_list()); + return; + } ask_next(jobId); } + // F15: the keywords go out without the core engine, which has not answered in KEYWORD_PATIENCE. + // The job is finished first, so the engines' answers to the cancellation find nothing to finish. + void answer_keywords_without_engine(std::uint64_t jobId) { + const Json clientId { jobs.at(jobId).clientId }; + ++keywordsWithoutEngine; + jobs.at(jobId).answeredBy = "mcppls"; + finish_job(jobId, Json(nullptr)); + for (const auto& engine : engines) engine->cancel(clientId); + } + void ask_next(std::uint64_t jobId) { auto it = jobs.find(jobId); if (it == jobs.end()) return; @@ -585,7 +671,11 @@ struct Workspace::Impl final : engine::Host { ask_next(jobId); return; case engine::Answer::Kind::unavailable: ask_next(jobId); return; - case engine::Answer::Kind::error: finish_job_with_error(jobId, std::move(answer.value)); return; + case engine::Answer::Kind::error: + // F15: a completion that has keywords to give gives them rather than the engine's error. + if (current->second.keywords.is_array() && !current->second.keywords.empty()) finish_job(jobId, Json(nullptr)); + else finish_job_with_error(jobId, std::move(answer.value)); + return; case engine::Answer::Kind::cancelled: finish_job_cancelled(jobId); return; } }); @@ -663,6 +753,11 @@ struct Workspace::Impl final : engine::Host { void finish_job(std::uint64_t jobId, Json result) { auto it = jobs.find(jobId); if (it == jobs.end()) return; + // F15: the module-syntax keywords, with the engine's items, or alone (and incomplete, so the + // client asks again) when the engine gave none. + if (it->second.keywords.is_array() && !it->second.keywords.empty()) { + result = result.is_null() ? completion::keywords_only(it->second.keywords) : completion::merge(result, it->second.keywords); + } result = explain_if_preparing(it->second, std::move(result)); note_request(it->second, result.is_null() ? "empty" : "result"); const Json id = it->second.clientId; @@ -1419,6 +1514,15 @@ struct Workspace::Impl final : engine::Host { void handle_timers() { const auto now = Clock::now(); for (const auto& engine : engines) engine->handle_timers(); + { + std::vector due; + for (const auto& [id, job] : jobs) { + if (job.keywordsBy && *job.keywordsBy <= now) due.push_back(id); + } + for (const auto id : due) { + if (jobs.contains(id)) answer_keywords_without_engine(id); + } + } if (leaseRenewAt && *leaseRenewAt <= now) { lease->renew(std::chrono::system_clock::now()); leaseRenewAt = now + LEASE_RENEWAL; @@ -1499,6 +1603,7 @@ void Workspace::start(Json clientParams, bool clientSupportsStatus, bool usePoll impl_->clientSupportsTokensRefresh = supported->get(); } impl_->onEngineSettled = std::move(onEngineSettled); + impl_->vscodeLike = completion::vscode_like(impl_->clientParams); impl_->dynamicWatch = !usePolling; if (usePolling) impl_->start_watch_polling(); log::info("mcppls {} ({}) root {}", base::VERSION, mcppls::os::FAMILY_NAME, root_); @@ -1774,10 +1879,15 @@ Json Workspace::report() const { { "p50Ms", percentile(0.5) }, { "p95Ms", percentile(0.95) }, { "maxMs", static_cast(stats.maxMs) }, { "answeredBy", std::move(answeredBy) } }; } + // F9, F15: what space-triggered completion cost, and how often keywords answered without the core engine. + Json completionCosts { { "spaceTrigger", Json { { "count", impl.spaceTrigger.count }, { "passed", impl.spaceTrigger.passed }, + { "maxMicros", impl.spaceTrigger.maxMicros }, { "totalMicros", impl.spaceTrigger.totalMicros } } }, + { "keywordsWithoutEngine", impl.keywordsWithoutEngine } }; return Json { { "root", root_ }, { "key", key_ }, { "cacheDirectory", impl.cacheDirectory }, { "trusted", impl.options.trusted }, { "state", std::string { to_string(impl.compute_state()) } }, { "project", std::move(project) }, { "toolEnvironment", std::move(environment) }, { "toolRuns", std::move(toolRuns) }, { "plan", std::move(plan) }, { "engines", std::move(engines) }, { "requests", std::move(requests) }, + { "completion", std::move(completionCosts) }, { "eventTotals", impl.journal.totals() }, { "events", impl.journal.recent(300) } }; } diff --git a/src/server/session.cpp b/src/server/session.cpp index 32ab3e7..5eb7ffb 100644 --- a/src/server/session.cpp +++ b/src/server/session.cpp @@ -20,6 +20,7 @@ import mcppls.lsp.protocol; import mcppls.engine.payload; import mcppls.orchestrator.report; import mcppls.orchestrator.client; +import mcppls.orchestrator.completion; import mcppls.orchestrator.routing; import mcppls.orchestrator.workspace; @@ -370,6 +371,9 @@ class Session { { "capabilities", capabilities.empty() ? orchestrator::merge_capabilities(Json::object()) : capabilities }, { "serverInfo", Json { { "name", "mcppls" }, { "version", std::string { base::VERSION } } } }, }; + // F9 (D4 layer 4): a space opens the module list after `import`, for a client that drops the + // other spaces itself (VS Code's middleware) or asked for it (completion.triggerOnSpace). + if (orchestrator::completion::space_trigger_wanted(clientParams_)) orchestrator::completion::add_space_trigger(result["capabilities"]); reply_(clientInitializeId_, std::move(result)); for (auto& root : roots_) root->allow_status_notifications(); } diff --git a/tests/test_completion.cpp b/tests/test_completion.cpp new file mode 100644 index 0000000..9c3a0d8 --- /dev/null +++ b/tests/test_completion.cpp @@ -0,0 +1,208 @@ +// Completion routing (fix plan 2026-09-26 F9, F15, D4, D5): the space-trigger gate and who is told +// about the space, the module-syntax keywords and where each one fits, their merge with the core +// engine's answer, and the module names import completion keeps between edits. +import std; +import nlohmann.json; +import mcppls.testing; +import mcppls.base.text; +import mcppls.project.scan; +import mcppls.engine.native.index; +import mcppls.engine.native.keywords; +import mcppls.orchestrator.completion; +import mcppls.orchestrator.routing; + +using Json = nlohmann::json; +using mcppls::base::Position; +namespace completion = mcppls::orchestrator::completion; +namespace idx = mcppls::index; + +namespace { + +std::vector labels_of(const Json& result) { + std::vector labels; + const Json items = result.is_object() ? result.value("items", Json::array()) : result; + if (!items.is_array()) return labels; + for (const auto& item : items) labels.push_back(item.value("label", std::string {})); + return labels; +} + +bool has(const std::vector& labels, std::string_view label) { return std::ranges::find(labels, label) != labels.end(); } + +// The keywords offered with the cursor at the end of `text`. +std::vector keywords_at_end(std::string_view text, bool scanned = true) { + const auto lines = mcppls::base::split_lines(text); + const int line { static_cast(lines.empty() ? 0 : lines.size() - 1) }; + const int character { static_cast(lines.empty() ? 0 : lines.back().size()) }; + const auto scan = mcppls::project::scan_source(text); + return labels_of(idx::keyword_completion(text, Position { line, character }, scanned ? &scan : nullptr)); +} + +} // namespace + +int main() { + using namespace mcppls::testing; + + "the space gate passes an import directive's keyword and one blank, and nothing else"_test = [] { + for (const std::string_view passes : { "import ", "export import ", " import ", "\timport\t", "export import ", "\texport\timport " }) { + expect(completion::is_import_line_prefix(passes)) << passes; + } + for (const std::string_view fails : { "import ", "import", "import a", "import a ", "int x = ", "", " ", "exportimport ", + "export ", "export module ", "module ", "importer ", "// import ", "x import " }) { + expect(!completion::is_import_line_prefix(fails)) << fails; + } + }; + + "the line before the cursor, by LSP position"_test = [] { + const std::string text { "int a;\nexport import \nb" }; + expect(completion::line_prefix(text, Position { 1, 14 }).value_or("?") == "export import "); + expect(completion::line_prefix(text, Position { 1, 7 }).value_or("?") == "export "); + expect(completion::line_prefix(text, Position { 0, 0 }).value_or("?").empty()); + expect(completion::line_prefix(text, Position { 2, 1 }).value_or("?") == "b"); + expect(!completion::line_prefix(text, Position { 9, 0 }).has_value()); + }; + + "a space-triggered request is told from every other"_test = [] { + expect(completion::is_space_trigger(Json::parse(R"({"context": {"triggerKind": 2, "triggerCharacter": " "}})"))); + expect(!completion::is_space_trigger(Json::parse(R"({"context": {"triggerKind": 2, "triggerCharacter": "."}})"))); + expect(!completion::is_space_trigger(Json::parse(R"({"context": {"triggerKind": 1}})"))); + expect(!completion::is_space_trigger(Json::parse(R"({"position": {"line": 0, "character": 0}})"))); + }; + + "the space is advertised to VS Code and its forks, or to a client that asks"_test = [] { + const auto client = [](std::string_view name, const Json& options = nullptr) { + Json params { { "clientInfo", Json { { "name", std::string { name } }, { "version", "1.100.0" } } } }; + if (!options.is_null()) params["initializationOptions"] = options; + return params; + }; + for (const std::string_view name : { "Visual Studio Code", "Visual Studio Code - Insiders", "Code - OSS", "VSCodium", "Cursor", "Windsurf", "Trae" }) { + expect(completion::vscode_like(client(name))) << name; + expect(completion::space_trigger_wanted(client(name))) << name; + } + for (const std::string_view name : { "Neovim", "Zed", "helix", "CLion", "mcppls-conformance", "" }) { + expect(!completion::space_trigger_wanted(client(name))) << name; + } + expect(!completion::space_trigger_wanted(Json::object())) << "no clientInfo"; + expect(completion::space_trigger_wanted(client("Neovim", Json::parse(R"({"completion": {"triggerOnSpace": true}})")))); + expect(!completion::space_trigger_wanted(client("Visual Studio Code", Json::parse(R"({"completion": {"triggerOnSpace": false}})")))); + expect(completion::space_trigger_wanted(client("Visual Studio Code", Json::parse(R"({"completion": {}})")))); + + Json clangd = Json::parse(R"({"completionProvider": {"triggerCharacters": [".", "<", ">", ":", "\"", "/", "*"], "resolveProvider": false}})"); + completion::add_space_trigger(clangd); + completion::add_space_trigger(clangd); + const Json& triggers = clangd["completionProvider"]["triggerCharacters"]; + expect(triggers.size() == 8u && triggers.back() == " ") << triggers.dump(); + expect(clangd["completionProvider"]["resolveProvider"] == false); + Json own = mcppls::orchestrator::merge_capabilities(Json::object()); + completion::add_space_trigger(own); + expect(own["completionProvider"]["triggerCharacters"] == Json::parse(R"([".", ":", " "])")); + }; + + "module keywords where a declaration can begin, filtered by what was typed"_test = [] { + // A plain file: import only; the module declarations while it has none; module; while nothing precedes. + const auto first = keywords_at_end("i"); + expect(first == std::vector { "import" }) << first.size(); + const auto m = keywords_at_end("m"); + expect(has(m, "module;") && has(m, "module") && !has(m, "module :private;")); + const auto e = keywords_at_end("e"); + expect(has(e, "export module") && !has(e, "export import")) << "no interface unit, nothing to export"; + expect(keywords_at_end("#include \nm") == std::vector { "module" }) << "module; comes first or not at all"; + expect(has(keywords_at_end("// a comment\n/* another */\nm"), "module;")); + // Everything, invoked on an empty line of an empty file. + expect(keywords_at_end("").size() == 4u); // import, module;, export module, module + + // An interface unit: export import, and the private fragment once. + const std::string interface { "export module hello;\nimport std;\n" }; + const auto afterDeclaration = keywords_at_end(interface + "e"); + expect(afterDeclaration == std::vector { "export import" }) << "the declaration is there already"; + expect(keywords_at_end(interface + "export i") == std::vector { "export import" }); + expect(keywords_at_end(interface + "export i") == std::vector { "export import" }); + expect(keywords_at_end(interface + "export") == std::vector { "export import" }); + expect(keywords_at_end(interface + "mod") == std::vector { "module :private;" }); + expect(keywords_at_end(interface + "module :private;\nint x;\nmod").empty()); + expect(keywords_at_end("export module hello:part;\nmod").empty()) << "a partition has no private fragment"; + // An implementation unit exports nothing. + expect(keywords_at_end("module hello;\ne").empty()); + expect(keywords_at_end("module hello;\ni") == std::vector { "import" }); + }; + + "no keywords inside braces, comments, literals, or the middle of a word"_test = [] { + const Json none = idx::keyword_completion("int f() {\n i", Position { 1, 3 }, nullptr); + expect(none.is_null()) << none.dump(); + expect(idx::keyword_completion("namespace n {\ni", Position { 1, 1 }, nullptr).is_null()); + expect(!idx::keyword_completion("namespace n {\n}\ni", Position { 2, 1 }, nullptr).is_null()); + expect(idx::keyword_completion("/* open\ni", Position { 1, 1 }, nullptr).is_null()); + expect(!idx::keyword_completion("const char* s = \"{\";\nchar c = '{';\n// {\ni", Position { 3, 1 }, nullptr).is_null()); + expect(!idx::keyword_completion("auto r = R\"x({ \")x\";\ni", Position { 1, 1 }, nullptr).is_null()); + expect(!idx::keyword_completion("#define OPEN {\ni", Position { 1, 1 }, nullptr).is_null()) << "a macro's brace is not the file's"; + expect(!idx::keyword_completion("int n = 1'000;\ni", Position { 1, 1 }, nullptr).is_null()); + expect(idx::keyword_completion("import", Position { 0, 3 }, nullptr).is_null()) << "the cursor inside a word"; + expect(idx::keyword_completion("x = i", Position { 0, 5 }, nullptr).is_null()); + expect(idx::keyword_completion("import ", Position { 0, 7 }, nullptr).is_null()) << "the module names' place, not the keywords'"; + expect(idx::keyword_completion("export module ", Position { 0, 14 }, nullptr).is_null()) << "D5: no name is suggested"; + expect(idx::keyword_completion("q", Position { 0, 1 }, nullptr).is_null()) << "no keyword starts with it"; + }; + + "a keyword replaces what was typed, and opens the module list where the client can"_test = [] { + const std::string text { "export module a;\n export im" }; + const auto scan = mcppls::project::scan_source(text); + const Json plain = idx::keyword_completion(text, Position { 1, 11 }, &scan); + expect(fatal(plain.is_array() && plain.size() == 1u)) << plain.dump(); + const Json& item = plain[0]; + expect(item["label"] == "export import" && item["kind"] == 14 && item["filterText"] == "export import"); + expect(item["textEdit"]["newText"] == "export import "); + expect(item["textEdit"]["range"]["start"] == Json::parse(R"({"line": 1, "character": 2})")); + expect(item["textEdit"]["range"]["end"] == Json::parse(R"({"line": 1, "character": 11})")); + expect(!item.contains("command")); + const Json vscode = idx::keyword_completion(text, Position { 1, 11 }, &scan, idx::KeywordOptions { .suggestModulesAfterImport = true }); + expect(vscode[0]["command"]["command"] == "editor.action.triggerSuggest"); + const Json module = idx::keyword_completion("m", Position { 0, 1 }, nullptr, idx::KeywordOptions { .suggestModulesAfterImport = true }); + for (const auto& entry : module) expect(!entry.contains("command")) << entry.dump(); + }; + + "keywords merge with the core engine's answer without duplicates"_test = [] { + const Json keywords = idx::keyword_completion("i", Position { 0, 1 }, nullptr); + const Json list = Json::parse(R"({"isIncomplete": true, "items": [{"label": " int"}, {"label": "import"}]})"); + const Json merged = completion::merge(list, keywords); + expect(merged["isIncomplete"] == true); + expect(labels_of(merged) == std::vector { " int", "import" }) << "clangd's own `import` item is kept, not doubled"; + const Json array = completion::merge(Json::parse(R"([{"label": "if"}])"), keywords); + expect(array["isIncomplete"] == false && labels_of(array) == std::vector { "if", "import" }); + const Json withDefaults = completion::merge(Json::parse(R"({"isIncomplete": false, "itemDefaults": {"insertTextFormat": 2}, "items": []})"), keywords); + expect(withDefaults.contains("itemDefaults") && labels_of(withDefaults) == std::vector { "import" }); + expect(completion::merge(nullptr, Json::array()).is_null()); + expect(completion::merge(list, Json(nullptr)) == list); + const Json alone = completion::keywords_only(keywords); + expect(alone["isIncomplete"] == true && labels_of(alone) == std::vector { "import" }); + expect(completion::empty_list() == Json::parse(R"({"isIncomplete": false, "items": []})")); + }; + + "module names for completion are kept until a declaration changes"_test = [] { + idx::ModuleIndex index; + index.update("/p/a.cppm", "export module a;\n"); + index.update("/p/b.cppm", "export module b;\nexport import a;\n"); + index.update("/p/main.cpp", "import a;\n"); + const auto names = [&] { return labels_of(index.completion("/p/x.cpp", "import ", Position { 0, 7 })); }; + expect(names() == std::vector { "a", "b" }); + const auto generation = index.structure_generation(); + // Edits that leave every declaration as it was: the same list, not rebuilt. + index.update("/p/main.cpp", "import a;\nimport b;\nint main() {}\n"); + index.update("/p/b.cppm", "export module b;\nexport import a;\nexport int f();\n"); + index.update("/p/other.cpp", "int g();\n"); + expect(index.structure_generation() == generation); + expect(names() == std::vector { "a", "b" }); + // A new module, a role that changes, a unit removed: rebuilt. + index.update("/p/c.cppm", "export module c;\n"); + expect(index.structure_generation() != generation); + expect(names() == std::vector { "a", "b", "c" }); + const auto withC = index.structure_generation(); + index.update("/p/c.cppm", "module c;\n"); + expect(index.structure_generation() != withC); + expect(names() == std::vector { "a", "b" }) << "an implementation unit provides nothing to import"; + index.remove("/p/b.cppm"); + expect(names() == std::vector { "a" }); + const Json detail = index.completion("/p/x.cpp", "import ", Position { 0, 7 }); + expect(detail["items"][0]["detail"] == "primary module interface unit") << detail.dump(); + }; + + return report(); +} From 6a1aa5eaf23e14f5d2269405b46f3364feecc7bd Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 26 Sep 2026 07:26:15 +0800 Subject: [PATCH 05/31] test(conformance): fixture completion-keywords, and the runner sends a trigger character, a client name and number bounds completion-keywords drives the server as VS Code (the scenario's new "client-info"): the space is advertised; a space after `import ` or `export import ` gets the module list, one after `int x = `, `import ` or `export module ` gets nothing at once; the module keywords appear where each fits and not in a function body, and on their own for a file clangd cannot answer for yet; the report counts the space trigger's cost. engine-none checks the keywords and the gate with no core engine at all. Against 0.0.4 the fixture fails nine checks. The runner's completion-contains takes "trigger" (a request sent as typing that character asked for it), several labels, "exact" and "absent"; completion-empty and capabilities are new kinds; an expectation can say "at-least". The fixture runs in CI on every platform. --- .github/workflows/ci.yml | 8 +- conformance/README.md | 13 +- .../completion-keywords/scenario.json | 226 ++++++++++++++++++ .../completion-keywords/src/greet/detail.cppm | 5 + .../completion-keywords/src/greet/greet.cppm | 7 + .../fixtures/completion-keywords/src/main.cpp | 7 + .../fixtures/engine-none/scenario.json | 4 + src/bin/conformance.cpp | 63 ++++- 8 files changed, 317 insertions(+), 16 deletions(-) create mode 100644 conformance/fixtures/completion-keywords/scenario.json create mode 100644 conformance/fixtures/completion-keywords/src/greet/detail.cppm create mode 100644 conformance/fixtures/completion-keywords/src/greet/greet.cppm create mode 100644 conformance/fixtures/completion-keywords/src/main.cpp diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7c47220..d302d4b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -282,21 +282,21 @@ jobs: part: 2 of 2 os: ubuntu-24.04 extras: true - fixtures: inferred engine-none module-faults untrusted mcpp-gcc mcpp-gcc@plain mcpp-emit-package-std mcpp-emit-broken mcpp-emit-unavailable mcpp-emit-hang mcpp-emit-needs-download mcpp-emit-watch mcpp-emit-watch@polling cmake-clang cmake-clang-bdb watch-polling payload-corrupt clangd-cannot-load generated-module generated-module@vscode generated-module-old-mcpp generated-module-old-mcpp@neovim module-faults@zed failure-at-base failure-at-base@neovim + fixtures: inferred engine-none module-faults untrusted mcpp-gcc mcpp-gcc@plain mcpp-emit-package-std mcpp-emit-broken mcpp-emit-unavailable mcpp-emit-hang mcpp-emit-needs-download mcpp-emit-watch mcpp-emit-watch@polling cmake-clang cmake-clang-bdb watch-polling payload-corrupt clangd-cannot-load generated-module generated-module@vscode generated-module-old-mcpp generated-module-old-mcpp@neovim module-faults@zed failure-at-base failure-at-base@neovim completion-keywords # generated-module{,-old-mcpp,-negotiated}'s mcpp-mock.json bakes in a POSIX driver path # (${env:HOME}/.mcpp/registry/..., no {exe}); it resolves the same way here as on Linux, so # these run on macOS but are left off win32-x64 below rather than fixed unverified. - platform: darwin-arm64 os: macos-14 extras: true - fixtures: inferred engine-none module-faults untrusted mcpp-llvm mcpp-split mcpp-all-cppm verify-changes mcpp-watch multi-root failure-at-base clangd-cannot-load failure-at-base@zed generated-module generated-module-old-mcpp generated-module-negotiated typing-import typing-import-spin workaround-canaries + fixtures: inferred engine-none module-faults untrusted mcpp-llvm mcpp-split mcpp-all-cppm verify-changes mcpp-watch multi-root failure-at-base clangd-cannot-load failure-at-base@zed generated-module generated-module-old-mcpp generated-module-negotiated typing-import typing-import-spin workaround-canaries completion-keywords # No mcpp on the arm64 runner (its tools are the cross-built ones), so the fixtures that # need no build tool and no compiler of their own: the semantic kit, clangd and the server # on aarch64, with module faults, a corrupt payload and polling included. - platform: linux-arm64 os: ubuntu-24.04-arm cross-tools: true - fixtures: inferred engine-none module-faults untrusted payload-corrupt clangd-cannot-load failure-at-base watch-polling typing-import typing-import-spin workaround-canaries + fixtures: inferred engine-none module-faults untrusted payload-corrupt clangd-cannot-load failure-at-base watch-polling typing-import typing-import-spin workaround-canaries completion-keywords - platform: win32-x64 part: 1 of 2 os: windows-2022 @@ -305,7 +305,7 @@ jobs: part: 2 of 2 os: windows-2022 extras: true - fixtures: inferred engine-none module-faults mcpp-emit-hang inferred-msvc untrusted mingw cmake-msvc cmake-msvc-bdb cmake-clangxx-msvc cmake-clang-cl mcpp-msvc mcpp-watch failure-at-base clangd-cannot-load + fixtures: inferred engine-none module-faults mcpp-emit-hang inferred-msvc untrusted mingw cmake-msvc cmake-msvc-bdb cmake-clangxx-msvc cmake-clang-cl mcpp-msvc mcpp-watch failure-at-base clangd-cannot-load completion-keywords defaults: run: shell: bash diff --git a/conformance/README.md b/conformance/README.md index 5651073..c38023e 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -23,7 +23,7 @@ checks fail at once with that reason instead of each waiting out its timeout. | Fixture | What it covers | |---|---| | `inferred` | Loose module sources, no build system and no compiler: the semantic kit provides libc++ semantics (design 13.5) | -| `engine-none` | The `inferred` project with `--engine none` (overall design 5.6): no core engine, so mcppls's own engine alone answers module navigation, hover, outline, import completion and module diagnostics; the status names `none` as the core engine and lists `mcppls` in `engines` (S3-4-5, S3-4-6). `inferred` checks the same fields with clangd, and starts a second server on the same workspace and cache, which must report the `shared-workspace` notice (design 6.3). Its S5 checks: module descriptions and interface summaries, exported symbols found without clangd, `unavailable` for references, and a file written to disk read by the next query | +| `engine-none` | The `inferred` project with `--engine none` (overall design 5.6): no core engine, so mcppls's own engine alone answers module navigation, hover, outline, import completion, module-syntax keywords and module diagnostics, and a space off an import line is answered with nothing (S3-6.2); the status names `none` as the core engine and lists `mcppls` in `engines` (S3-4-5, S3-4-6). `inferred` checks the same fields with clangd, and starts a second server on the same workspace and cache, which must report the `shared-workspace` notice (design 6.3). Its S5 checks: module descriptions and interface summaries, exported symbols found without clangd, `unavailable` for references, and a file written to disk read by the next query | | `verify-changes` | `mcpp-split`'s project in a git repository, for S5's verification after an edit: snippets checked in place (passing, failing, and leaving no unsaved content behind), a partition interface renamed on disk that breaks the implementation unit using it, the working tree's changes from git, and the restored file passing again once its importer is built again | | `untrusted` | An mcpp package in an untrusted workspace: nothing is executed, the kit answers, the status is `degraded` with the reason | | `mcpp-gcc` | mcpp with GCC 16, described by mcpp's own `emit build-database` (mcpp 2026.9.15.1): level 3 once the S1 library has structured mcpp's level 2 document, GCC arguments translated for clangd (P1), libstdc++'s `std` from its manifest, a test that imports the package's module across sets, and the workspace unchanged | @@ -66,6 +66,7 @@ checks fail at once with that reason instead of each waiting out its timeout. | `clangd-cannot-load` | 0.0.3 plan B1: its `prepare` step puts a stand-in clangd in the workspace (mcppls-mock-mcpp with an `unavailable` config) that writes a loader's message, a `GLIBCXX` version not found, to standard error and exits 1; `--clangd` points the server at it. `initialize` must be answered within 20 s (it used to wait for good), the status must reach `error` with issue `engine-incompatible`, and mcppls's own module features must work | | `typing-import` | Import-hang plan §8: `import hello.greet;` in `main.cpp` and `export module hello.greet;` in its interface typed one key at a time, through `import hello.` and `export module hello.`, which clangd 23.1 never finishes building (WA-CLANGD-001); again with every step saved, as autosave does. Every request is answered within 5 s, the status never turns `degraded`, hover works right after, and nothing restarts clangd or is set aside | | `typing-import-spin` | The same typing with WA-CLANGD-001 turned off (`--disable-workaround`), so clangd really spins (Linux, macOS; on Windows the same text crashes it instead): a spin is found within its 20 s budget (event `engine-spin`), the file set aside with that text remembered and clangd restarted, a crash is restarted as before (`engine-exit`), and either way features come back while typing goes on (import-hang plan §4). If a clangd update removes the defect, its T2 check fails as well | +| `completion-keywords` | Fix plan 2026-09-26 F9 and F15, as VS Code (`client-info`): the space is a completion trigger character; a space after `import ` or `export import ` opens the module list from mcppls's own index, and one typed anywhere else, or after `import ` or `export module `, is answered at once with nothing. The module-syntax keywords are offered where each can begin a declaration and not inside a function body, merged with clangd's answer, and on their own within 1.5 s for a file clangd cannot answer for yet (a new file waiting for clangd to read a database that has it); the report counts what the space trigger cost (S3-6.2-1 to S3-6.2-5) | | `workaround-canaries` | Import-hang plan §9: one `clangd-check` per registered workaround with a canary, run against the payload's clangd. A failure here means a clangd update fixed that defect and the workaround it names can be removed | | `payload-corrupt` | Its `prepare` step copies the payload the runner was given and truncates clangd in the copy (usable plan W9.4); `server-arguments` then points `--payload` at that broken copy, and status must reach `error` with issue `payload-corrupt` | | `multi-root` | Two workspace folders (usable plan W9.1): an `inferred` root and an mcpp-built `mcpp-llvm` root (level 3, from mcpp's own build database), each getting its own project model and clangd, each `cxxModules/status` telling them apart by `project.root` | @@ -125,6 +126,8 @@ scenario names, are relative to the fixture's own root, never to a specific work `initializationOptions` (over whatever `--client` profile set), so a fixture can ask for something `--client` does not, such as `{"semanticTokens": {"moduleType": true}}` (design doc 2026-09-25 K/§7). +`"client-info"` on the scenario is the `clientInfo` the runner sends in `initialize` (none otherwise): +what a server tells VS Code can differ from what it tells other clients (fix plan 2026-09-26 F9). `"initialize-within": SECONDS` on the scenario fails the run when `initialize` is answered later than that (the runner itself waits up to 120 s): a server that answers eventually is not enough where the point is that it answers at once (`clangd-cannot-load`). @@ -160,7 +163,9 @@ always has been. | `definition` / `declaration` | a location ends with `expect` | | `definition-any` | there is at least one location | | `hover-contains` | the hover text contains `expect`, or any one of them when `expect` is a list | -| `completion-contains` | a completion label starts with `expect`; `insert: [line, text]` adds a line first, `edit` changes another open buffer without saving it | +| `completion-contains` | a completion label starts with `expect` (with a list, every one does; with `"exact": true`, a label is it), and none is one of `"absent"`; `insert: [line, text]` adds a line first, `edit` changes another open buffer without saving it; `"trigger"` sends the request as typing that character asked for it (`context.triggerKind` 2) | +| `completion-empty` | the completion (with `"trigger"` as above) has no items, within `"within-ms"` when given (fix plan 2026-09-26 F9: a space off an import line) | +| `capabilities` | the server capabilities `initialize` answered with meet `"expect"` (expectations as for `report`) | | `references-span` | the references include every path in `expect` | | `document-symbol-contains` | the outline has a top-level symbol named `expect` | | `semantic-tokens` | `textDocument/semanticTokens/full` (or `/range`, with `"range"`) for `"file"` (optionally with an unsaved `"text"`), decoded with the legend `initialize` gave, has every entry of `"expect"` (`{"line", "text", "type", "modifiers"?}`; `"modifiers"` is a list, and optional) among its tokens (design doc 2026-09-25 K/§7) | @@ -178,8 +183,8 @@ always has been. An expectation of `mcp`, `cli` and `report` names a JSON pointer in `"path"`, where a `*` segment stands for every element of an array, and one of `"equals"` (a value the pointer names equals it), `"contains"` (a string -contains it, or an array has an element that includes all its members), `"min-items"`, `"max-items"`, `"exists"` or -`"absent"`; it holds when any value the pointer names satisfies it. `"each-contains"` is the one that +contains it, or an array has an element that includes all its members), `"min-items"`, `"max-items"`, `"at-least"` +(a number at least it), `"exists"` or `"absent"`; it holds when any value the pointer names satisfies it. `"each-contains"` is the one that every value the pointer names must satisfy instead (each is a string containing it), and it holds when the pointer names none. diff --git a/conformance/fixtures/completion-keywords/scenario.json b/conformance/fixtures/completion-keywords/scenario.json new file mode 100644 index 0000000..8fcb1e1 --- /dev/null +++ b/conformance/fixtures/completion-keywords/scenario.json @@ -0,0 +1,226 @@ +{ + "name": "completion-keywords", + "description": "Fix plan 2026-09-26 F9 and F15 (decisions D4, D5), as VS Code: a space is a trigger character; a space typed after `import` opens the module list from mcppls's own index, and one typed anywhere else is answered at once with nothing, never reaching clangd. The module-syntax keywords (`import`, `export import`, `module;`, `export module`, `module`, `module :private;`) come from mcppls where each can begin a declaration, merged with clangd's answer, and on their own when clangd cannot answer for the file yet: a file just opened waits for clangd to read a database that has it, and its keywords do not.", + "server-arguments": [ + "--no-discover" + ], + "client-info": { + "name": "Visual Studio Code", + "version": "1.105.0" + }, + "checks": [ + { + "id": "S1", + "kind": "status", + "source": "inferred", + "state": "ready", + "engine-name": "clangd" + }, + { + "id": "C0-space-is-a-trigger", + "kind": "capabilities", + "expect": [ + { + "path": "/completionProvider/triggerCharacters", + "contains": " " + }, + { + "path": "/completionProvider/triggerCharacters", + "contains": "." + } + ] + }, + { + "id": "T0-clangd-answers", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 90 + }, + { + "id": "K5-not-in-a-body", + "kind": "completion-contains", + "file": "src/main.cpp", + "insert": [ + 4, + " i" + ], + "at": [ + 4, + 5 + ], + "expect": [ + "int" + ], + "absent": [ + "import", + "export module", + "module;" + ] + }, + { + "id": "K1-import", + "kind": "completion-contains", + "file": "src/main.cpp", + "insert": [ + 2, + "i" + ], + "at": [ + 2, + 1 + ], + "expect": [ + "import", + "int" + ], + "absent": [ + "export import", + "module;" + ], + "exact": true + }, + { + "id": "K2-module-declarations", + "kind": "completion-contains", + "file": "src/scratch.cppm", + "text": "// a new unit\ne", + "at": [ + 1, + 1 + ], + "expect": [ + "export module" + ], + "absent": [ + "export import" + ], + "exact": true + }, + { + "id": "K3-export-import", + "kind": "completion-contains", + "file": "src/iface.cppm", + "text": "export module scratch.iface;\nimport std;\nexport i", + "at": [ + 2, + 8 + ], + "expect": [ + "export import" + ], + "absent": [ + "export module" + ], + "exact": true + }, + { + "id": "K4-private-fragment", + "kind": "completion-contains", + "file": "src/iface.cppm", + "text": "export module scratch.iface;\nimport std;\nexport int f();\nmod", + "at": [ + 3, + 3 + ], + "expect": [ + "module :private;" + ], + "absent": [ + "module;", + "export module" + ], + "exact": true + }, + { + "id": "P1-space-elsewhere", + "kind": "completion-empty", + "file": "src/space.cpp", + "text": "int x = ", + "at": [ + 0, + 8 + ], + "trigger": " ", + "within-ms": 2000 + }, + { + "id": "P2-space-after-import", + "kind": "completion-contains", + "file": "src/space.cpp", + "text": "import std;\nimport ", + "at": [ + 1, + 7 + ], + "trigger": " ", + "expect": [ + "hello.greet" + ] + }, + { + "id": "P3-space-after-export-import", + "kind": "completion-contains", + "file": "src/space.cppm", + "text": "export module space;\nexport import ", + "at": [ + 1, + 14 + ], + "trigger": " ", + "expect": [ + "hello.greet" + ] + }, + { + "id": "P4-two-spaces", + "kind": "completion-empty", + "file": "src/space.cpp", + "text": "import ", + "at": [ + 0, + 8 + ], + "trigger": " ", + "within-ms": 2000 + }, + { + "id": "P5-export-module-names-nothing", + "kind": "completion-empty", + "file": "src/space.cppm", + "text": "export module ", + "at": [ + 0, + 14 + ], + "trigger": " ", + "within-ms": 2000 + }, + { + "id": "R1-costs-counted", + "kind": "report", + "expect": [ + { + "path": "/roots/0/completion/spaceTrigger/count", + "at-least": 5 + }, + { + "path": "/roots/0/completion/spaceTrigger/passed", + "at-least": 2 + }, + { + "path": "/roots/0/completion/spaceTrigger/maxMicros", + "exists": true + }, + { + "path": "/roots/0/completion/keywordsWithoutEngine", + "at-least": 1 + } + ] + } + ] +} diff --git a/conformance/fixtures/completion-keywords/src/greet/detail.cppm b/conformance/fixtures/completion-keywords/src/greet/detail.cppm new file mode 100644 index 0000000..56cb780 --- /dev/null +++ b/conformance/fixtures/completion-keywords/src/greet/detail.cppm @@ -0,0 +1,5 @@ +export module hello.greet:detail; +import std; +export namespace hello::detail { + std::string prefix() { return "Hello, "; } +} diff --git a/conformance/fixtures/completion-keywords/src/greet/greet.cppm b/conformance/fixtures/completion-keywords/src/greet/greet.cppm new file mode 100644 index 0000000..21a6536 --- /dev/null +++ b/conformance/fixtures/completion-keywords/src/greet/greet.cppm @@ -0,0 +1,7 @@ +export module hello.greet; +export import :detail; +import std; + +export namespace hello { + std::string greet(std::string_view who) { return detail::prefix() + std::string(who); } +} diff --git a/conformance/fixtures/completion-keywords/src/main.cpp b/conformance/fixtures/completion-keywords/src/main.cpp new file mode 100644 index 0000000..71169db --- /dev/null +++ b/conformance/fixtures/completion-keywords/src/main.cpp @@ -0,0 +1,7 @@ +import std; +import hello.greet; + +int main(int argc, char* argv[]) { + std::println("{}", hello::greet("mcpp")); + return 0; +} diff --git a/conformance/fixtures/engine-none/scenario.json b/conformance/fixtures/engine-none/scenario.json index bdccff0..f6312f2 100644 --- a/conformance/fixtures/engine-none/scenario.json +++ b/conformance/fixtures/engine-none/scenario.json @@ -9,6 +9,10 @@ { "id": "M2-hover", "kind": "hover-contains", "file": "src/main.cpp", "at": [1, 9], "expect": "module hello.greet" }, { "id": "M3-outline", "kind": "document-symbol-contains", "file": "src/greet/greet.cppm", "expect": "hello.greet" }, { "id": "M4-import-completion", "kind": "completion-contains", "file": "src/scratch.cpp", "text": "import hel", "at": [0, 10], "expect": "hello.greet" }, + { "id": "M4-keywords", "kind": "completion-contains", "file": "src/keywords.cpp", "text": "i", "at": [0, 1], "expect": ["import"], "exact": true }, + { "id": "M4-export-import", "kind": "completion-contains", "file": "src/keywords.cppm", "text": "export module keywords;\nexport i", "at": [1, 8], "expect": ["export import"], "exact": true }, + { "id": "M4-space-elsewhere", "kind": "completion-empty", "file": "src/keywords.cpp", "text": "int x = ", "at": [0, 8], "trigger": " ", "within-ms": 2000 }, + { "id": "M4-space-after-import", "kind": "completion-contains", "file": "src/keywords.cpp", "text": "import ", "at": [0, 7], "trigger": " ", "expect": "hello.greet" }, { "id": "M5-unresolved", "kind": "diagnostic-code", "file": "src/unresolved.cpp", "text": "import missing.module;\nint f();\n", "expect": "unresolved-module" }, { "id": "M6-graph", "kind": "module-graph-contains", "expect": "hello.greet:detail" }, { "id": "Q4-module", "kind": "mcp", "tool": "cxx_module", "arguments": { "name": "hello.greet" }, diff --git a/src/bin/conformance.cpp b/src/bin/conformance.cpp index f628e1b..64e1ed8 100644 --- a/src/bin/conformance.cpp +++ b/src/bin/conformance.cpp @@ -614,7 +614,7 @@ bool includes(const Json& candidate, const Json& expected) { } // A fixture's expectations of a JSON result (conformance/README.md, S5 checks): each names a pointer and -// one of equals, contains, min-items, max-items, exists or absent, and holds when any value the pointer names satisfies it -- +// one of equals, contains, min-items, max-items, at-least (a number), exists or absent, and holds when any value the pointer names satisfies it -- // except each-contains, which every value the pointer names must satisfy (and holds when it names none). std::pair expectations_hold(const Json& value, const Json& expectations) { for (const auto& expectation : expectations) { @@ -643,6 +643,9 @@ std::pair expectations_hold(const Json& value, const Json& ex } else if (expectation.contains("max-items")) { const std::size_t wanted { expectation.value("max-items", std::size_t { 0 }) }; held = std::ranges::any_of(matches, [&](const Json* match) { return (match->is_array() || match->is_object()) && match->size() <= wanted; }); + } else if (expectation.contains("at-least")) { + const double wanted { expectation.value("at-least", 0.0) }; + held = std::ranges::any_of(matches, [&](const Json* match) { return match->is_number() && match->get() >= wanted; }); } if (!held) { std::string found { matches.empty() ? std::string { "nothing" } : lsp::dump(*matches.front()) }; @@ -933,6 +936,7 @@ class Scenario { std::unique_ptr mcpDaemon_; // the first mcp check "via": "daemon" std::string mcpFailure_; Json semanticTokensLegend_ = Json::object(); // initialize's capabilities.semanticTokensProvider.legend + Json capabilities_ = Json::object(); // initialize's capabilities, for "capabilities" checks McpClient* mcp_client(bool daemon) { auto& kept = daemon ? mcpDaemon_ : mcp_; @@ -956,13 +960,24 @@ class Scenario { Scenario(Client& client, const Options& options, std::vector serverArguments, std::string workspace, std::chrono::seconds timeout, std::map prepared, std::string cacheDirectory, bool expectWarm, - std::map> moduleFilesBefore, Json semanticTokensLegend = Json::object()) + std::map> moduleFilesBefore, Json semanticTokensLegend = Json::object(), + Json capabilities = Json::object()) : client_ { client }, options_ { options }, serverArguments_ { std::move(serverArguments) }, workspace_ { std::move(workspace) }, timeout_ { timeout }, prepared_ { std::move(prepared) }, cacheDirectory_ { std::move(cacheDirectory) }, expectWarm_ { expectWarm }, moduleFilesBefore_ { std::move(moduleFilesBefore) }, - semanticTokensLegend_ ( std::move(semanticTokensLegend) ) {} + semanticTokensLegend_ ( std::move(semanticTokensLegend) ), capabilities_ ( std::move(capabilities) ) {} std::string uri(std::string_view relative) const { return base::path_to_uri(base::join_path(workspace_, relative)); } + // A completion request at the check's "at"; "trigger" sends it as typing that character asked for it + // (CompletionTriggerKind.TriggerCharacter), the way an editor does for a trigger character. + Json completion_params(const Json& check, std::string_view file) const { + Json params { { "textDocument", Json { { "uri", uri(file) } } }, { "position", position(check.at("at")) } }; + if (const auto trigger = check.find("trigger"); trigger != check.end() && trigger->is_string()) { + params["context"] = Json { { "triggerKind", 2 }, { "triggerCharacter", trigger->get() } }; + } + return params; + } + std::string text_of(std::string_view relative) { if (auto it = open_.find(std::string { relative }); it != open_.end()) return it->second.first; return fs::read_file(base::join_path(workspace_, relative)).value_or(""); @@ -1490,17 +1505,46 @@ class Scenario { // Touch the importing buffer so it is rebuilt against the edited module. change(file, text_of(file) + " "); } - const std::string expected { check.value("expect", std::string {}) }; - auto [ok, result] = retry("textDocument/completion", - [&] { return Json { { "textDocument", Json { { "uri", uri(file) } } }, { "position", position(check.at("at")) } }; }, + // "expect": a label prefix, or several that must all be there ("exact": whole labels); "absent": labels that must not be. + const bool exact { check.value("exact", false) }; + std::vector expected; + if (const auto wanted = check.find("expect"); wanted != check.end() && wanted->is_array()) { + for (const auto& one : *wanted) expected.push_back(one.get()); + } else { + expected.push_back(check.value("expect", std::string {})); + } + std::vector absent; + for (const auto& one : check.value("absent", Json::array())) absent.push_back(one.get()); + auto [ok, result] = retry("textDocument/completion", [&] { return completion_params(check, file); }, [&](const Json& value) { const auto labels = completion_labels(value); - return std::ranges::any_of(labels, [&](const std::string& label) { return label.starts_with(expected); }); + const auto present = [&](const std::string& prefix) { + return std::ranges::any_of(labels, [&](const std::string& label) { return exact ? label == prefix : label.starts_with(prefix); }); + }; + return std::ranges::all_of(expected, present) + && std::ranges::none_of(absent, [&](const std::string& label) { return std::ranges::find(labels, label) != labels.end(); }); }); auto labels = completion_labels(result); if (labels.size() > 12) labels.resize(12); return { ok, lsp::dump(labels) }; } + if (kind == "completion-empty") { + // F9 (fix plan 2026-09-26, D4): a completion answered with no items, within "within-ms" when given -- + // a space typed outside an import line is answered at once, without asking the core engine. + open(file); + const auto start = Clock::now(); + auto answer = client_.request("textDocument/completion", completion_params(check, file), timeout_); + const auto elapsed = std::chrono::duration_cast(Clock::now() - start).count(); + if (!answer) return { false, "no answer" }; + const bool empty { completion_labels(*answer).empty() }; + const bool inTime { !check.contains("within-ms") || elapsed <= check.value("within-ms", std::int64_t { 0 }) }; + return { empty && inTime, std::format("{} in {} ms", lsp::dump(*answer).substr(0, 120), elapsed) }; + } + if (kind == "capabilities") { + // The server capabilities initialize answered with, held to "expect" like a tool's result. + auto [held, detail] = expectations_hold(capabilities_, check.value("expect", Json::array())); + return { held, held ? lsp::dump(capabilities_.value("completionProvider", Json::object())).substr(0, 160) : detail }; + } if (kind == "references-span") { open(file); std::vector expected; @@ -1880,6 +1924,9 @@ int run(Options options) { Json initializeParams { { "processId", nullptr }, { "rootUri", base::path_to_uri(workspace) }, { "workspaceFolders", workspaceFolders }, { "capabilities", capabilities } }; if (!initializationOptions.empty()) initializeParams["initializationOptions"] = initializationOptions; + // A scenario's own "client-info": the client this runner says it is (fix plan 2026-09-26 F9: what a + // server tells VS Code differs from what it tells any other client). + if (const auto info = scenario.find("client-info"); info != scenario.end() && info->is_object()) initializeParams["clientInfo"] = *info; auto initialized = client.request("initialize", std::move(initializeParams), std::chrono::seconds { 120 }); if (!initialized || !initialized->is_object()) { say("FAIL initialize: no result"); @@ -1902,7 +1949,7 @@ int run(Options options) { ? (*initialized)["capabilities"]["semanticTokensProvider"]["legend"] : Json::object(); Scenario runner { client, options, serverArguments, workspace, options.timeout, std::move(prepared), cacheDirectory, options.expectWarm, - std::move(moduleFilesBefore), semanticTokensLegend }; + std::move(moduleFilesBefore), semanticTokensLegend, initialized->value("capabilities", Json::object()) }; int failures { advertised ? 0 : 1 }; // "initialize-within": seconds. The handshake is answered at all, and in time (0.0.3 plan B1). if (const auto within = scenario.find("initialize-within"); within != scenario.end() && within->is_number()) { From 2b3b8d43ccc643add69b2c8618c0854c1c944839 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 26 Sep 2026 07:27:01 +0800 Subject: [PATCH 06/31] feat(vscode): a space after import asks for the module list, and every other space is dropped before it is sent The extension's middleware answers a completion that a typed space asked for with nothing unless the line up to the cursor is `import ` or `export import ` -- the server's own gate, in TypeScript, tested against the server's cases -- so the space the server advertises as a trigger character costs nothing anywhere else. The setting mcppls.completion.triggerOnSpace (default true) is sent as initializationOptions.completion.triggerOnSpace; off, the server does not advertise the space. The settings pages and the Neovim README say how other editors ask for it. --- docs/30-settings.md | 1 + docs/zh-CN/30-settings.md | 1 + editors/nvim/README.md | 9 +++++++ editors/vscode/README.md | 1 + editors/vscode/package.json | 5 ++++ editors/vscode/src/completionGate.ts | 19 +++++++++++++ editors/vscode/src/extension.ts | 18 ++++++++++++- .../vscode/test/unit/completionGate.test.ts | 27 +++++++++++++++++++ 8 files changed, 80 insertions(+), 1 deletion(-) create mode 100644 editors/vscode/src/completionGate.ts create mode 100644 editors/vscode/test/unit/completionGate.test.ts diff --git a/docs/30-settings.md b/docs/30-settings.md index 8a470c9..e9fcda4 100644 --- a/docs/30-settings.md +++ b/docs/30-settings.md @@ -12,6 +12,7 @@ | `mcppls.ai.enabled` | `false` (default) | Whether the model-backed half of change review may be used. Off means the server makes no model calls | | `mcppls.detectConflicts` | `true` (default) | Offer once to turn off another C++ extension's language features in this workspace, and say so when one becomes active later | | `mcppls.semanticTokens.modules` | `true` (default) | Color `import`, `module`, `export` and module names from the server's semantic tokens. Off: only the grammar's colors | +| `mcppls.completion.triggerOnSpace` | `true` (default) | Show the module list as soon as a space is typed after `import` or `export import`. A space anywhere else never reaches the server. Other editors ask for the same with `initializationOptions.completion.triggerOnSpace` | | `mcppls.trace.server` | `off` (default), `messages`, `verbose` | Log the LSP traffic to the C++ Modules output channel (at Trace level); `verbose` adds the server's debug log (at Debug level). Set the channel's log level to see them | ## Commands diff --git a/docs/zh-CN/30-settings.md b/docs/zh-CN/30-settings.md index 6474691..a554f1e 100644 --- a/docs/zh-CN/30-settings.md +++ b/docs/zh-CN/30-settings.md @@ -14,6 +14,7 @@ | `mcppls.ai.enabled` | `false`（默认） | 是否启用变更审查里依赖模型的那部分。关闭时服务端不发起任何模型调用 | | `mcppls.detectConflicts` | `true`（默认） | 在此工作区中提议关闭另一个 C++ 扩展的语言功能（只提议一次），之后又有冲突扩展启用时会提示 | | `mcppls.semanticTokens.modules` | `true`（默认） | 用服务端的语义 token 给 `import`、`module`、`export` 和模块名上色。关闭后只用语法文件的颜色 | +| `mcppls.completion.triggerOnSpace` | `true`（默认） | 在 `import` 或 `export import` 后输入空格时立即弹出模块列表；其他位置的空格不会发给服务端。其他编辑器用 `initializationOptions.completion.triggerOnSpace` 开启同样的行为 | | `mcppls.trace.server` | `off`（默认）, `messages`, `verbose` | 把 LSP 通信记录到 C++ Modules 输出通道（Trace 级别）；`verbose` 还会打开服务端的 debug 日志（Debug 级别）。要看到它们，需把该输出通道的日志级别调到对应级别 | ## 命令 diff --git a/editors/nvim/README.md b/editors/nvim/README.md index 27368f0..aecebba 100644 --- a/editors/nvim/README.md +++ b/editors/nvim/README.md @@ -113,6 +113,15 @@ vim.api.nvim_set_hl(0, '@lsp.mod.partition', { italic = true }) -- only the par vim.api.nvim_set_hl(0, '@lsp.type.keyword', { link = 'Keyword' }) -- import / module / export ``` +## Completing `import` + +The server completes module names after `import ` and offers the module keywords (`import`, +`export import`, `module;`, `export module`, `module`, `module :private;`) where each can begin a +declaration. It does not make a space a trigger character for Neovim, since every space typed +would then ask for a completion; to have the module list open on the space after `import`, ask for +it with `init_options = { completion = { triggerOnSpace = true } }` — the server answers the other +spaces with nothing, at once. + ## Commands and statusline | Command | | diff --git a/editors/vscode/README.md b/editors/vscode/README.md index 1d6e182..2d0c96e 100644 --- a/editors/vscode/README.md +++ b/editors/vscode/README.md @@ -59,6 +59,7 @@ All settings are optional. | `mcppls.ai.enabled` | `false` | Show the review commands | | `mcppls.detectConflicts` | `true` | Offer once to turn off other C++ extensions' language features in the workspace, and notice again if one becomes active later | | `mcppls.semanticTokens.modules` | `true` | Module keywords and names from the server's semantic tokens; turn off to use only your own grammar or tree-sitter colors for module syntax | +| `mcppls.completion.triggerOnSpace` | `true` | Show the module list as soon as a space is typed after `import` or `export import`; a space anywhere else never reaches the server | | `mcppls.trace.server` | `off` | Trace the language server protocol in the log | | `mcppls.buildTool` | `offline` | How mcppls may run the project's build tool (mcpp, CMake) to learn how it is built: `offline` runs it without the network, offering to run it in a terminal when it needs a download; `online` lets it reach the network, with up to ten minutes; `off` never runs it, using only the cache or scanned sources | | `mcppls.toolEnvironment` | `auto` | Which environment build tools are started in: `auto` reads the login shell's environment once in the background on Linux and macOS (Windows always matches the editor); `editor` always uses the editor process's own environment | diff --git a/editors/vscode/package.json b/editors/vscode/package.json index 3dfda9c..3ce108c 100644 --- a/editors/vscode/package.json +++ b/editors/vscode/package.json @@ -221,6 +221,11 @@ "default": true, "description": "Module keywords and names from the server's semantic tokens. Turn this off to use only your own grammar or tree-sitter colors for module syntax." }, + "mcppls.completion.triggerOnSpace": { + "type": "boolean", + "default": true, + "markdownDescription": "Show the module list as soon as a space is typed after `import` or `export import`. A space anywhere else never asks the server for anything." + }, "mcppls.trace.server": { "type": "string", "enum": [ diff --git a/editors/vscode/src/completionGate.ts b/editors/vscode/src/completionGate.ts new file mode 100644 index 0000000..dab3246 --- /dev/null +++ b/editors/vscode/src/completionGate.ts @@ -0,0 +1,19 @@ +// Fix plan 2026-09-26 F9 (decision D4, layer 1): the server takes a space as a completion trigger +// character, for the module list after `import`. VS Code would then ask on every space typed +// anywhere; this is what drops those requests before anything is sent. The same rule as the +// server's own gate (src/orchestrator/completion.cppm, is_import_line_prefix), in plain TypeScript +// so it runs without VS Code in the unit tests. + +// `^\s*(export\s+)?import\s$`: an import directive's keyword and exactly one blank after it, with +// nothing typed after that. +const IMPORT_LINE_PREFIX = /^[ \t\v\f]*(?:export[ \t\v\f]+)?import[ \t\v\f]$/; + +export function isImportLinePrefix(prefix: string): boolean { + return IMPORT_LINE_PREFIX.test(prefix); +} + +// Whether a completion VS Code asks for because `triggerCharacter` was typed at `character` of a +// line should reach the server. Only a space is ever dropped. +export function sendTriggeredCompletion(triggerCharacter: string | undefined, lineText: string, character: number): boolean { + return triggerCharacter !== ' ' || isImportLinePrefix(lineText.slice(0, character)); +} diff --git a/editors/vscode/src/extension.ts b/editors/vscode/src/extension.ts index f2b52f3..426cd51 100644 --- a/editors/vscode/src/extension.ts +++ b/editors/vscode/src/extension.ts @@ -26,6 +26,7 @@ import { } from 'vscode-languageclient/node'; import { CommandLineToolsController, withInstallCommandFallback } from './commandLineTools'; import { registerCommands, reloadBuildDescription } from './commands'; +import { sendTriggeredCompletion } from './completionGate'; import { checkConflicts, ConflictCheck, watchForNewConflicts } from './conflicts'; import { resolveLaunch } from './payload'; import { ServerLogLevel, ServerLogRouter } from './serverLog'; @@ -259,6 +260,20 @@ class ServerHost implements vscode.Disposable { modules: configuration.get('semanticTokens.modules', true), moduleType: true, }, + // Fix plan 2026-09-26 F9: a space after `import` opens the module list. The server + // advertises the space as a trigger character to this client unless this is off. + completion: { + triggerOnSpace: configuration.get('completion.triggerOnSpace', true), + }, + }, + middleware: { + // Fix plan 2026-09-26 F9 (D4 layer 1): of the completions a typed space asks for, only + // the one after `import` or `export import` is sent; every other is answered here, with + // nothing, before it costs a message. + provideCompletionItem: (document, position, context, token, next) => + sendTriggeredCompletion(context.triggerCharacter, document.lineAt(position.line).text, position.character) + ? next(document, position, context, token) + : [], }, errorHandler: { error: () => ({ action: ErrorAction.Continue, handled: true }), @@ -496,7 +511,8 @@ export function activate(context: vscode.ExtensionContext): TestApi { vscode.workspace.onDidChangeConfiguration((event) => { if (event.affectsConfiguration('mcppls.compiler') || event.affectsConfiguration('mcppls.semanticKit') || event.affectsConfiguration('mcppls.engine') || event.affectsConfiguration('mcppls.buildTool') - || event.affectsConfiguration('mcppls.toolEnvironment') || event.affectsConfiguration('mcppls.semanticTokens.modules')) { + || event.affectsConfiguration('mcppls.toolEnvironment') || event.affectsConfiguration('mcppls.semanticTokens.modules') + || event.affectsConfiguration('mcppls.completion.triggerOnSpace')) { void host.restart(); } }), diff --git a/editors/vscode/test/unit/completionGate.test.ts b/editors/vscode/test/unit/completionGate.test.ts new file mode 100644 index 0000000..e519e08 --- /dev/null +++ b/editors/vscode/test/unit/completionGate.test.ts @@ -0,0 +1,27 @@ +// The editor side of the space trigger (fix plan 2026-09-26 F9, D4 layer 1), in plain Node: no VS +// Code, same reason test/unit/serverLog.test.ts is. The cases are the server's own +// (tests/test_completion.cpp), so the two gates cannot drift apart unnoticed. +import * as assert from 'assert'; +import { isImportLinePrefix, sendTriggeredCompletion } from '../../src/completionGate'; + +suite('space-triggered completion', () => { + test('an import directive keyword and one blank passes', () => { + for (const prefix of ['import ', 'export import ', ' import ', '\timport\t', 'export import ', '\texport\timport ']) { + assert.strictEqual(isImportLinePrefix(prefix), true, JSON.stringify(prefix)); + } + }); + + test('anything else does not', () => { + for (const prefix of ['import ', 'import', 'import a', 'import a ', 'int x = ', '', ' ', 'exportimport ', 'export ', + 'export module ', 'module ', 'importer ', '// import ', 'x import ']) { + assert.strictEqual(isImportLinePrefix(prefix), false, JSON.stringify(prefix)); + } + }); + + test('only a space is ever dropped, and only off an import line', () => { + assert.strictEqual(sendTriggeredCompletion(' ', 'int x = 1;', 8), false); + assert.strictEqual(sendTriggeredCompletion(' ', 'import hello;', 7), true, 'the line before the cursor is what counts'); + assert.strictEqual(sendTriggeredCompletion('.', 'import hello.', 13), true); + assert.strictEqual(sendTriggeredCompletion(undefined, 'int x = ', 8), true, 'typed or invoked completion always goes'); + }); +}); From 4adb312f8a7a0eab7a0de00a94e3208e646a3b53 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 26 Sep 2026 07:27:06 +0800 Subject: [PATCH 07/31] spec(s3): completion of module syntax -- a space after import, and the module keywords S3 section 6.2: a server may make a space a completion trigger character for the module names after `import`; one that does answers every other space-triggered request at once, empty, without its semantic engine (S3-6.2-1), and advertises the space only to a client that asks with initializationOptions.completion.triggerOnSpace, or that it knows drops the other spaces itself, never to one that said false (S3-6.2-2, S3-6.2-3). The module-syntax keywords are offered where each can begin a declaration, merged with the semantic engine's result, and without it when it does not answer in time (S3-6.2-4, S3-6.2-5). Additive: protocol version 1. Evidence in conformance/traceability.json. --- conformance/traceability.json | 68 +++++++++++++++++++++++++++++++++ docs/specs/CHANGELOG.md | 11 ++++++ docs/specs/s3-lsp-extensions.md | 18 +++++++++ 3 files changed, 97 insertions(+) diff --git a/conformance/traceability.json b/conformance/traceability.json index 255f80b..280894a 100644 --- a/conformance/traceability.json +++ b/conformance/traceability.json @@ -1032,6 +1032,74 @@ "test": "tests/test_tokens.cpp: clangd's legend, including its own duplicates, maps by name" } ], + "S3-6.2-1": [ + { + "test": "tests/test_completion.cpp: the space gate passes an import directive's keyword and one blank, and nothing else" + }, + { + "check": "completion-keywords/P1-space-elsewhere" + }, + { + "check": "completion-keywords/P4-two-spaces" + }, + { + "check": "engine-none/M4-space-elsewhere" + } + ], + "S3-6.2-2": [ + { + "test": "tests/test_completion.cpp: the space is advertised to VS Code and its forks, or to a client that asks" + }, + { + "script": "src/server/session.cpp", + "contains": "if (orchestrator::completion::space_trigger_wanted(clientParams_)) orchestrator::completion::add_space_trigger" + } + ], + "S3-6.2-3": [ + { + "test": "tests/test_completion.cpp: the space is advertised to VS Code and its forks, or to a client that asks" + }, + { + "check": "completion-keywords/C0-space-is-a-trigger" + } + ], + "S3-6.2-4": [ + { + "test": "tests/test_completion.cpp: module keywords where a declaration can begin, filtered by what was typed" + }, + { + "test": "tests/test_completion.cpp: no keywords inside braces, comments, literals, or the middle of a word" + }, + { + "test": "tests/test_completion.cpp: keywords merge with the core engine's answer without duplicates" + }, + { + "check": "completion-keywords/K1-import" + }, + { + "check": "completion-keywords/K3-export-import" + }, + { + "check": "completion-keywords/K4-private-fragment" + }, + { + "check": "completion-keywords/K5-not-in-a-body" + } + ], + "S3-6.2-5": [ + { + "check": "completion-keywords/K2-module-declarations" + }, + { + "check": "completion-keywords/R1-costs-counted" + }, + { + "check": "engine-none/M4-keywords" + }, + { + "check": "engine-none/M4-export-import" + } + ], "S4-3-1": [ { "validate": "S4 schema rejects unknown kit-version" diff --git a/docs/specs/CHANGELOG.md b/docs/specs/CHANGELOG.md index d7741f2..8a03cad 100644 --- a/docs/specs/CHANGELOG.md +++ b/docs/specs/CHANGELOG.md @@ -2,6 +2,17 @@ Changes to the specifications in this directory. Each specification is versioned independently. +## 2026-09-26 — S3: completion of module syntax + +Added section 6.2. A server may make a space a completion trigger character, for the module names +after `import`; if it does, a space-triggered request anywhere but right after `import ` or +`export import ` is answered at once, empty, without the semantic engine (S3-6.2-1), and the space is +advertised only to a client that asks for it (`initializationOptions.completion.triggerOnSpace`) or +that the server knows drops the other spaces itself; never to one that declared `false` (S3-6.2-2, +S3-6.2-3). Servers offer the module-syntax keywords where each can begin a declaration, merged with +the semantic engine's result, and without it when it does not answer in time (S3-6.2-4, S3-6.2-5). +All additive: protocol version stays 1. + ## 2026-09-25 — S3: issue categories, the degraded hold, module syntax in semantic tokens Added `category` (optional) to `CxxModulesIssue`: `code`, `engine`, `environment` or `project`, diff --git a/docs/specs/s3-lsp-extensions.md b/docs/specs/s3-lsp-extensions.md index 7f9bd0d..faa1853 100644 --- a/docs/specs/s3-lsp-extensions.md +++ b/docs/specs/s3-lsp-extensions.md @@ -229,6 +229,7 @@ The content of each `roots` entry is the server's own and may change between ser |---|---|---| | Go to the primary interface unit or partition declaration from a module name | `textDocument/definition` | the server's module index | | Import completion: module names and partitions of the same module | `textDocument/completion` | the server's module index | +| Module-syntax keywords where a declaration can begin (6.2) | `textDocument/completion` | the server, merged with the semantic engine's result | | Module-name hover: providers, role, semantic profile | `textDocument/hover` | the server's module index | | Module declaration as a top-level outline node | `textDocument/documentSymbol` | merged with the semantic engine's result | | Search by module name | `workspace/symbol` | merged with the semantic engine's result | @@ -253,6 +254,23 @@ interface CxxModulesInitializationOptions { A module name is sent with the token type `module`, and a partition name also with the modifier `partition`, only to a client that declared `moduleType: true`; to any other client a server **MUST** send module and partition names as `namespace`, so that a theme that knows only the standard types still colors them. A server **MUST NOT** add module-syntax tokens for a client that declared `modules: false`. The legend is the server's: it maps the core engine's token types and modifiers into it by name. S3-6.1-2, S3-6.1-3 +### 6.2 Completion of module syntax + +A space typed after `import` is where a person expects the module names. A server **MAY** add `" "` to `completionProvider.triggerCharacters` for it. A server that does **MUST** answer a completion request triggered by a space (`context.triggerKind` 2, `context.triggerCharacter` `" "`) whose line, up to the position, is anything but optional white space, an optional `export` and white space, `import` and exactly one white-space character, at once with an empty result, without giving it to the semantic engine. S3-6.2-1 + +Every space typed anywhere reaches the server as such a request, so a client that is sent the trigger should drop the others itself before they are sent. A server **MUST NOT** add `" "` for a client that declared `completion.triggerOnSpace: false`, and **SHOULD NOT** add it for a client that did not declare `true`, unless the server knows that client drops them. S3-6.2-2, S3-6.2-3 + +```ts +// Client → server: InitializeParams.initializationOptions +interface CxxModulesInitializationOptions { + completion?: { + triggerOnSpace?: boolean; // true: a space after `import` triggers completion; false: never + }; +} +``` + +A semantic engine may offer some module-syntax keywords, not their combined forms, and nothing while it cannot answer for a file. Where a module declaration or an import declaration can begin, a server **SHOULD** offer the keywords that can appear there — `import`; `export import` in a module interface unit; `module;` before anything else in the file; `export module` and `module` in a file with no module declaration; `module :private;` in a primary module interface unit without one — merged with the semantic engine's result without duplicate labels, and **SHOULD** still offer them when the semantic engine gives no result in time. S3-6.2-4, S3-6.2-5 No module name is offered after `export module`: the name is being declared, not referred to. + ## 7. Versioning The protocol version is an integer. Version 1 is defined by this document. A later version adds optional fields and new messages only; a change that is not backward compatible requires a new method prefix. From 2185dcd39cb40f7aecfe5ac13a2269b999cee776 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 26 Sep 2026 07:28:46 +0800 Subject: [PATCH 08/31] feat(bundle): what leaves the machine in a report names no one, is packed in a zip, and is checked before it is written Issue #23 fix plan F18. A new directory of modules, mcppls.bundle.*, that nothing in the engines depends on. redact: the user's home directory becomes ~ in every spelling a log or a report carries it in -- either separator, a backslash doubled by JSON or doubled again, an escaped slash, percent-encoding in a file:// URI, either case of the drive letter, the WSL view of a Windows drive, a Windows profile under its 8.3 name -- and only as a whole path, so /home/speaker is not /home/speak. Other people's profile directories (/home/, /Users/, C:\Users\) get their own placeholder, the same one every time. A distinctive user or host name is replaced wherever it stands as a word; a short or common one (runner, admin, a) only where it names a directory, so that a bundle of such a user is not refused forever. Secrets are found by their shape (GitHub, GitLab, Slack, OpenAI and AWS tokens, credentials in a URL, Bearer) and by their names (JSON keys, NAME=value, -DNAME=value, Authorization: headers), and e-mail addresses go too. A residue check looks for the home directory, the names and known tokens again afterwards, including the home as a plain string whatever follows it. zip: a zip written in memory, entries stored or deflated with fixed codes, CRC-32, UTF-8 names, and a reader for what it writes. The server carries no archive library. writer: the bundle -- manifest, report, environment (system, editor, extension, settings, payload, versions, probed toolchains, a whitelist of environment variables), the server's logs of the last sessions, each root's incidents and engine database and plan, the client's log, crash dumps only when asked for -- redacted by one Redactor so an original has one placeholder in every file, checked, compressed entry by entry and kept by priority within 25 MB, written to /bundles/ (the newest five kept) or where asked. Any residue and nothing is written. The tests cover every spelling on Linux and Windows forms, the secrets, the consistency of placeholders, the residue check, the deflate round trip and a whole bundle written by a child process whose HOME, user and cache carry a long user name in every path: no file names the user, the home or a secret, and the manifest's digests are the contents'. --- src/bundle/identity.cpp | 89 +++++ src/bundle/identity.cppm | 18 + src/bundle/redact.cpp | 816 +++++++++++++++++++++++++++++++++++++++ src/bundle/redact.cppm | 75 ++++ src/bundle/writer.cpp | 536 +++++++++++++++++++++++++ src/bundle/writer.cppm | 61 +++ src/bundle/zip.cpp | 381 ++++++++++++++++++ src/bundle/zip.cppm | 41 ++ tests/test_bundle.cpp | 373 ++++++++++++++++++ 9 files changed, 2390 insertions(+) create mode 100644 src/bundle/identity.cpp create mode 100644 src/bundle/identity.cppm create mode 100644 src/bundle/redact.cpp create mode 100644 src/bundle/redact.cppm create mode 100644 src/bundle/writer.cpp create mode 100644 src/bundle/writer.cppm create mode 100644 src/bundle/zip.cpp create mode 100644 src/bundle/zip.cppm create mode 100644 tests/test_bundle.cpp diff --git a/src/bundle/identity.cpp b/src/bundle/identity.cpp new file mode 100644 index 0000000..632ae23 --- /dev/null +++ b/src/bundle/identity.cpp @@ -0,0 +1,89 @@ +module mcppls.bundle.identity; + +import std; +import mcppls.os; +import mcppls.base.path; +import mcppls.base.text; +import mcppls.platform.dirs; +import mcppls.platform.env; +import mcppls.platform.fs; +import mcppls.bundle.redact; + +namespace mcppls::bundle { + +namespace { + +void add_unique(std::vector& list, std::string value) { + value = std::string { base::trim(value) }; + if (value.empty()) return; + if (std::ranges::any_of(list, [&](const std::string& known) { return base::iequals_ascii(known, value); })) return; + list.push_back(std::move(value)); +} + +std::string first_line(std::string_view text) { + const auto lines = base::split_lines(text); + return lines.empty() ? std::string {} : std::string { base::trim(lines.front()) }; +} + +// The value of `namevalue` in an XML property list. +std::optional plist_string(std::string_view plist, std::string_view key) { + const std::string marker { std::format("{}", key) }; + const std::size_t at { plist.find(marker) }; + if (at == std::string_view::npos) return std::nullopt; + const std::size_t open { plist.find("", at + marker.size()) }; + if (open == std::string_view::npos || open - (at + marker.size()) > 64) return std::nullopt; + const std::size_t close { plist.find("", open) }; + if (close == std::string_view::npos) return std::nullopt; + return std::string { plist.substr(open + 8, close - open - 8) }; +} + +std::vector host_names() { + std::vector hosts; + for (const std::string_view name : { "COMPUTERNAME", "HOSTNAME", "HOST" }) { + if (auto value = platform::env::get(name)) add_unique(hosts, *value); + } + if constexpr (mcppls::os::FAMILY == mcppls::os::Family::linux) { + for (const std::string_view file : { "/proc/sys/kernel/hostname", "/etc/hostname" }) { + if (auto text = platform::fs::read_file(file)) add_unique(hosts, first_line(*text)); + } + } else if constexpr (mcppls::os::FAMILY == mcppls::os::Family::macos) { + // What `scutil --get LocalHostName` and `ComputerName` answer, without starting scutil. A + // binary property list is not read; the environment and the bundle's own check remain. + if (auto plist = platform::fs::read_file("/Library/Preferences/SystemConfiguration/preferences.plist"); plist && !plist->starts_with("bplist")) { + for (const std::string_view key : { "LocalHostName", "ComputerName", "HostName" }) { + if (auto value = plist_string(*plist, key)) add_unique(hosts, *value); + } + } + } + return hosts; +} + +} // namespace + +Identity current_identity() { + Identity identity; + const std::string home { platform::dirs::home_directory() }; + add_unique(identity.homes, base::normalize_path(home)); + add_unique(identity.homes, base::normalize_path(platform::fs::canonical_path(home))); + if constexpr (mcppls::os::FAMILY == mcppls::os::Family::windows) { + // HOMEDRIVE and HOMEPATH can name another directory than USERPROFILE (a redirected home). + auto drive = platform::env::get("HOMEDRIVE"); + auto path = platform::env::get("HOMEPATH"); + if (drive && path && !path->empty() && *path != "\\") add_unique(identity.homes, base::normalize_path(*drive + *path)); + } + for (const std::string_view name : { "USER", "LOGNAME", "USERNAME" }) { + if (auto value = platform::env::get(name)) add_unique(identity.users, *value); + } + for (const auto& spelling : identity.homes) add_unique(identity.users, std::string { base::file_name(spelling) }); + identity.hosts = host_names(); + return identity; +} + +Identity hiding_workspaces(Identity identity, std::vector roots) { + for (auto& root : roots) { + if (!root.empty()) identity.workspaces.push_back(base::normalize_path(root)); + } + return identity; +} + +} // namespace mcppls::bundle diff --git a/src/bundle/identity.cppm b/src/bundle/identity.cppm new file mode 100644 index 0000000..01449cc --- /dev/null +++ b/src/bundle/identity.cppm @@ -0,0 +1,18 @@ +// Who this process runs as, for what a report or a bundle must not name (issue #23 fix plan F18): +// read from the environment and a few system files, never by starting a program, so the event loop +// may ask for it. +export module mcppls.bundle.identity; + +import std; +import mcppls.bundle.redact; + +export namespace mcppls::bundle { + +// The home directory as set and as the file system names it (a symbolic link, an 8.3 alias +// resolved), the login name and the home directory's last component, and the machine's names. +Identity current_identity(); + +// A copy of `identity` that also hides `roots` as , , ... +Identity hiding_workspaces(Identity identity, std::vector roots); + +} // namespace mcppls::bundle diff --git a/src/bundle/redact.cpp b/src/bundle/redact.cpp new file mode 100644 index 0000000..967d1dd --- /dev/null +++ b/src/bundle/redact.cpp @@ -0,0 +1,816 @@ +module mcppls.bundle.redact; + +import std; +import nlohmann.json; + +namespace mcppls::bundle { + +namespace { + +using Json = nlohmann::json; + +bool is_alpha(unsigned char c) { return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); } +bool is_digit(unsigned char c) { return c >= '0' && c <= '9'; } +bool is_alnum(unsigned char c) { return is_alpha(c) || is_digit(c); } +// A letter of any script (a UTF-8 byte of a multi-byte character) or a digit: what a word is made of. +bool is_word(unsigned char c) { return is_alnum(c) || c >= 0x80; } +char lower(char c) { return c >= 'A' && c <= 'Z' ? static_cast(c - 'A' + 'a') : c; } +char upper(char c) { return c >= 'a' && c <= 'z' ? static_cast(c - 'a' + 'A') : c; } + +std::string lowered(std::string_view text) { + std::string out { text }; + for (char& c : out) c = lower(c); + return out; +} + +int hex_value(char c) { + if (c >= '0' && c <= '9') return c - '0'; + if (c >= 'a' && c <= 'f') return c - 'a' + 10; + if (c >= 'A' && c <= 'F') return c - 'A' + 10; + return -1; +} + +// The byte a %XX escape at `i` stands for. +std::optional escaped_at(std::string_view text, std::size_t i) { + if (i + 2 >= text.size() || text[i] != '%') return std::nullopt; + const int high { hex_value(text[i + 1]) }; + const int low { hex_value(text[i + 2]) }; + if (high < 0 || low < 0) return std::nullopt; + return static_cast(high * 16 + low); +} + +// ---- paths in any spelling --------------------------------------------------------------------- +// +// One path, written the ways it reaches a log or a report: '/' or '\' separators, a backslash +// doubled (JSON) or doubled again (JSON inside JSON), "\/" (a JSON writer escaping '/'), any byte +// percent-encoded (a file:// URI: "c%3A/Users/John%20Doe"), the drive letter in either case, a +// Windows drive seen from WSL (/mnt/c/...), and a Windows profile under its 8.3 name (RUNNER~1). +// Names compare case-insensitively: Windows and macOS file systems do, and nothing is lost on Linux +// by also catching "/HOME/SPEAK". + +enum class PieceKind { drive, separator, literal, short_name }; + +struct Piece { + PieceKind kind { PieceKind::literal }; + std::string text; // drive: the letter; literal: the name; short_name: the 8.3 prefix +}; + +struct PathPattern { + std::vector pieces; + std::string rule; + std::string replacement; +}; + +// The number of bytes a separator takes at `i`, 0 when there is none. +std::size_t match_separator(std::string_view text, std::size_t i) { + if (i >= text.size()) return 0; + if (text[i] == '/') return 1; + if (text[i] == '\\') { + std::size_t j { i }; + while (j < text.size() && text[j] == '\\') ++j; + if (j < text.size() && text[j] == '/') return j + 1 - i; // "\/", a JSON writer escaping '/' + return j - i; + } + if (const auto escaped = escaped_at(text, i); escaped && (*escaped == '/' || *escaped == '\\')) return 3; + return 0; +} + +// The number of bytes `literal` takes at `i`, each of its bytes as itself or percent-encoded; 0 when it is not there. +std::size_t match_literal(std::string_view text, std::size_t i, std::string_view literal) { + std::size_t j { i }; + for (const char expected : literal) { + if (j < text.size() && lower(text[j]) == lower(expected)) { + ++j; + continue; + } + if (const auto escaped = escaped_at(text, j); escaped && lower(*escaped) == lower(expected)) { + j += 3; + continue; + } + return 0; + } + return j - i; +} + +std::size_t match_short_name(std::string_view text, std::size_t i, std::string_view prefix) { + const std::size_t head { match_literal(text, i, prefix) }; + if (head == 0) return 0; + std::size_t j { i + head }; + if (j >= text.size() || text[j] != '~') return 0; + ++j; + const std::size_t digits { j }; + while (j < text.size() && is_digit(static_cast(text[j]))) ++j; + return j == digits ? 0 : j - i; +} + +std::size_t match_drive(std::string_view text, std::size_t i, char letter) { + if (i >= text.size() || lower(text[i]) != letter) return 0; + if (i + 1 < text.size() && text[i + 1] == ':') return 2; + if (const auto escaped = escaped_at(text, i + 1); escaped && *escaped == ':') return 4; + return 0; +} + +// Whether a name goes on at `j`: "/home/speak" is not in "/home/speaker" nor in "/home/speak.old", +// but it is in "/home/speak." at the end of a sentence and in "/home/speak/src". +bool name_continues(std::string_view text, std::size_t j) { + if (j >= text.size()) return false; + const auto c = static_cast(text[j]); + if (is_word(c) || c == '_') return true; + if (c == '.' || c == '-' || c == '+' || c == '~') { + if (j + 1 >= text.size()) return false; + const auto next = static_cast(text[j + 1]); + return is_word(next) || next == '_'; + } + if (const auto escaped = escaped_at(text, j)) return is_alnum(static_cast(*escaped)); + return false; +} + +// The end of `pattern` matched at `i`, when it is there as a whole path of its own. +std::optional match_path(std::string_view text, std::size_t i, const PathPattern& pattern) { + if (pattern.pieces.empty()) return std::nullopt; + if (i > 0) { + // Not the tail of a longer name: "/data/home/speak" does not contain the home "/home/speak". + const auto before = static_cast(text[i - 1]); + if (is_word(before) || before == '_' || before == '.' || before == '-') return std::nullopt; + } + std::size_t j { i }; + for (const auto& piece : pattern.pieces) { + std::size_t taken { 0 }; + switch (piece.kind) { + case PieceKind::drive: taken = match_drive(text, j, piece.text.front()); break; + case PieceKind::separator: taken = match_separator(text, j); break; + case PieceKind::literal: taken = match_literal(text, j, piece.text); break; + case PieceKind::short_name: taken = match_short_name(text, j, piece.text); break; + } + if (taken == 0) return std::nullopt; + j += taken; + } + if (name_continues(text, j)) return std::nullopt; + return j; +} + +// Whether a pattern could start at `i`, cheaply, before match_path looks. +bool could_start(std::string_view text, std::size_t i, const PathPattern& pattern) { + const Piece& first { pattern.pieces.front() }; + const char c { text[i] }; + if (first.kind == PieceKind::drive) return lower(c) == first.text.front(); + return c == '/' || c == '\\' || c == '%'; +} + +// The segments of a '/'-separated path, empty ones dropped. +std::vector segments_of(std::string_view path) { + std::vector segments; + std::string current; + for (const char c : path) { + if (c == '/' || c == '\\') { + if (!current.empty()) segments.push_back(std::move(current)); + current.clear(); + } else { + current += c; + } + } + if (!current.empty()) segments.push_back(std::move(current)); + return segments; +} + +bool is_windows_path(std::string_view path) { + return path.size() >= 2 && is_alpha(static_cast(path[0])) && path[1] == ':'; +} + +PathPattern pattern_of(std::string_view path, std::string rule, std::string replacement) { + PathPattern pattern { {}, std::move(rule), std::move(replacement) }; + std::vector segments; + if (is_windows_path(path)) { + pattern.pieces.push_back(Piece { PieceKind::drive, std::string(1, lower(path[0])) }); + segments = segments_of(path.substr(2)); + } else { + segments = segments_of(path); + } + for (auto& segment : segments) { + pattern.pieces.push_back(Piece { PieceKind::separator, {} }); + pattern.pieces.push_back(Piece { PieceKind::literal, std::move(segment) }); + } + return pattern; +} + +// The 8.3 name Windows gives a directory whose own name is not one (longer than eight bytes, or +// with a space or a second dot): its first six valid characters, upper case, then ~N. +std::optional short_name_prefix(std::string_view name) { + std::string valid; + for (const char c : name) { + if (c == ' ' || c == '.' || std::string_view { "\"*+,/:;<=>?[\\]|" }.contains(c)) continue; + valid += upper(c); + } + const bool needsOne { name.size() > 8 || name.contains(' ') || std::ranges::count(name, '.') > 1 || valid.size() != name.size() }; + if (!needsOne || valid.size() < 2) return std::nullopt; + return valid.substr(0, 6); +} + +// Every spelling of a path worth looking for: itself, and for a Windows path the WSL view of it +// and, for its last component, the 8.3 name. +std::vector path_patterns(std::string_view path, std::string_view rule, std::string_view replacement) { + std::vector patterns; + patterns.push_back(pattern_of(path, std::string { rule }, std::string { replacement })); + if (is_windows_path(path)) { + const std::string wsl { std::format("/mnt/{}{}", lower(path[0]), path.substr(2)) }; + patterns.push_back(pattern_of(wsl, std::string { rule }, std::string { replacement })); + auto shortened = patterns.front(); + auto& last = shortened.pieces.back(); + if (last.kind == PieceKind::literal) { + if (auto prefix = short_name_prefix(last.text)) { + last = Piece { PieceKind::short_name, std::move(*prefix) }; + patterns.push_back(std::move(shortened)); + } + } + } + return patterns; +} + +// Replaces every match of `pattern`; returns how many there were. +std::size_t replace_path(std::string& text, const PathPattern& pattern) { + if (pattern.pieces.empty() || text.empty()) return 0; + std::string out; + std::size_t count { 0 }; + std::size_t copied { 0 }; + for (std::size_t i { 0 }; i < text.size();) { + if (could_start(text, i, pattern)) { + if (const auto end = match_path(text, i, pattern)) { + if (out.empty()) out.reserve(text.size()); + out.append(text, copied, i - copied); + out += pattern.replacement; + copied = i = *end; + ++count; + continue; + } + } + ++i; + } + if (count == 0) return 0; + out.append(text, copied); + text = std::move(out); + return count; +} + +std::optional find_path(std::string_view text, const PathPattern& pattern) { + if (pattern.pieces.empty()) return std::nullopt; + for (std::size_t i { 0 }; i < text.size(); ++i) { + if (could_start(text, i, pattern) && match_path(text, i, pattern)) return i; + } + return std::nullopt; +} + +// ---- words ------------------------------------------------------------------------------------- + +// Case-insensitive search for `needle` from `from`. +std::size_t ifind(std::string_view haystack, std::string_view needle, std::size_t from = 0) { + if (needle.empty() || haystack.size() < needle.size()) return std::string_view::npos; + const char first { lower(needle.front()) }; + for (std::size_t i { from }; i + needle.size() <= haystack.size(); ++i) { + if (lower(haystack[i]) != first) continue; + bool same { true }; + for (std::size_t k { 1 }; k < needle.size() && same; ++k) same = lower(haystack[i + k]) == lower(needle[k]); + if (same) return i; + } + return std::string_view::npos; +} + +// A whole word: neither neighbour a letter or digit. '_', '-' and '.' separate words here, so +// "speak_dev" and "speak.log" name the user "speak", "speaker" does not. +bool whole_word_at(std::string_view text, std::size_t at, std::size_t length) { + if (at > 0 && is_word(static_cast(text[at - 1]))) return false; + const std::size_t end { at + length }; + return end >= text.size() || !is_word(static_cast(text[end])); +} + +// A name as a word, but not as a JSON object's key: a key is this program's own vocabulary ("plan", +// "server"), and a user of that name leaks nothing through it. +bool name_word_at(std::string_view text, std::size_t at, std::size_t length) { + if (!whole_word_at(text, at, length)) return false; + const std::size_t end { at + length }; + if (at == 0 || text[at - 1] != '"' || end >= text.size() || text[end] != '"') return true; + std::size_t next { end + 1 }; + while (next < text.size() && (text[next] == ' ' || text[next] == '\t')) ++next; + return next >= text.size() || text[next] != ':'; +} + +// A directory name: right after a separator, and not going on as a longer name. +bool directory_name_at(std::string_view text, std::size_t at, std::size_t length) { + if (at == 0 || (text[at - 1] != '/' && text[at - 1] != '\\')) return false; + return !name_continues(text, at + length); +} + +template +std::size_t replace_words(std::string& text, std::string_view word, std::string_view replacement, Accept accept) { + if (word.empty()) return 0; + std::string out; + std::size_t count { 0 }; + std::size_t copied { 0 }; + for (std::size_t at { ifind(text, word) }; at != std::string::npos; at = ifind(text, word, at + 1)) { + if (at < copied || !accept(std::string_view { text }, at, word.size())) continue; + out.append(text, copied, at - copied); + out += replacement; + copied = at + word.size(); + ++count; + } + if (count == 0) return 0; + out.append(text, copied); + text = std::move(out); + return count; +} + +// ---- secrets ----------------------------------------------------------------------------------- + +// The words of an identifier: "apiKey", "API_KEY", "x-api-key" and "ApiKey" are all {api, key}. +std::vector words_of(std::string_view name) { + std::vector words; + std::string current; + const auto flush = [&] { + if (!current.empty()) words.push_back(lowered(current)); + current.clear(); + }; + for (std::size_t i { 0 }; i < name.size(); ++i) { + const auto c = static_cast(name[i]); + if (!is_alnum(c)) { + flush(); + continue; + } + if (!current.empty()) { + const auto previous = static_cast(name[i - 1]); + const bool lowerToUpper { (previous >= 'a' && previous <= 'z') && (c >= 'A' && c <= 'Z') }; + // "APIKey": the K starts a word because a lower-case letter follows it. + const bool upperRunEnds { (previous >= 'A' && previous <= 'Z') && (c >= 'A' && c <= 'Z') && i + 1 < name.size() + && name[i + 1] >= 'a' && name[i + 1] <= 'z' }; + if (lowerToUpper || upperRunEnds) flush(); + } + current += static_cast(c); + } + flush(); + return words; +} + +constexpr std::array SECRET_WORDS { + "token", "secret", "secrets", "password", "passwords", "passwd", "pwd", "passphrase", "credential", "credentials", + "authorization", "auth", "cookie", "apikey", "privatekey", "accesskey", "secretkey", "bearer", +}; + +constexpr std::array, 4> SECRET_PAIRS { + std::pair { "api", "key" }, std::pair { "private", "key" }, std::pair { "access", "key" }, std::pair { "session", "key" }, +}; + +// Values that are not secrets whatever their key says. +bool harmless_value(std::string_view value) { + const std::string lowerValue { lowered(value) }; + return value.empty() || value.starts_with('<') || lowerValue == "null" || lowerValue == "true" || lowerValue == "false" + || lowerValue == "none" || lowerValue == "***"; +} + +// Whether the `"` at `at` is escaped by the backslashes before it. +bool escaped_quote(std::string_view text, std::size_t at) { + std::size_t backslashes { 0 }; + while (at > backslashes && text[at - 1 - backslashes] == '\\') ++backslashes; + return backslashes % 2 == 1; +} + +struct Span { + std::size_t begin { 0 }; + std::size_t end { 0 }; +}; + +// Spans are collected over the original text, then replaced in one pass. +std::string replace_spans(std::string_view text, std::vector spans, std::string_view replacement) { + std::ranges::sort(spans, {}, &Span::begin); + std::string out; + out.reserve(text.size()); + std::size_t copied { 0 }; + for (const auto& span : spans) { + if (span.begin < copied) continue; // overlaps one already replaced + out.append(text, copied, span.begin - copied); + out += replacement; + copied = span.end; + } + out.append(text, copied); + return out; +} + +// The end of a value that follows `=` or `: `: at white space, a quote that is not escaped, `&`, +// or the end of the line. +std::size_t value_end(std::string_view text, std::size_t begin, bool toEndOfLine = false) { + std::size_t j { begin }; + while (j < text.size()) { + const char c { text[j] }; + if (c == '\n' || c == '\r') break; + if (!toEndOfLine && (c == ' ' || c == '\t' || c == '\'' || c == '&')) break; + if (c == '"' && !escaped_quote(text, j)) break; + ++j; + } + return j; +} + +bool token_char(unsigned char c) { return is_alnum(c) || c == '_' || c == '-'; } + +constexpr std::array TOKEN_PREFIXES { + "ghp_", "gho_", "ghu_", "ghs_", "ghr_", "github_pat_", "glpat-", "xoxb-", "xoxp-", "xoxa-", "xoxr-", "sk-", "AKIA", +}; + +// Tokens by their well-known prefix: GitHub, GitLab, Slack, OpenAI and Anthropic keys, AWS access keys. +void find_prefixed_tokens(std::string_view text, std::vector& spans) { + for (const auto prefix : TOKEN_PREFIXES) { + const bool aws { prefix == "AKIA" }; + for (std::size_t at { text.find(prefix) }; at != std::string_view::npos; at = text.find(prefix, at + 1)) { + if (at > 0 && (is_alnum(static_cast(text[at - 1])) || text[at - 1] == '_')) continue; + std::size_t end { at + prefix.size() }; + while (end < text.size() && token_char(static_cast(text[end]))) ++end; + const std::size_t body { end - at - prefix.size() }; + if (aws ? body != 16 : body < 16) continue; + spans.push_back(Span { at, end }); + } + } +} + +// "scheme://user:password@host": the credentials. +void find_url_credentials(std::string_view text, std::vector& spans) { + for (std::size_t at { text.find("://") }; at != std::string_view::npos; at = text.find("://", at + 3)) { + const std::size_t begin { at + 3 }; + std::size_t j { begin }; + std::optional colon; + std::optional atSign; + while (j < text.size()) { + const char c { text[j] }; + if (c == '/' || c == ' ' || c == '"' || c == '\'' || c == '\n' || c == '\\' || c == '?' || c == '#') break; + if (c == ':' && !colon) colon = j; + if (c == '@') atSign = j; + ++j; + } + if (atSign && colon && *colon < *atSign && *atSign > begin) spans.push_back(Span { begin, *atSign }); + } +} + +// "Bearer ", "Basic ". +void find_authorization_schemes(std::string_view text, std::vector& spans) { + for (const std::string_view scheme : { "Bearer ", "bearer ", "Basic ", "basic " }) { + for (std::size_t at { text.find(scheme) }; at != std::string_view::npos; at = text.find(scheme, at + 1)) { + if (at > 0 && is_alnum(static_cast(text[at - 1]))) continue; + const std::size_t begin { at + scheme.size() }; + std::size_t end { begin }; + while (end < text.size() && (token_char(static_cast(text[end])) || std::string_view { "._~+/=" }.contains(text[end]))) ++end; + if (end - begin >= 8 && !harmless_value(text.substr(begin, end - begin))) spans.push_back(Span { begin, end }); + } + } +} + +// "key": "value", where the key names a secret: the value. +void find_json_secrets(std::string_view text, std::vector& spans) { + for (std::size_t open { text.find('"') }; open != std::string_view::npos;) { + std::size_t close { open + 1 }; + while (close < text.size() && close - open <= 64 && text[close] != '"' && text[close] != '\n') { + close += text[close] == '\\' ? 2 : 1; + } + if (close >= text.size() || text[close] != '"' || close - open > 64) { + open = text.find('"', open + 1); + continue; + } + std::size_t j { close + 1 }; + while (j < text.size() && (text[j] == ' ' || text[j] == '\t')) ++j; + if (j >= text.size() || text[j] != ':') { + open = text.find('"', close); + continue; + } + ++j; + while (j < text.size() && (text[j] == ' ' || text[j] == '\t')) ++j; + if (j < text.size() && text[j] == '"' && secret_name(text.substr(open + 1, close - open - 1))) { + std::size_t end { j + 1 }; + while (end < text.size() && !(text[end] == '"' && !escaped_quote(text, end)) && text[end] != '\n') ++end; + if (end < text.size() && text[end] == '"' && !harmless_value(text.substr(j + 1, end - j - 1))) spans.push_back(Span { j + 1, end }); + open = end < text.size() ? text.find('"', end + 1) : std::string_view::npos; + continue; + } + open = text.find('"', close + 1); + } +} + +// NAME=value (an environment variable, a --flag=value, a -DNAME=value define) whose name names a secret. +void find_assigned_secrets(std::string_view text, std::vector& spans) { + for (std::size_t equals { text.find('=') }; equals != std::string_view::npos; equals = text.find('=', equals + 1)) { + std::size_t start { equals }; + while (start > 0 && (is_alnum(static_cast(text[start - 1])) || text[start - 1] == '_' || text[start - 1] == '-' + || text[start - 1] == '.')) { + --start; + } + while (start < equals && text[start] == '-') ++start; // --flag + if (start == equals) continue; + std::string_view name { text.substr(start, equals - start) }; + bool secret { secret_name(name) }; + // -DAPI_KEY=...: the define's name follows the D. + if (!secret && start >= 1 && text[start - 1] == '-' && name.size() > 1 && name.front() == 'D') secret = secret_name(name.substr(1)); + if (!secret) continue; + const std::size_t end { value_end(text, equals + 1) }; + if (end > equals + 1 && !harmless_value(text.substr(equals + 1, end - equals - 1))) spans.push_back(Span { equals + 1, end }); + } +} + +// "Authorization: ...", "password: ..." in plain text: the rest of the line. Only names that are +// never anything else; a bare "token:" is too common in a log to mean a secret. +void find_header_secrets(std::string_view text, std::vector& spans) { + static constexpr std::array HEADERS { "authorization:", "proxy-authorization:", "password:", "passwd:", "x-api-key:", + "api-key:", "api_key:", "private-token:", "client_secret:", "secret:" }; + for (const auto header : HEADERS) { + for (std::size_t at { ifind(text, header) }; at != std::string_view::npos; at = ifind(text, header, at + 1)) { + if (at > 0 && (is_alnum(static_cast(text[at - 1])) || text[at - 1] == '_' || text[at - 1] == '-' || text[at - 1] == '"')) continue; + std::size_t begin { at + header.size() }; + while (begin < text.size() && (text[begin] == ' ' || text[begin] == '\t')) ++begin; + const std::size_t end { value_end(text, begin, true) }; + if (end > begin && !harmless_value(text.substr(begin, end - begin))) spans.push_back(Span { begin, end }); + } + } +} + +bool email_local_char(unsigned char c) { return is_alnum(c) || std::string_view { "._%+-" }.contains(static_cast(c)); } + +// local@domain.tld, the domain with at least two labels and an alphabetic top-level label. +void find_emails(std::string_view text, std::vector& spans) { + for (std::size_t at { text.find('@') }; at != std::string_view::npos; at = text.find('@', at + 1)) { + std::size_t begin { at }; + while (begin > 0 && at - begin < 64 && email_local_char(static_cast(text[begin - 1]))) --begin; + if (begin == at || text[begin] == '.' || text[at - 1] == '.') continue; + if (lowered(text.substr(begin, at - begin)) == "git") continue; // git@github.com:owner/repo is an address, not a person + std::size_t end { at + 1 }; + std::size_t labels { 0 }; + std::size_t lastLabel { end }; + bool alphabeticTop { false }; + while (end < text.size()) { + const std::size_t labelBegin { end }; + while (end < text.size() && (is_alnum(static_cast(text[end])) || text[end] == '-')) ++end; + if (end == labelBegin) break; + ++labels; + lastLabel = labelBegin; + alphabeticTop = end - labelBegin >= 2 && std::ranges::all_of(text.substr(labelBegin, end - labelBegin), [](char c) { return is_alpha(static_cast(c)); }); + if (end + 1 < text.size() && text[end] == '.' && is_alnum(static_cast(text[end + 1]))) { + ++end; + continue; + } + break; + } + (void)lastLabel; + if (labels >= 2 && alphabeticTop) spans.push_back(Span { begin, end }); + } +} + +constexpr std::array COMMON_NAMES { + "admin", "administrator", "user", "users", "guest", "root", "test", "tests", "tester", "runner", "build", "builder", + "developer", "home", "public", "default", "shared", "ubuntu", "debian", "fedora", "centos", "docker", "vagrant", "ec2-user", + "macos", "windows", "linux", "local", "localhost", "owner", "work", "workspace", "code", "mcpp", "clang", "clangd", "server", + "client", "data", "temp", "system", "service", "demo", +}; + +// Directories under a profile root that are no person's. +constexpr std::array NOBODY { "public", "default", "default user", "all users", "shared", "linuxbrew", "guest", "defaultapppool" }; + +} // namespace + +bool distinctive_name(std::string_view name) { + if (name.size() < 4) return false; + const std::string lowerName { lowered(name) }; + return std::ranges::find(COMMON_NAMES, std::string_view { lowerName }) == COMMON_NAMES.end(); +} + +bool secret_name(std::string_view name) { + const auto words = words_of(name); + for (std::size_t i { 0 }; i < words.size(); ++i) { + if (std::ranges::find(SECRET_WORDS, std::string_view { words[i] }) != SECRET_WORDS.end()) return true; + if (i + 1 < words.size()) { + for (const auto& [first, second] : SECRET_PAIRS) { + if (words[i] == first && words[i + 1] == second) return true; + } + } + } + return false; +} + +struct Redactor::Impl { + Identity identity; + std::vector workspacePatterns; // longest first + std::vector homePatterns; // longest first + std::vector homeTexts; // each home in its '/' and '\' spellings, for the residue check + std::vector shortNames; // 8.3 prefixes of the user names ("RUNNER" of runneradmin) + std::vector users; + std::vector hosts; + std::map> userPlaceholders; // lower-case name -> , , ... + std::map> hits; + + explicit Impl(Identity who) : identity { std::move(who) } { + const auto byLength = [](const PathPattern& a, const PathPattern& b) { return a.pieces.size() > b.pieces.size(); }; + for (std::size_t i { 0 }; i < identity.workspaces.size(); ++i) { + if (identity.workspaces[i].empty()) continue; + const std::string placeholder { i == 0 ? std::string { "" } : std::format("", i + 1) }; + for (auto& pattern : path_patterns(identity.workspaces[i], RULE_WORKSPACE, placeholder)) workspacePatterns.push_back(std::move(pattern)); + } + std::ranges::stable_sort(workspacePatterns, byLength); + for (const auto& home : identity.homes) { + if (segments_of(home).empty()) continue; // "/" or "C:/": nothing personal to hide + for (auto& pattern : path_patterns(home, RULE_HOME, "~")) homePatterns.push_back(std::move(pattern)); + std::string backslashed { home }; + std::ranges::replace(backslashed, '/', '\\'); + homeTexts.push_back(home); + homeTexts.push_back(std::move(backslashed)); + } + std::ranges::stable_sort(homePatterns, byLength); + for (const auto& user : identity.users) { + if (user.empty() || std::ranges::find_if(users, [&](const std::string& known) { return lowered(known) == lowered(user); }) != users.end()) continue; + users.push_back(user); + userPlaceholders.emplace(lowered(user), ""); + if (auto prefix = short_name_prefix(user)) shortNames.push_back(std::move(*prefix)); + } + for (const auto& host : identity.hosts) { + if (host.empty()) continue; + hosts.push_back(host); + // "mac-mini.local": the name before the domain is the machine too. + if (const auto dot = host.find('.'); dot != std::string::npos && dot > 0) hosts.push_back(host.substr(0, dot)); + } + std::ranges::sort(hosts, [](const std::string& a, const std::string& b) { return a.size() > b.size(); }); + } + + void count(std::string_view rule, std::size_t n) { + if (n == 0) return; + auto it = hits.find(rule); + if (it == hits.end()) it = hits.emplace(std::string { rule }, 0).first; + it->second += n; + } + + std::string placeholder_for_user(std::string_view name) { + const std::string key { lowered(name) }; + if (const auto known = userPlaceholders.find(key); known != userPlaceholders.end()) return known->second; + // is the user's own; everybody else counts from 2, in the order they are met. + const auto others = std::ranges::count_if(userPlaceholders, [](const auto& item) { return item.second != ""; }); + std::string placeholder { std::format("", others + 2) }; + userPlaceholders.emplace(key, placeholder); + return placeholder; + } + + // Profile directories of other people: /home/, /Users/, C:\Users\, /mnt/c/Users/. + // What follows the profile root is the person's name, whoever it is. + std::size_t replace_profile_directories(std::string& text) { + return replace_profile_directories(text, "home") + replace_profile_directories(text, "users"); + } + + std::size_t replace_profile_directories(std::string& text, std::string_view root) { + std::size_t count { 0 }; + std::string out; + std::size_t copied { 0 }; + for (std::size_t at { ifind(text, root) }; at != std::string::npos; at = ifind(text, root, at + 1)) { + if (at < copied || at == 0) continue; + // The root segment follows a separator run that starts the path, follows a drive, or follows /mnt/. + std::size_t runStart { at }; + while (runStart > 0 && (text[runStart - 1] == '/' || text[runStart - 1] == '\\')) --runStart; + if (runStart == at) continue; + const auto before = [&](std::size_t back) { return static_cast(text[runStart - back]); }; + const bool atRoot { runStart == 0 || !(is_word(before(1)) || before(1) == '_' || before(1) == '.') }; + const bool afterDrive { runStart >= 2 && before(1) == ':' && is_alpha(before(2)) && (runStart == 2 || !is_word(before(3))) }; + const bool afterMount { runStart >= 6 && is_alpha(before(1)) && lowered(std::string_view { text }.substr(runStart - 6, 5)) == "/mnt/" }; + if (!atRoot && !afterDrive && !afterMount) continue; + const std::size_t separator { match_separator(text, at + root.size()) }; + if (separator == 0) continue; + const std::size_t nameBegin { at + root.size() + separator }; + // The name ends at the next separator; a space belongs to it only when a separator follows the name. + static constexpr std::string_view ENDS_NAME { "/\\\"'\n\r\t:<>|*?,;()[]{}" }; + std::size_t nameEnd { nameBegin }; + std::optional firstSpace; + while (nameEnd < text.size() && !ENDS_NAME.contains(text[nameEnd])) { + if (text[nameEnd] == ' ' && !firstSpace) firstSpace = nameEnd; + ++nameEnd; + } + const bool separatorFollows { nameEnd < text.size() && (text[nameEnd] == '/' || text[nameEnd] == '\\') }; + if (firstSpace && !separatorFollows) nameEnd = *firstSpace; + while (nameEnd > nameBegin && text[nameEnd - 1] == '.') --nameEnd; // the end of a sentence, not of the name + if (nameEnd == nameBegin) continue; + const std::string_view name { std::string_view { text }.substr(nameBegin, nameEnd - nameBegin) }; + if (name.starts_with('<') || name.starts_with('~') || name.starts_with('%') || name.starts_with('$') + || std::ranges::find(NOBODY, std::string_view { lowered(name) }) != NOBODY.end()) continue; + out.append(text, copied, nameBegin - copied); + out += placeholder_for_user(name); + copied = nameEnd; + ++count; + } + if (count == 0) return 0; + out.append(text, copied); + text = std::move(out); + return count; + } +}; + +Redactor::Redactor(Identity identity) : impl_ { std::make_unique(std::move(identity)) } {} +Redactor::~Redactor() = default; +Redactor::Redactor(Redactor&&) noexcept = default; +Redactor& Redactor::operator=(Redactor&&) noexcept = default; + +const std::map>& Redactor::hits() const { return impl_->hits; } + +std::string Redactor::redact(std::string_view input) { + Impl& impl { *impl_ }; + // Secrets first, on the text as it came: a token in a URL or a define is found by its shape + // before any path around it is rewritten. + std::vector secrets; + find_prefixed_tokens(input, secrets); + find_url_credentials(input, secrets); + find_authorization_schemes(input, secrets); + find_json_secrets(input, secrets); + find_assigned_secrets(input, secrets); + find_header_secrets(input, secrets); + std::string text; + if (secrets.empty()) { + text = std::string { input }; + } else { + impl.count(RULE_SECRET, secrets.size()); + text = replace_spans(input, std::move(secrets), ""); + } + std::vector emails; + find_emails(text, emails); + if (!emails.empty()) { + impl.count(RULE_EMAIL, emails.size()); + text = replace_spans(text, std::move(emails), ""); + } + // The project's own paths before the home they are under, the longest spelling first. + for (const auto& pattern : impl.workspacePatterns) impl.count(RULE_WORKSPACE, replace_path(text, pattern)); + for (const auto& pattern : impl.homePatterns) impl.count(RULE_HOME, replace_path(text, pattern)); + // A profile under its 8.3 name without the drive before it (RUNNER~1). + for (const auto& prefix : impl.shortNames) { + std::size_t n { 0 }; + for (std::size_t at { ifind(text, prefix) }; at != std::string::npos; at = ifind(text, prefix, at + 1)) { + if (at > 0 && is_word(static_cast(text[at - 1]))) continue; + const std::size_t length { match_short_name(text, at, prefix) }; + if (length == 0 || (at + length < text.size() && is_word(static_cast(text[at + length])))) continue; + text.replace(at, length, ""); + ++n; + } + impl.count(RULE_SHORT_NAME, n); + } + impl.count(RULE_USER, impl.replace_profile_directories(text)); + for (const auto& user : impl.users) { + impl.count(RULE_USER, distinctive_name(user) ? replace_words(text, user, "", name_word_at) + : replace_words(text, user, "", directory_name_at)); + } + for (const auto& host : impl.hosts) { + // A host name that is a common word ("ubuntu", "localhost") identifies nobody and stays. + if (distinctive_name(host)) impl.count(RULE_HOST, replace_words(text, host, "", name_word_at)); + } + return text; +} + +Json Redactor::redact_json(const Json& value) { + const std::string redacted { redact(value.dump(-1, ' ', false, Json::error_handler_t::replace)) }; + Json parsed = Json::parse(redacted, nullptr, false); + // Never expected: no placeholder has a quote or a backslash, and no rule ends inside an escape. + // Should it happen, the text is kept rather than the original. + return parsed.is_discarded() ? Json(redacted) : parsed; +} + +std::vector Redactor::residue(std::string_view text, std::size_t limit) const { + const Impl& impl { *impl_ }; + std::vector found; + const auto add = [&](std::string_view rule, std::size_t offset) { + if (found.size() < limit) found.push_back(Residue { std::string { rule }, offset }); + }; + for (const auto& pattern : impl.homePatterns) { + if (const auto at = find_path(text, pattern)) add(RULE_HOME, *at); + } + // The home as a plain string, whatever follows it: stricter than the rule that replaced it, so a + // spelling that rule does not know is found here rather than shipped. + for (const auto& home : impl.homeTexts) { + if (const auto at = ifind(text, home); at != std::string_view::npos) add(RULE_HOME, at); + } + for (const auto& pattern : impl.workspacePatterns) { + if (const auto at = find_path(text, pattern)) add(RULE_WORKSPACE, *at); + } + for (const auto& prefix : impl.shortNames) { + for (std::size_t at { ifind(text, prefix) }; at != std::string_view::npos; at = ifind(text, prefix, at + 1)) { + if (match_short_name(text, at, prefix) > 0) { + add(RULE_SHORT_NAME, at); + break; + } + } + } + for (const auto& user : impl.users) { + const bool distinctive { distinctive_name(user) }; + for (std::size_t at { ifind(text, user) }; at != std::string_view::npos; at = ifind(text, user, at + 1)) { + if (distinctive ? name_word_at(text, at, user.size()) : directory_name_at(text, at, user.size())) { + add(RULE_USER, at); + break; + } + } + } + for (const auto& host : impl.hosts) { + if (!distinctive_name(host)) continue; + for (std::size_t at { ifind(text, host) }; at != std::string_view::npos; at = ifind(text, host, at + 1)) { + if (name_word_at(text, at, host.size())) { + add(RULE_HOST, at); + break; + } + } + } + std::vector tokens; + find_prefixed_tokens(text, tokens); + for (const auto& token : tokens) add(RULE_SECRET, token.begin); + std::ranges::sort(found, {}, &Residue::offset); + return found; +} + +} // namespace mcppls::bundle diff --git a/src/bundle/redact.cppm b/src/bundle/redact.cppm new file mode 100644 index 0000000..4d29a95 --- /dev/null +++ b/src/bundle/redact.cppm @@ -0,0 +1,75 @@ +// What leaves this machine in a report or a diagnostic bundle (issue #23 fix plan F18): the user's +// home directory, the user's and the machine's names and anything that looks like a secret are +// replaced by placeholders, the same original always by the same placeholder so that paths stay +// comparable, and a check afterwards finds whatever is left. The mapping from placeholder back to +// original lives only in the Redactor and is never written anywhere. +// +// Pure text in, text out: no file, no environment. Who the user is comes in as an Identity, which +// mcppls.bundle.bundle reads from the machine. +export module mcppls.bundle.redact; + +import std; +import nlohmann.json; + +export namespace mcppls::bundle { + +// Who a bundle must not name. +struct Identity { + std::vector homes; // the home directory, '/'-separated, in each spelling known (as set, canonical) + std::vector users; // the login name and the home directory's last component + std::vector hosts; // the machine's names + std::vector workspaces; // roots replaced by , , ...; empty: roots are kept +}; + +// The rules, by the id the manifest counts them under. +inline constexpr std::string_view RULE_HOME { "home-directory" }; +inline constexpr std::string_view RULE_WORKSPACE { "workspace-root" }; +inline constexpr std::string_view RULE_USER { "user-name" }; +inline constexpr std::string_view RULE_SHORT_NAME { "short-name" }; +inline constexpr std::string_view RULE_HOST { "host-name" }; +inline constexpr std::string_view RULE_SECRET { "secret" }; +inline constexpr std::string_view RULE_EMAIL { "email" }; + +// Whether a user or host name is distinctive enough to be replaced wherever it stands as a word. +// A short name ("a", "bob") or a common word or generic account ("admin", "runner", "ubuntu") is +// replaced only where it names a directory: replacing it everywhere would rewrite ordinary words, +// and a check for it everywhere would fail every bundle of that user for ever. +bool distinctive_name(std::string_view name); + +// Whether a key or variable name (camelCase, snake_case, kebab-case, UPPER_CASE) names a secret: +// token, secret, password, api key, authorization, ... +bool secret_name(std::string_view name); + +// A place where something the Identity names was found after redaction. +struct Residue { + std::string rule; + std::size_t offset { 0 }; +}; + +class Redactor { +public: + explicit Redactor(Identity identity); + ~Redactor(); + Redactor(Redactor&&) noexcept; + Redactor& operator=(Redactor&&) noexcept; + + // The text with every rule applied, counting what each replaced. + std::string redact(std::string_view text); + // A JSON value redacted through its text, parsed back; its structure is unchanged, since no + // placeholder carries a quote or a backslash. + nlohmann::json redact_json(const nlohmann::json& value); + + // What the Identity names that is still in `text`: the home directory in any spelling, the user + // and host names under the same rules as redact(), and anything that looks like a known token. + // At most `limit` places. + std::vector residue(std::string_view text, std::size_t limit = 8) const; + + // Replacements made so far, by rule id. + const std::map>& hits() const; + +private: + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace mcppls::bundle diff --git a/src/bundle/writer.cpp b/src/bundle/writer.cpp new file mode 100644 index 0000000..b838831 --- /dev/null +++ b/src/bundle/writer.cpp @@ -0,0 +1,536 @@ +module mcppls.bundle.writer; + +import std; +import nlohmann.json; +import mcppls.os; +import mcppls.base.log; +import mcppls.base.path; +import mcppls.base.sha256; +import mcppls.base.text; +import mcppls.base.version; +import mcppls.platform.dirs; +import mcppls.platform.env; +import mcppls.platform.fs; +import mcppls.platform.process; +import mcppls.platform.toolrun; +import mcppls.bundle.identity; +import mcppls.bundle.redact; +import mcppls.bundle.zip; + +namespace mcppls::bundle { + +namespace { + +using Json = nlohmann::json; +namespace log = base::log; + +// What goes first when the bundle would be larger than its cap: the report and the environment +// always, then the incidents, the engine, the client's log, the server's logs newest first, dumps last. +enum class Kind { report, environment, incident, engine, client_log, log, dump }; + +struct Candidate { + std::string name; // in the archive + std::string content; + Kind kind { Kind::report }; + bool text { true }; // redacted and checked; a dump is neither + bool truncated { false }; + std::string note; +}; + +constexpr std::size_t KIB { 1024 }; +constexpr std::size_t MIB { 1024 * KIB }; +constexpr std::size_t LOG_FILE_CAP { 2 * MIB }; +constexpr std::size_t LOG_HEAD { 256 * KIB }; +constexpr std::size_t INCIDENT_FILE_CAP { 4 * MIB }; +constexpr std::size_t DATABASE_CAP { 8 * MIB }; +constexpr std::size_t PROBES_CAP { 1 * MIB }; +constexpr std::size_t CLIENT_LOG_CAP { 2 * MIB }; +constexpr std::uint64_t MANIFEST_RESERVE { 256 * KIB }; +constexpr std::size_t LOG_SESSIONS { 3 }; // the last sessions, whenever they were +constexpr std::size_t LOG_SESSIONS_MAX { 6 }; // and any other of the last day, up to this many in all + +std::string dump(const Json& value) { return value.dump(2, ' ', false, Json::error_handler_t::replace); } + +std::string utc_now(std::string_view format) { + const auto now = std::chrono::floor(std::chrono::system_clock::now()); + return std::vformat(format, std::make_format_args(now)); +} + +bool binary(std::string_view content) { return content.substr(0, std::min(content.size(), 8 * KIB)).contains('\0'); } + +// The first and the last of a long text, with a line that says how much was left out. +std::string head_and_tail(std::string content, std::size_t cap, bool& truncated) { + if (content.size() <= cap) return content; + truncated = true; + const std::size_t tail { cap - LOG_HEAD }; + std::string kept { content.substr(0, LOG_HEAD) }; + kept += std::format("\n[... mcppls left out {} bytes here to keep the bundle small ...]\n", content.size() - LOG_HEAD - tail); + kept += content.substr(content.size() - tail); + return kept; +} + +std::optional read_text(std::string_view path) { + auto content = platform::fs::read_file(path); + if (!content) return std::nullopt; + return std::move(*content); +} + +// A system's own answer about itself (a version, a code page), bounded and in this process's own +// environment: never the login shell, never the network. +std::optional system_answer(std::string_view program, std::vector arguments) { + const auto executable = platform::env::find_executable(program); + if (!executable) return std::nullopt; + auto result = platform::toolrun::run(platform::toolrun::Request { + .program = *executable, + .arguments = std::move(arguments), + .workDirectory = platform::dirs::temp_directory(), + .purpose = "bundle", + .bounds = platform::RunBounds { .hard = std::chrono::seconds { 5 } }, + .environment = platform::env::variables(), + }); + if (!result || result->exitCode != 0 || result->timedOut) return std::nullopt; + return std::move(result->output); +} + +// "Key:\tvalue" lines, the way sw_vers answers. +std::string field(std::string_view text, std::string_view key) { + for (const auto line : base::split_lines(text)) { + const auto trimmed = base::trim(line); + if (trimmed.starts_with(key)) { + auto rest = trimmed.substr(key.size()); + if (!rest.empty() && rest.front() == ':') rest.remove_prefix(1); + return std::string { base::trim(rest) }; + } + } + return {}; +} + +std::string unquoted(std::string_view value) { + value = base::trim(value); + if (value.size() >= 2 && (value.front() == '"' || value.front() == '\'') && value.back() == value.front()) value = value.substr(1, value.size() - 2); + return std::string { value }; +} + +Json operating_system() { + Json os { { "platform", std::string { mcppls::os::PLATFORM } }, { "family", std::string { mcppls::os::FAMILY_NAME } } }; + if constexpr (mcppls::os::FAMILY == mcppls::os::Family::linux) { + if (auto release = read_text("/proc/sys/kernel/osrelease")) os["kernel"] = std::string { base::trim(*release) }; + if (auto release = read_text("/etc/os-release")) { + for (const auto line : base::split_lines(*release)) { + if (line.starts_with("PRETTY_NAME=")) os["distribution"] = unquoted(line.substr(12)); + } + } + } else if constexpr (mcppls::os::FAMILY == mcppls::os::Family::macos) { + if (auto versions = system_answer("sw_vers", {})) { + os["name"] = field(*versions, "ProductName"); + os["version"] = field(*versions, "ProductVersion"); + os["build"] = field(*versions, "BuildVersion"); + } + } else { + if (auto version = system_answer("cmd", { "/d", "/c", "ver" })) os["version"] = std::string { base::trim(*version) }; + } + return os; +} + +Json memory() { + if constexpr (mcppls::os::FAMILY == mcppls::os::Family::linux) { + if (auto info = read_text("/proc/meminfo")) { + for (const auto line : base::split_lines(*info)) { + if (!line.starts_with("MemTotal:")) continue; + std::uint64_t kib { 0 }; + const auto digits = base::trim(line.substr(9)); + (void)std::from_chars(digits.data(), digits.data() + digits.size(), kib); + return Json { { "totalBytes", kib * 1024 } }; + } + } + } else if constexpr (mcppls::os::FAMILY == mcppls::os::Family::macos) { + if (auto bytes = system_answer("sysctl", { "-n", "hw.memsize" })) { + std::uint64_t total { 0 }; + const auto digits = base::trim(*bytes); + (void)std::from_chars(digits.data(), digits.data() + digits.size(), total); + return Json { { "totalBytes", total } }; + } + } + return nullptr; +} + +Json locale() { + Json locale = Json::object(); + for (const std::string_view name : { "LANG", "LC_ALL", "LC_CTYPE" }) { + if (auto value = platform::env::get(name)) locale[std::string { name }] = *value; + } + if constexpr (mcppls::os::FAMILY == mcppls::os::Family::windows) { + // The ANSI code page is what a narrow string from the system is in (issue #23 had a GBK machine). + if (auto answer = system_answer("reg", { "query", "HKLM\\SYSTEM\\CurrentControlSet\\Control\\Nls\\CodePage", "/v", "ACP" })) { + for (const auto line : base::split_lines(*answer)) { + const auto words = base::split(base::trim(line), ' '); + std::vector parts; + for (const auto word : words) { + if (!base::trim(word).empty()) parts.push_back(base::trim(word)); + } + if (parts.size() >= 3 && parts.front() == "ACP") locale["ansiCodePage"] = std::string { parts.back() }; + } + } + } + return locale; +} + +// Only these: a variable can carry a credential, and nothing else here needs one. +bool whitelisted_variable(std::string_view name) { + const std::string upper { [&] { + std::string out { name }; + for (char& c : out) c = static_cast(std::toupper(static_cast(c))); + return out; + }() }; + return upper.starts_with("MCPP_") || upper.starts_with("XLINGS_") || upper.starts_with("LC_") || upper == "LANG" || upper == "PATH"; +} + +Json environment_variables() { + Json variables = Json::object(); + for (const auto& entry : platform::env::variables()) { + const std::size_t equals { entry.find('=', 1) }; + if (equals == std::string::npos) continue; + const std::string name { entry.substr(0, equals) }; + if (whitelisted_variable(name)) variables[name] = entry.substr(equals + 1); + } + return variables; +} + +Json environment(const BundleInput& input) { + const Json& report { input.report }; + Json versions { { "mcppls", std::string { base::VERSION } } }; + if (const auto payload = report.find("payload"); payload != report.end() && payload->is_object()) { + versions["clangd"] = payload->value("clangdVersion", std::string {}); + } + Json producers = Json::array(); + if (const auto roots = report.find("roots"); roots != report.end() && roots->is_array()) { + for (const auto& root : *roots) { + const Json* project { root.contains("project") && root["project"].is_object() ? &root["project"] : nullptr }; + if (project == nullptr) continue; + producers.push_back(Json { { "producer", project->value("producer", std::string {}) }, { "version", project->value("producerVersion", std::string {}) } }); + } + } + versions["producers"] = std::move(producers); + Json payload = nullptr; + const std::string payloadDirectory { report.contains("payload") && report["payload"].is_object() ? report["payload"].value("directory", std::string {}) : std::string {} }; + if (!payloadDirectory.empty()) { + if (auto text = read_text(base::join_path(payloadDirectory, "payload.json"))) { + payload = Json::parse(*text, nullptr, false); + if (payload.is_discarded()) payload = nullptr; + } + } + Json probes = nullptr; + if (auto text = read_text(base::join_path(platform::dirs::cache_directory(), "toolchains/probe.json")); text && text->size() <= PROBES_CAP) { + probes = Json::parse(*text, nullptr, false); + if (probes.is_discarded()) probes = nullptr; + } + return Json { + { "generatedAt", utc_now("{:%FT%TZ}") }, + { "os", operating_system() }, + { "cpu", Json { { "logicalProcessors", std::thread::hardware_concurrency() } } }, + { "memory", memory() }, + { "locale", locale() }, + { "editor", report.value("client", Json(nullptr)) }, + { "client", input.client }, + { "initializationOptions", input.initializationOptions }, + { "server", report.value("server", Json(nullptr)) }, + { "payload", payload }, + { "versions", std::move(versions) }, + { "toolchainProbes", std::move(probes) }, + { "environmentVariables", environment_variables() }, + }; +} + +// server---.log and its rotations (.log.1, .log.2): the time it started. +std::optional session_start(std::string_view name) { + static constexpr std::string_view PREFIX { "server-" }; + if (!name.starts_with(PREFIX) || name.size() < PREFIX.size() + 15) return std::nullopt; + const std::string_view stamp { name.substr(PREFIX.size(), 15) }; + int year { 0 }, month { 0 }, day { 0 }, hour { 0 }, minute { 0 }, second { 0 }; + const auto number = [&](std::size_t at, std::size_t length, int& out) { + return std::from_chars(stamp.data() + at, stamp.data() + at + length, out).ec == std::errc {}; + }; + if (stamp[8] != '-' || !number(0, 4, year) || !number(4, 2, month) || !number(6, 2, day) || !number(9, 2, hour) || !number(11, 2, minute) + || !number(13, 2, second)) { + return std::nullopt; + } + const std::chrono::year_month_day date { std::chrono::year { year }, std::chrono::month { static_cast(month) }, + std::chrono::day { static_cast(day) } }; + if (!date.ok()) return std::nullopt; + return std::chrono::sys_days { date } + std::chrono::hours { hour } + std::chrono::minutes { minute } + std::chrono::seconds { second }; +} + +// The server's logs worth reading: the last LOG_SESSIONS sessions, and any other of the last day, +// up to LOG_SESSIONS_MAX in all, each with its rotations, newest first. +std::vector recent_logs(std::string_view directory, std::string_view currentLog) { + std::map, std::greater<>> sessions; // "server-