#| highlighting

[vezwork]

Fixes #409

• Adds semantic highlighting for hash-pipe comments in code blocks
• Yaml validation already existed, but makes it so yaml validation (in both frontmatter and in hash-pipe comments) happens as the user types rather than on-save.

Design considerations

• note: this feature uses semantic tokens for highlighting
  • code was added in this PR to merge Python/R/etc. LSP semantic highlighting with hash-pipe semantic highlighting
  • According to an LLM, the yaml frontmatter highlighting in qmd files is done by "declarative grammar delegation". This approach is ideal because it delegates highlighting of the yaml in frontmatters to the yaml textmate grammar in VSCode (so its consistent with how yaml would be highlight in a .yaml file). I would've liked to use this approach for hash-pipe comments, but it does not sound like it possible, since the hash-pipe comments break up the yaml and it sounds like delegation requires a contiguous block of yaml.

TODO

• test in VSCode
• test in Positron

[posit-snyk-bot]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[vezwork]

@mine-cetinkaya-rundel sent me her mock-ups of this UI:

I'm gonna now see if I can do something like this.

[vezwork]

Got it working

[vezwork]

@juliasilge I see you added "Test packages" in #1063 and its failing for a reason I don't understand; something about Package libsecret-1.

edit: it seems to be a flake across all tests. It now passes after re-running on this PR. Its is failing in the "Build vscode extension" step of the Quarto pre-release test in my other PR: https://github.com/quarto-dev/quarto/actions/runs/31631041559/job/94229555095?pr=1086

[juliasilge]

This is so nice 🤩 and the offset math between the assembled YAML and the document positions looks correct to me! I tried block scalars, flow collections, comments, anchors, duplicate keys, and tabs, and the positions stayed correct. I have some inline comments, plus three general points below.

• The CHANGELOG covers only half of this PR. The entry speaks about semantic highlighting, but this branch also moves yaml validation to the pull path. Diagnostics now show while the user types, and before they showed on open and on save. This is arguably the more visible half of the change, so let's document it.

• Live validation is a decision about the user experience. A half-typed line such as #| echo: is now linted during the edit. Did you look at how much noise this makes? Errors that come and go during typing can look like a defect.

• There are no tests. hashPipeYaml and emitYamlTokens are pure functions that we could write some unit tests for. In apps/vscode/src/test/semanticTokens.test.ts, we already test the encode, decode, and remap helpers next to them, so that seems like a good place for these tests.

[juliasilge]

This code creates the promise before the cancellation check on line 216. The early return on line 217 does not await it. If quarto.getYamlDiagnostics rejects, the rejection is unhandled. apps/lsp/src/index.ts does not install an unhandledRejection handler. As a result, IIUC it terminates the server process.

Could we do something like this instead?

const yamlDiagnostics = provideYamlDiagnostics(this.#quarto, doc)
  .catch(() => []);

[juliasilge]

This computation is inside the withVirtualDocUri callback, so three earlier returns never reach it. Lines 201 and 217 return await next(document, token), and line 193 does the same for documents that are not Quarto documents. On those paths the #| cell option tokens are lost.

Two cases reach line 201 with a visible .qmd file:

• Two .qmd files are open side by side. Only one of them is window.activeTextEditor.document.
• A webview has focus (preview or visual editor), so window.activeTextEditor is undefined.

In each case the visible document loses its cell option colors after the next tokenization pass. The colors come back when the user edits the document while it has focus.

To make the behavior the same on all paths, could we move this line to the top of the provider, before the early returns? Then merge the entries into the result of next(), in the same way as line 263.

[juliasilge]

Related: provideDocumentSemanticTokens in hash-pipe-yaml.ts:91 returns builder.build() even when there are no tokens. An empty result is not null, so it can win the document over this provider if the registration order changes. Can we return null when there are no tokens?

[juliasilge]

This pattern differs from optionCommentPattern in packages/core/src/jupyter/options.ts:66-68, which builds /^#\s*\| ?/.

Could we use optionCommentPattern here, so that the highlighting agrees with what Quarto parses?

[juliasilge]

isWholeLine: true and a bottom border do not compose the way this code expects. A #| line that is long enough to soft-wrap gets one rule under each of its visual rows, not one rule under the whole line:

This is because of the mechanism in VS Code itself, so I don't think we can code around it here very easily.

I think I prefer to remove the separator and let the darker background show the boundary. The background already wraps correctly, and it is the only option here that is always correct. If you want to keep a rule, we could put it on the first line after the option block as a top border (borderWidth: "1px 0 0 0").

[vezwork]

I figured out how to keep the bottom line but fix this problem in #1092 with a css trick.

[juliasilge]

These five scopes all resolve to fully saturated code colors, so a cell option block really competes with the code below it. I think the highlighting needs to be quieter, because the background already shows where the block is.

I resolved the scopes against the default themes in the Positron repo:

Token	Light+ / Positron Light	Dark+ / Positron Dark
quartoYamlKey	#800000	#569cd6
quartoYamlString	#0000ff	#ce9178
quartoYamlNumber	#098658	#b5cea8
quartoYamlBoolean	#0000ff	#569cd6
(comment, for comparison)	#008000	#6A9955

Two problems come out of this.

string.unquoted.plain.out.yaml is pure blue in the light themes. Light+ and Positron Light have a rule for this exact scope, which gives #0000ff. This is the most saturated color on the screen. The scope is also incorrect for quoted values, because it is the scope for plain scalars. Use string, which gives #a31515 and #ce9178, the same as strings in the code.

One line can show three different hues. #| fig-cap: "..." gives a maroon key and a blue value. #| echo: false gives maroon and blue. #| fig-width: 6 gives maroon and green. Five token types is more detail than cell options need, because almost all of them are one key and one short scalar.

My suggestion:

• Collapse the five types to two: quartoYamlKey and quartoYamlValue. Then remove quartoYamlNumber, quartoYamlBoolean, and quartoYamlNull.
• Map quartoYamlValue to comment, so values keep the comment color of the theme. Keep one accent color on the key.

This gives one accent per line instead of three. If you want the quietest result, map both types to comment and let the background do all of the work.

Note that the legend in apps/quarto-utils/src/semantic-tokens-legend.ts says to append only, so removals there need a check of the index order.

[vezwork]

Ah, nice catch. Looking into the mismatching highlighting you pointed out lead to me trying out this alternative approach #1092, which doesn't have mismatching highlighting, but might be a bit visually loud.

[vezwork]

@julia what do you think of this alternative approach #1092? I originally thought it wouldn't work but it actually does work well, and it is much simpler.

edit: I just updated the alternative PR #1092 to also have background decorations, and IMO it is simpler and better. LMK what you think though.