Skip to content

sea: generate a compile cache for the VFS modules - #66768

Merged
nodejs-github-bot merged 1 commit into
nodejs:mainfrom
mcollina:sea-vfs-code-cache
Nov 9, 2026
Merged

nodejs-github-bot merged 1 commit into
nodejs:mainfrom
mcollina:sea-vfs-code-cache

Conversation

@mcollina

@mcollina mcollina commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

"useCodeCache": true is currently rejected together with "useVfs": true. The reason is a plumbing gap rather than a V8 limitation: the SEA code cache is only consumed by the embedder compile paths (compileFunctionForCJSLoader(..., is_sea_main=true) and the kEmbedder branch in module_wrap.cc), while a main script in the VFS runs through the regular CJS/ESM loaders. A main-only cache would also be of little use for a VFS bundle, where the dependency graph is what costs at startup.

This PR implements the build-time half of the fix:

  • When both flags are set, --build-sea / --experimental-sea-config spins up a Node.js instance (CommonEnvironmentSetup + LoadEnvironment), mounts the assets (or the vfsArchive) exactly like the runtime does, and compiles every .js/.cjs/.mjs (and .ts/.cts/.mts with --strip-types) module in the mount without executing it, via wrapSafe / compileSourceTextModule, with a portable CompileCacheHandler whose keys are relative to the mount point.
  • The resulting entries are serialized in the compile cache file layout and embedded as reserved node:sea:compile-cache/<key> assets. The SEA provider hides node:sea: keys so they don't show up as files in the mount. Modules that fail to compile are skipped.
  • CompileCacheHandler gains a key_root for portable caches and SerializeEntries(), which shares its serialization with Persist(). The internal enableCompileCache binding takes the key root as a 4th argument.
  • The build instance runs with a cloned environment where NODE_COMPILE_CACHE* is removed, so the user's compile cache settings cannot hijack the build.

Reading the embedded entries at startup (a read-only, blob-backed CompileCacheHandler enabled from initSeaVfs, including in workers under vfsArchive) is left to a follow-up PR, which will also decide whether the main-only code cache blob still needs to be generated under VFS.

Tests: test/sea/test-single-executable-application-vfs-code-cache.js checks the number of embedded entries for the plain, ESM and ZIP fixtures (5/4/3), that NODE_COMPILE_CACHE / NODE_DISABLE_COMPILE_CACHE in the build environment do not affect the result, and that the built executable still runs. The rejection case was removed from test-build-sea-vfs-incompatible-options.js. The existing VFS SEA tests and test/parallel/test-compile-cache* pass.

—-

AI generated, reviewed by me

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/single-executable

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. single-executable Issues and PRs related to single-executable applications. vfs Issues and PRs related to the virtual filesystem subsystem. labels Oct 15, 2026
@mcollina
mcollina force-pushed the sea-vfs-code-cache branch 2 times, most recently from a35d240 to 5ee0908 Compare October 15, 2026 15:58
@mcollina
mcollina force-pushed the sea-vfs-code-cache branch 2 times, most recently from 942338e to 5d06ff8 Compare October 27, 2026 16:03
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.02902% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.67%. Comparing base (3b209b2) to head (cadb162).
⚠️ Report is 42 commits behind head on main.

Files with missing lines Patch % Lines
src/node_sea.cc 81.52% 8 Missing and 9 partials ⚠️
lib/internal/vfs/sea_compile_cache.js 93.58% 14 Missing and 1 partial ⚠️
src/compile_cache.cc 94.44% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #66768      +/-   ##
==========================================
- Coverage   90.68%   90.67%   -0.02%     
==========================================
  Files         797      798       +1     
  Lines      279367   279961     +594     
  Branches    53737    53845     +108     
==========================================
+ Hits       253344   253849     +505     
- Misses      16319    16394      +75     
- Partials     9704     9718      +14     
Files with missing lines Coverage Δ
lib/internal/vfs/providers/sea.js 100.00% <100.00%> (ø)
lib/internal/vfs/sea.js 100.00% <100.00%> (ø)
src/compile_cache.h 100.00% <ø> (ø)
src/env.cc 84.43% <100.00%> (-0.14%) ⬇️
src/env.h 97.40% <ø> (ø)
src/node_modules.cc 79.91% <100.00%> (+0.15%) ⬆️
src/compile_cache.cc 81.97% <94.44%> (+1.02%) ⬆️
lib/internal/vfs/sea_compile_cache.js 93.58% <93.58%> (ø)
src/node_sea.cc 87.81% <81.52%> (+0.08%) ⬆️

... and 64 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 28, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 28, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@mcollina mcollina added the resume-ci Add this label to resume the latest eligible Jenkins CI run on a PR with an approving review. label Oct 31, 2026
@github-actions github-actions Bot added resume-ci-failed Resuming CI with the resume-ci label failed and requires manual intervention. and removed resume-ci Add this label to resume the latest eligible Jenkins CI run on a PR with an approving review. labels Oct 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Failed to resume CI

   ✖  test/sea/test-single-executable-application-vfs-code-cache.js
   ℹ  https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win10-COMPILED_BY-vs2022_clang/44172/consoleText
✖  Refusing to resume CI: failures reference files changed by this PR
Full Auto Start CI output
�[36m⠋�[39m Validating Jenkins credentials
�[36m⠋�[39m Validating Jenkins credentials
✔  Jenkins credentials valid
�[36m⠙�[39m Looking for CI runs for pull request 66768
�[36m⠙�[39m Looking for CI runs for pull request 66768
�[36m⠙�[39m Getting PR from nodejs/node/pull/66768
�[36m⠙�[39m Getting reviews from nodejs/node/pull/66768
�[36m⠙�[39m Getting comments from nodejs/node/pull/66768
✔  Found PR CI job 78696
�[36m⠹�[39m Querying data for job/node-test-pull-request/78696/
�[36m⠹�[39m Querying data for job/node-test-pull-request/78696/
�[36m⠹�[39m Querying API for job/node-test-pull-request/78696/
✔  Build data downloaded
�[36m⠹�[39m Checking whether PR CI job 78696 can be resumed
�[36m⠹�[39m Checking whether PR CI job 78696 can be resumed
✔  Jenkins offers a Resume build action
�[36m⠹�[39m Checking failures against changed PR files
�[36m⠹�[39m Checking failures against changed PR files
�[36m⠸�[39m Querying data for job/node-test-pull-request/78696/
�[36m⠸�[39m Querying API for job/node-test-pull-request/78696/
✔  Build data downloaded
�[36m⠸�[39m Querying failures of job/node-test-commit/93624/
�[36m⠸�[39m Querying failures of job/node-test-commit/93624/
�[36m⠸�[39m Querying API for job/node-test-commit-windows-fanned/81480/
�[36m⠼�[39m Querying API for job/node-test-binary-windows-js-suites/44172/
�[36m⠼�[39m Querying API for job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win10-COMPILED_BY-vs2022_clang/44172/
�[36m⠼�[39m Querying API for job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win11-COMPILED_BY-vs2022_clang/44172/
�[36m⠼�[39m Querying API for job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win2019-COMPILED_BY-vs2022_clang/44172/
�[36m⠼�[39m Querying API for job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win2022-COMPILED_BY-vs2022_clang/44172/
�[36m⠴�[39m Querying console text for job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win10-COMPILED_BY-vs2022_clang/44172/
�[36m⠴�[39m Querying console text for job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win11-COMPILED_BY-vs2022_clang/44172/
�[36m⠴�[39m Querying console text for job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win2019-COMPILED_BY-vs2022_clang/44172/
�[36m⠴�[39m Querying console text for job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win2022-COMPILED_BY-vs2022_clang/44172/
✔  Data downloaded
   ✖  test/sea/test-single-executable-application-vfs-code-cache.js
   ℹ  https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win10-COMPILED_BY-vs2022_clang/44172/consoleText
not ok 1431 sea/test-single-executable-application-vfs-code-cache
  ---
  duration_ms: 6725.23600
  severity: fail
  exitcode: 1
  stack: |-
    1..0 # Skipped: Cannot find signtool: Error: - process terminated with status 1, expected 0
        at signSEA (c:\workspace\node-test-binary-windows-js-suites\node\test\common\sea.js:220:7)
        at buildSEA (c:\workspace\node-test-binary-windows-js-suites\node\test\common\sea.js:132:3)
        at main (c:\workspace\node-test-binary-windows-js-suites\node\test\sea\test-single-executable-application-vfs-code-cache.js:104:22)
        at process.processTicksAndRejections (node:internal/process/task_queues:104:5) {
      command: 'where signtool'
    }
    Mismatched noop function calls. Expected exactly 1, actual 0.
        at Proxy.mustCall (c:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:470:10)
        at Object.<anonymous> (c:\workspace\node-test-binary-windows-js-suites\node\test\sea\test-single-executable-application-vfs-code-cache.js:128:20)
        at Module._compile (node:internal/modules/cjs/loader:1968:14)
        at Object..js (node:internal/modules/cjs/loader:2108:10)
        at Module.load (node:internal/modules/cjs/loader:1690:32)
        at Module._load (node:internal/modules/cjs/loader:1480:12)
        at wrapModuleLoad (node:internal/modules/cjs/loader:261:19)
        at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:173:5)
        at node:internal/main/run_main_module:33:47
    (node:1932) ExperimentalWarning: The zlib ZIP archive API is an experimental feature and might change at any time
    (Use `node --trace-warnings ...` to show where the warning was created)
    [process 2576]: --- stderr ---
    INFO: Could not find files for the given pattern(s).
    
    [process 2576]: --- stdout ---
    
    [process 2576]: status = 1, signal = null
  ...
✖  Refusing to resume CI: failures reference files changed by this PR

View workflow run

@mcollina mcollina added request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. and removed resume-ci-failed Resuming CI with the resume-ci label failed and requires manual intervention. labels Nov 1, 2026
@github-actions github-actions Bot added request-ci-failed Starting CI with the request-ci label failed and requires manual intervention. and removed request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. labels Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Failed to start CI

   ℹ  Existing CI run found: https://ci.nodejs.org/job/node-test-pull-request/78696/
   ✖  Refusing to start a potentially duplicate CI job. Resume CI by adding the "resume-ci" label to the PR, or run: ncu-ci resume https://github.com/nodejs/node/pull/66768
Full Auto Start CI output
�[36m⠋�[39m Getting reviews from nodejs/node/pull/66768
�[36m⠋�[39m Getting commits from nodejs/node/pull/66768
�[36m⠙�[39m Validating Jenkins credentials
�[36m⠙�[39m Validating Jenkins credentials
✔  Jenkins credentials valid
�[36m⠹�[39m Getting comments from nodejs/node/pull/66768
�[36m⠸�[39m Querying data for job/node-test-pull-request/78696/
�[36m⠸�[39m Querying data for job/node-test-pull-request/78696/
�[36m⠸�[39m Querying API for job/node-test-pull-request/78696/
✔  Build data downloaded
   ℹ  Existing CI run found: https://ci.nodejs.org/job/node-test-pull-request/78696/
   ✖  Refusing to start a potentially duplicate CI job. Resume CI by adding the "resume-ci" label to the PR, or run: ncu-ci resume https://github.com/nodejs/node/pull/66768

View workflow run

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@mcollina mcollina added request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. and removed request-ci-failed Starting CI with the request-ci label failed and requires manual intervention. labels Nov 4, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Nov 4, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Nov 5, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Nov 5, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

"useCodeCache" was rejected together with "useVfs" because the SEA code
cache is only consumed by the embedder compile paths, while a main
script in the VFS runs through the regular loaders and a main-only cache
would not cover the modules it depends on anyway.

Compile every JavaScript (and, with --strip-types, TypeScript) module in
the VFS when the preparation blob is generated, without executing them,
through a portable CompileCacheHandler keyed by the path relative to the
mount point. The entries are serialized in the cache file layout and
embedded as reserved node:sea:compile-cache/<key> assets, which the SEA
provider hides from the mount. Modules that fail to compile are skipped.

CompileCacheHandler gains a key root for portable caches and in-memory
serialization of its entries, shared with Persist(). The build runs in a
Node.js instance whose environment ignores NODE_COMPILE_CACHE.

Reading the embedded entries at startup is left to a follow-up.

Tests skip up front when building a single executable application on
Windows without signtool, so the async VFS code cache test is skipped
cleanly instead of failing on an unmatched mustCall.

Signed-off-by: Matteo Collina <hello@matteocollina.com>
Assisted-by: pi
@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Nov 8, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Nov 8, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@mcollina mcollina added commit-queue PRs queued for automated landing through the Commit Queue. semver-minor PRs that contain new features and should be released in the next minor version. labels Nov 9, 2026
@nodejs-github-bot
nodejs-github-bot merged commit 5d59de8 into nodejs:main Nov 9, 2026
68 checks passed
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Landed in 5d59de8

@nodejs-github-bot nodejs-github-bot removed the commit-queue PRs queued for automated landing through the Commit Queue. label Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. semver-minor PRs that contain new features and should be released in the next minor version. single-executable Issues and PRs related to single-executable applications. vfs Issues and PRs related to the virtual filesystem subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants