feat(cli): tell the user when their sim is out of date by mzxchandra · Pull Request #7420 · simstudioai/sim

mzxchandra · 2026-09-02T22:06:14Z

Summary

sim tools execute shipped in 2.1.5. Someone on 2.1.2 looking for it saw a help listing without it and concluded the CLI could not do it. A missing subcommand is indistinguishable from a feature that was never built, and nothing in the CLI could tell them otherwise: there was no update check, no client/server version negotiation, and no way to learn what "current" is.

The notifier (feat(cli): tell the user when their sim is out of date)

Once a day, at an interactive terminal, a root preAction hook asks registry.npmjs.org for the dist-tags of the channel it was installed from and prints one line on stderr:

Update available: sim 2.1.2 → 2.1.5. Run: npm install -g sim@latest

preAction, not a teardown. Commander answers --help and --version during parsing, so the two latency-sensitive invocations are excluded by construction rather than by a check; and some commands call process.exit directly, where a finally would never run.
Silent by default. Nothing prints when stderr is not a terminal, in CI (five variables), under npx, from a checkout, or on a prerelease install. SIM_NO_UPDATE_CHECK=1 turns it off.
The checkout and prerelease guards are not politeness. The repo manifest trails npm permanently by design — the publish workflow bumps the version in-job under permissions: contents: read and never commits it back — so without the checkout guard every engineer here would be told daily to upgrade to a version their own tree already contains. And staging publishes on every push, so advising a prerelease user would be stale within the hour.
Comparison is channel-scoped, which makes "upgrade" to an older stable version structurally impossible rather than merely guarded against. The comparator implements semver precedence including the numeric prerelease rule (preview.9 precedes preview.44, which a string comparison gets backwards). No semver dependency: the package bundles to a single file, and this is +7.3KB (+1.2%) with no new dep.
Timeout is a hard 1s, not SIM_TIMEOUT_SECONDS (which defaults to an hour and governs work the user actually asked for). The check is stamped whether or not it succeeds, so a blackholed registry costs one second a day rather than one per command.

Review fixes (fix(cli): close the update-notifier findings from pre-landing review)

Body read under a 64KB budget instead of buffering whatever a mirror sends; redirect: 'error'; the registry's answer is parsed before it is persisted, so nothing unvalidated reaches disk.
A configured mirror's own path and query are preserved — new URL(relative, base) discards both, so a token-authenticated Artifactory/Nexus base was being rewritten into a request the mirror answers with a 404.
One normalisation for every module-path decision (separators and case), so a Windows or case-insensitive checkout isn't read as a global install by one guard and a checkout by the other.

Privacy

This is the first non-Sim host the CLI has ever contacted, in a tool with no telemetry. Three mitigations, stated up front:

The User-Agent is reduced to sim-cli/<version> — deliberately not the exported USER_AGENT, which also carries node version, platform and arch. This is now an assertion, not a comment, so a future "DRY up the user agent" refactor cannot silently start sending it.
No credential, workspace id, endpoint or command name is reachable from the request. The path is a constant with no interpolation and no query string.
Documented in the README and the CLI configuration guide, with a documented off switch.

Test Coverage

Tests: 843 → 861 (+18). An independent coverage audit measured 65% weighted branch coverage on the first pass; the gaps it and the testing specialist named are what the +18 close.

The more useful number: 10 mutations applied, 10 killed. Each new assertion was verified to actually fail when the thing it guards is broken.

Mutation	Result
delete the `preAction` hook	killed
default writer → stdout	killed
flip `comparePrerelease` empty-list arm	killed
drop `redirect: 'error'`	killed
drop `AbortSignal.timeout`	killed
swap reduced UA → full `USER_AGENT`	killed
drop the content-length cap	killed
mirror path via relative URL	killed
drop `CI` from `CI_VARIABLES`	killed
invert ASCII prerelease ordering	killed

Before these, three tests could not fail: deleting the hook, switching the default writer to stdout, and flipping the empty-prerelease arm all left the suite green. The stdout one was vacuous because the test helper always injected a writer — so the single safety property this feature claims (never touch stdout) was unprotected.

One honest limitation: the hook test asserts registration rather than a resulting request, because the check suppresses itself when running from a checkout — and inside the suite import.meta.url is a checkout, so the behavioural path is unreachable there by construction. It is covered directly in check.test.ts and walked against the real registry from a staged global install (below).

Pre-Landing Review

22 findings from 4 specialists + a coverage audit + a plan audit. 14 fixed, 3 declined with reasons, rest were confirmations.

Declined:

~10s lingering-socket exit delay (CRITICAL, performance). Could not reproduce through the CLI. Measured 1.11–1.38s across three runs on node v23.11.0 against a blackholed IP, including a command that only sets exitCode. No node:https rewrite.
Drop announced / resetUpdateCheck (maintainability). Same shape as the existing resetEnvironmentNotices and resetRenameWarnings seams in this package.
Collapse the channel type to a boolean (maintainability). It is what a decision to notify prerelease users would extend. Its docs now say what the code does instead of describing a comparison it never performs.

Security review found no credential reachable from the request, registryBase protocol-allowlisted and failing closed on embedded userinfo, and terminal-escape injection closed by the anchored version regex (a dist-tag carrying ANSI/OSC bytes fails to parse, so the notice is suppressed rather than printed).

Design Review

No frontend files changed — design review skipped.

Eval Results

No prompt-related files changed — evals skipped.

Scope Drift

Scope Check: CLEAN. Intent: tell users when their CLI is stale. Delivered: exactly that, plus its docs. No generated artifact is touched — check:cli-docs and check:cli-api both pass unchanged, which is the proof the command tree did not move. That is why this uses an env var rather than a --no-update-check flag: a root flag would force a docs regen and permanently consume a global-flag slot, for a decision nobody makes per-invocation.

Plan Completion

31 done / 4 changed / 0 skipped, independently audited. The four "changed" are deliberate: the Unicode arrow (matching the existing SIM_DEBUG trace output in http/client.ts), a broader checkout guard than specified, CLI_VERSION imported rather than injected for the UA header, and the test cases now closed by the +18.

The companion minCliVersion floor on GET /api/v2/meta was explicitly deferred — it answers a different question ("this deployment needs a newer CLI", for self-hosted installs), and it would have been silent through the incident that prompted this work.

Verification Results

No dev server applies to a CLI. Verified by hand against the real npm registry, from a staged global install driven under a pty (a checkout is deliberately suppressed, so this is the only way to exercise the real path):

Case	Result
stale install, real registry	banner, naming the real `2.1.5`; cache written, dir 0700
second run same day	silent, zero registry requests
stdout piped, stderr still a TTY	still prints — the case that must not regress
blackholed registry (`192.0.2.1`)	silent, 1.12s, `latestVersion: null` stamped
`CI=1` / `SIM_NO_UPDATE_CHECK=1` / `--version` / `--help` / bare group command	silent, and no cache file written
non-TTY stderr	silent
prerelease install (`2.1.3-preview.44.1`)	silent, no request, no stable-version suggestion
npx layout	silent
real repo `dist/index.js`	silent (the contributor-nagging guard)

Gates: bun run test (861 passed, 1 skipped), type-check, lint:check, build (a real gate — --reject-unresolved), check:audits (45 audits), check:cli-docs unchanged.

Documentation

packages/sim-cli/README.md — SIM_NO_UPDATE_CHECK row, what the check sends, and a pointer to the canonical list.
apps/docs/content/docs/cli/configuration.mdx — the canonical "Update notices" section: env row, exactly what is sent, and every case where the notice stays quiet.
apps/docs/content/docs/cli/troubleshooting.mdx — "a documented command is missing" (the incident's own symptom) and "an update notice appears in output I am parsing".

Both .mdx files are hand-written GUIDE_PAGES, not generated output.

Test plan

bun run test in packages/sim-cli — 861 passed, 1 skipped
bun run type-check, bun run lint:check, bun run build
bun run check:audits — 45 audits
bun run check:cli-docs / check:cli-api — pass unchanged
Manual E2E against the live registry under a pty (table above)
Mutation sweep — 10/10 killed

🤖 Generated with Claude Code

`sim tools execute` shipped in 2.1.5. Someone on 2.1.2 looking for it saw a help listing without it and concluded the CLI could not do it - a missing subcommand is indistinguishable from a feature that was never built, and nothing in the CLI could tell them otherwise. It had no update check, no version negotiation, and no way to learn what "current" is. Once a day, at an interactive terminal, the root `preAction` hook asks `registry.npmjs.org` for the dist-tags of the channel it was installed from and prints one line on stderr when a newer version exists. The request carries the CLI version and nothing else - no key, no workspace, no command - and `SIM_NO_UPDATE_CHECK=1` turns it off. Everything about it fails silently, and it says nothing when stderr is not a terminal, in CI, under `npx`, from a checkout, or to a prerelease install. The last two are not politeness: the repo manifest trails npm permanently by design because the publish workflow bumps the version in-job under `permissions: contents: read` and never commits it back, so without the checkout guard every engineer here would be told daily to upgrade to a version their own tree already contains; and `staging` publishes on every push, so advising a prerelease user would be stale within the hour. Comparison is scoped to one channel, which is what makes "upgrade" to an older stable version structurally impossible rather than merely guarded against. The comparator implements semver precedence including the numeric prerelease rule - `preview.9` precedes `preview.44`, which a string comparison gets backwards. The `preAction` hook is deliberate over a teardown in the entrypoint: commander answers `--help` and `--version` during parsing, so the two latency-sensitive invocations are excluded by construction, and some commands call `process.exit` directly where a `finally` would never run. Timeout is a hard 1s rather than `SIM_TIMEOUT_SECONDS`, which defaults to an hour and governs work the user actually asked for. The check is stamped whether or not it succeeds, so a blackholed registry costs one second a day instead of one per command.

…tifier

Mutation testing found three tests that could not fail: deleting the `preAction` hook, switching the default writer to stdout, and flipping `comparePrerelease`'s empty-list arm all left the suite green. The stdout one was vacuous because the test helper always injected a writer, so the single safety property this feature claims - never touch stdout - was unprotected. The hook now has a positive test. It asserts registration rather than a resulting request, because the check suppresses itself when running from a checkout, and inside the suite `import.meta.url` IS a checkout: the behavioural path is unreachable there by construction. It is covered directly in check.test.ts and walked against the real registry from a staged global install. Security review: the response body is now read under a 64KB budget instead of buffering whatever a mirror sends, the request refuses to follow redirects, and the registry's answer is parsed before it is persisted, so nothing unvalidated reaches the disk. The reduced User-Agent was a comment; it is now an assertion, so a future "DRY up the user agent" refactor cannot silently start handing npm the user's node version, platform and arch. A configured mirror's own path and query are preserved. `new URL(relative, base)` discards both, so a token-authenticated Artifactory or Nexus base was being rewritten into a request the mirror answers with a 404. Also: one normalisation for every module-path decision (separators AND case, so a Windows or case-insensitive checkout is not read as a global install by one guard and a checkout by the other), the package name is named once rather than spelled in two unrelated places, and `delete process.env.SIM_CONFIG_DIR` in teardown - assigning `undefined` stores the literal string and leaves later tests pointed at a relative `./undefined` directory. Tests: 843 -> 861. Ten mutations applied to verify the new assertions actually fail when the thing they guard is broken; all ten killed. Declined, with reasons: the ~10s lingering-socket exit delay could not be reproduced through the CLI (measured 1.11-1.38s across three runs on node v23.11.0, including a command that only sets exitCode), so no node:https rewrite. `announced` plus `resetUpdateCheck` stays - it is the same shape as the existing resetEnvironmentNotices and resetRenameWarnings seams. The channel type stays rather than collapsing to a boolean, because it is what a decision to notify prerelease users would extend; its docs now say what the code does instead of describing a comparison it never performs.

vercel · 2026-09-02T22:06:20Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 2, 2026 10:34pm UTC

greptile-apps · 2026-09-02T22:11:26Z

Greptile Summary

The PR adds a daily, opt-out CLI update notification for interactive stable installations and documents its behavior.

Registers a root command hook that checks the selected npm registry without delaying help or version output.
Adds bounded registry response handling, version comparison, cache throttling, installation-aware upgrade commands, and suppression rules.
Updates CLI configuration and troubleshooting documentation, including an explicit disclosure for credentials embedded in private mirror URLs.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/sim-cli/src/update/check.ts	Implements the guarded, cached registry check and accurately limits configured mirror credentials to the selected origin by refusing redirects.
packages/sim-cli/src/update/semver.ts	Adds strict semantic-version parsing and precedence comparison for release-channel checks.
packages/sim-cli/src/program.ts	Registers the update notifier as a root pre-action hook while leaving parser-handled help and version invocations untouched.
apps/docs/content/docs/cli/configuration.mdx	Documents update notices, suppression conditions, caching, private mirrors, and the credential behavior identified in the prior review.
packages/sim-cli/README.md	Adds a concise update-check overview and accurately caveats credentials embedded in private registry URLs.
packages/sim-cli/src/update/check.test.ts	Covers notification, suppression, caching, request privacy, response limits, mirror URL preservation, and upgrade-command selection.

Sequence Diagram

sequenceDiagram
  participant User
  participant CLI
  participant Cache as Update cache
  participant Registry as npm registry or mirror
  User->>CLI: Run an actionable command
  CLI->>CLI: Apply TTY, CI, install, and channel guards
  CLI->>Cache: Read last-check timestamp
  alt Cache is fresh
    Cache-->>CLI: Skip update request
  else Check is due
    CLI->>Registry: Request sim dist-tags
    Registry-->>CLI: Return bounded JSON response
    CLI->>Cache: Record check result
    alt Newer stable version exists
      CLI-->>User: Print update notice to stderr
    end
  end

_{Reviews (4): Last reviewed commit: "docs(cli): name the update cache path fo..." | Re-trigger Greptile}

cubic-dev-ai

All reported issues were addressed across 10 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…rsing Review round 1: five findings, all valid. The privacy statement was too absolute. The request carries no Sim API key, but `npm_config_registry` can point at a private mirror, and a token embedded in that URL is sent with the request - it has to be, or the mirror rejects it. Both docs now say which credentials are involved and where they go: your registry's, to the host you configured, never Sim's. `parseVersion` accepted zero-padded prerelease identifiers. Semver forbids them, and accepting `2.1.3-preview.09` was worse than cosmetic: `09` failed the numeric test and fell through to being an alphanumeric identifier, and alphanumerics outrank every number, so `preview.010` sorted ABOVE `preview.2`. The file's own doc comment already claimed leading zeroes were rejected "the way the specification rejects them" - true of the release triple, not of the prerelease. Now true of both. The `--version`/`--help` test did not hold the guarantee it advertised. It watched for a request and a cache file, but neither ever appears from inside a checkout no matter what runs, because the check suppresses itself there - so it would have passed even if the hook fired, which is the exact regression it claims to prevent. It now swaps a sentinel into commander's registered preAction hooks and asserts the sentinel does not fire while parsing those two, then asserts it DOES fire for a real action command, so the negative assertion means something. No module mocking, which this package bans. The troubleshooting page hardcoded `npm install -g`, which installs a second copy under a different package manager rather than replacing the executable on PATH. It now shows all three, and says the notice already prints the one matching your install - which the notifier has always done. Tests: 861 -> 863. Both new guards mutation-checked: dropping the leading-zero rejection and deleting the hook each fail the suite.

…tifier

mzxchandra · 2026-09-02T22:18:27Z

@greptile

mzxchandra · 2026-09-02T22:18:28Z

@cubic review

cubic-dev-ai · 2026-09-02T22:18:35Z

@cubic review

@mzxchandra I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 10 files

Confidence score: 5/5

In packages/sim-cli/src/program.test.ts, assigning undefined leaves SIM_CONFIG_DIR as the truthy string 'undefined', so the test may exercise the wrong configuration path; use delete process.env.SIM_CONFIG_DIR to reliably unset it.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/sim-cli/src/program.test.ts">

<violation number="1" location="packages/sim-cli/src/program.test.ts:209">
P3: `process.env.SIM_CONFIG_DIR = undefined` does not unset the variable — Node coerces the assignment to the string 'undefined', which stays truthy for `paths.ts`'s `process.env.SIM_CONFIG_DIR || ...` lookup. Use `delete process.env.SIM_CONFIG_DIR` in the finally block so the environment is restored rather than polluted with a bogus config-dir value.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…does Review round 2. Three findings, all valid. The previous commit's message claimed it had replaced `process.env.SIM_CONFIG_DIR = undefined` with `delete` in the test teardowns. It had not: it added a comment explaining why the assignment is wrong and left the assignment in place, so the teardown still stored the literal string "undefined". Both files now actually delete it. The same pattern exists in profile.test.ts and configure.test.ts, which predate this branch and are left alone. Two documentation claims were stronger than the implementation. "At most once a day" is only true with a writable `~/.sim`. The pace lives in a timestamp file, so a read-only home in a container - or a `~/.sim` left root-owned by an earlier sudo install - means the pace cannot be remembered and the check runs per command. That was already noted in a code comment; it is now in the docs where users read it, along with the fact that it stays bounded by the same one-second timeout. "The tag it was installed from" described behaviour that does not exist. The check only ever queries `latest`, because prerelease installs return before any request. Both docs now say that plainly instead of implying the CLI can ask about the staging or dev channel.

mzxchandra · 2026-09-02T22:27:13Z

@greptile

mzxchandra · 2026-09-02T22:27:15Z

@cubic review

cubic-dev-ai · 2026-09-02T22:27:20Z

@cubic review

@mzxchandra I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 10 files

Confidence score: 5/5

In apps/docs/content/docs/cli/configuration.mdx, the cache-file instruction points users with SIM_CONFIG_DIR set to the wrong directory, which can lead them to inspect or update the wrong timestamp file—document the SIM_CONFIG_DIR-aware location.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/docs/content/docs/cli/configuration.mdx">

<violation number="1" location="apps/docs/content/docs/cli/configuration.mdx:158">
P3: When `SIM_CONFIG_DIR` is set, the CLI stores the update timestamp under that directory, not under `~/.sim`; this instruction sends users with relocated configuration to the wrong cache file. Document the `SIM_CONFIG_DIR` location as an alternative.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

Review round 3. The cache is derived from `configDir()`, so it moves with `SIM_CONFIG_DIR` like the config and credentials files do - but the docs named only the `~/.sim` default, sending anyone with a relocated config dir to a file that is not there.

mzxchandra · 2026-09-02T22:32:31Z

@greptile

mzxchandra · 2026-09-02T22:32:32Z

@cubic review

cubic-dev-ai · 2026-09-02T22:32:41Z

@cubic review

@mzxchandra I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 existing issue remains and no new issues found across 10 files

Confidence score: 5/5

apps/docs/content/docs/cli/configuration.mdx documents updateCachePath() as placing the cache beside a relocated SIM_CONFIG_FILE or SIM_CREDENTIALS_FILE, but it currently remains under ~/.sim, which could mislead users configuring isolated paths — clarify the documentation to state that the cache follows SIM_CONFIG_FILE only when applicable, or update the implementation to match the documented behavior.

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic}

mzxchandra added 3 commits September 2, 2026 14:20

Merge remote-tracking branch 'origin/staging' into feat/cli-update-no…

afbec15

…tifier

vercel Bot deployed to Preview September 2, 2026 22:07 View deployment

greptile-apps Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread apps/docs/content/docs/cli/configuration.mdx

cubic-dev-ai Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread packages/sim-cli/README.md Outdated

Comment thread packages/sim-cli/src/program.test.ts Outdated

Comment thread apps/docs/content/docs/cli/troubleshooting.mdx Outdated

Comment thread packages/sim-cli/src/update/semver.ts

mzxchandra added 2 commits September 2, 2026 15:16

Merge remote-tracking branch 'origin/staging' into feat/cli-update-no…

93da3c7

…tifier

vercel Bot deployed to Preview September 2, 2026 22:19 View deployment

cubic-dev-ai Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread apps/docs/content/docs/cli/configuration.mdx Outdated

Comment thread packages/sim-cli/src/program.test.ts

Comment thread packages/sim-cli/README.md Outdated

vercel Bot deployed to Preview September 2, 2026 22:28 View deployment

cubic-dev-ai Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread apps/docs/content/docs/cli/configuration.mdx Outdated

vercel Bot deployed to Preview September 2, 2026 22:34 View deployment

cubic-dev-ai Bot reviewed Sep 2, 2026

View reviewed changes

Uh oh!

Conversation

mzxchandra commented Sep 2, 2026

Summary

Privacy

Test Coverage

Pre-Landing Review

Design Review

Eval Results

Scope Drift

Plan Completion

Verification Results

Documentation

Test plan

Uh oh!

vercel Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mzxchandra commented Sep 2, 2026

Uh oh!

mzxchandra commented Sep 2, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 2, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mzxchandra commented Sep 2, 2026

Uh oh!

mzxchandra commented Sep 2, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 2, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mzxchandra commented Sep 2, 2026

Uh oh!

mzxchandra commented Sep 2, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 2, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Sep 2, 2026 •

edited

Loading

greptile-apps Bot commented Sep 2, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading