diff --git a/CHANGELOG.md b/CHANGELOG.md index 97819a6..28a63e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,14 +10,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - **See every session on the database.** Click the Sessions row in the GemDB view, or run **GemDB: - Show Database Sessions**, for a tab listing every session on the database, not only this - window's: what each one is (a notebook, a GemDB Shell, another window, topaz, the MCP server, the - database's own gems), its user, process, how old its view is and how many commits behind, and - which one is holding back garbage collection. **Stop…** on a row asks first, as **Stop a Database - Session…** does. The tab reads when it opens and when you press Refresh, never on a timer: each - read is a brief administrator login, and logins are what the database has few of. When that - login is refused — the database has no session left to give — it shows the names and processes - any session can see, and still offers to close this window's own notebooks. + Show Database Sessions**, for a tab listing every session on the database, not only this window's: + what each one is (a notebook, a GemDB Shell, another window, topaz, the MCP server, the database's + own gems), its user, process, how old its view is and how many commits behind, and which one is + holding back garbage collection. **Stop…** on a row asks first, as **Stop a Database Session…** + does. The tab reads when it opens and when you press Refresh, never on a timer: each read is a + brief administrator login, and logins are what the database has few of. When that login is refused + — the database has no session left to give — it shows the names and processes any session can see, + and still offers to close this window's own notebooks. + +### Changed + +- **Your database now starts from GemStone's Rowan image, so start with a fresh one.** Python + support and the MCP server are now Rowan projects and install with Rowan, which a database created + by an earlier GemDB Code does not have and cannot be given. Delete `~/GemDB/db` before starting + this release; GemDB Code refuses an earlier database with a message saying so, rather than failing + partway through installing Python support. A database another administrator runs needs to have + been created from `extent0.rowan3.dbf` too. +- **A newer Python runtime.** Your own copy of a package Grail ships (Flask, Werkzeug, Jinja2, + MarkupSafe, ItsDangerous, Blinker, Django, asgiref, sqlparse, twilio), installed in a venv or on + `sys.path`, is now imported in preference to the bundled one, which used to win every time. A + class statement binds its name only once the class is complete, so a class whose body, metaclass + or decorator raises leaves the name as it was. Assigning `__defaults__` or `__kwdefaults__` takes + effect even when it adds or removes a default. `import flask` no longer leaves each new session + with something to commit, and a request to a Flask app stored in the database writes nothing, so + `with gemdb.transaction():` works after either. +- **A newer MCP server (0.9.0).** Every session it starts reaches the database by `localhost`, and + `find_python_senders` reports each call where Python sends it. ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 45dfc1a..d50901b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -260,6 +260,15 @@ first-install path, and `assertSafeToStart` before the stone starts (the an extension update reaches that line without preparing anything — engine downloaded, database present, Grail staged, so `isInstalled()` is true. +**The database starts from the Rowan image, and one that did not is refused +the same way.** Grail and the MCP server became Rowan projects on 2026-10-19, +and Rowan cannot be added to an image afterwards, so `createDatabase` copies +`bin/extent0.rowan3.dbf` and records it in `conf/image`. +`assertDatabaseHasRowan`, beside `assertDatabaseMatchesEngine` in both places, +refuses a database without that record — one GemDB 1.8 or earlier made from +the plain `extent0.dbf` — because the engine version is the same, so the +version guard cannot tell them apart. See `docs/grail.md`. + **Below its free-space threshold the stone stops collecting garbage.** GemDB caps the extent at the license's 10240 MB, reserves it on disk, and sets a 500 MB threshold (`withSpaceLimits` in `database.ts`). The threshold is not a diff --git a/docs/automation-boundary.md b/docs/automation-boundary.md index e35a6ec..f0a7f8d 100644 --- a/docs/automation-boundary.md +++ b/docs/automation-boundary.md @@ -220,8 +220,8 @@ commits past holds nothing back, and a prompt then would be noise. asked, once per build, and only by someone who asked for the database** (`grailFileInAgreed` in `lifecycle.ts`). It used to be automated, behind `gemdb.reinstallPythonOnUpdate` (default on), on the reasoning that Grail is -GemDB's own code. It is not inert: `install.gs` removes Grail's dictionaries and -rebuilds every runtime class, and committed objects keep pointing at the old +GemDB's own code. It is not inert: the install (then `install.gs`, now a Rowan +load) removes Grail's dictionaries and rebuilds every runtime class, and committed objects keep pointing at the old ones. Measured 2026-10-15 on a database populated under 1.6.0's Grail (`beef473`), Brain Freeze included: after filing in `9fb5d23`, committed datetimes, dates, timedeltas, Decimals, tuples, sets, instances of user classes diff --git a/docs/grail.md b/docs/grail.md index dac519a..4130327 100644 --- a/docs/grail.md +++ b/docs/grail.md @@ -18,7 +18,23 @@ prepared extent made the first install fast and made every upgrade afterwards take a different, less-exercised path, which is precisely backwards: the path that has to keep working for the life of the database is the one that should run every time. So `createDatabase` always copies the engine's own -`bin/extent0.dbf`, and `ensureRunning` files Grail in. +`bin/extent0.rowan3.dbf`, and `ensureRunning` installs Grail into it. + +**Grail is a Rowan project, so the database starts from the Rowan image.** +From Grail #1721 (2026-10-19) Grail is Tonel packages in `src/Grail-*` with a +Rowan project in `rowan/`, and its installer loads them with Rowan +(`scripts/install_rowan.gs`) instead of filing in `src/smalltalk/install.gs`. +Rowan cannot be added to an image afterwards, so GemDB creates its database +from `extent0.rowan3.dbf` (`INITIAL_EXTENT` in `database.ts`) and records that +in `conf/image`. A database without the record was made by GemDB 1.8 or earlier +from the plain `extent0.dbf`, and `assertDatabaseHasRowan` refuses it before +setup and before the stone starts, as `assertDatabaseMatchesEngine` does for an +engine change. Moving such a database across is manual: recreate it. +`install-grail.sh` also probes for Rowan before loading, which is what catches +an external database. The Rowan load runs in one gem with +`GEM_TEMPOBJ_CACHE_SIZE=1000000`, Grail's own figure, because it compiles the +whole project as one patch set. It needs no privilege beyond the two `gemdb` +holds: measured on 4.0.0.a5, `grail.test.ts` loads it as `gemdb`. What that costs is minutes of topaz on a first run, worth paying once per user and not worth paying once per integration test file — so diff --git a/docs/mcp-server.md b/docs/mcp-server.md index 16f2124..88490a6 100644 --- a/docs/mcp-server.md +++ b/docs/mcp-server.md @@ -10,14 +10,18 @@ MCP-specific ones, which are here. ## What the payload is -Thirty-eight Smalltalk class file-outs (`.gs`) plus the loaders that `input` -them. That single fact settles most of the design, and it is worth stating -plainly because the obvious analogue — Grail — is the opposite in every -respect: +A Rowan project: six Tonel packages in `src/Mcp-*` (Core, Auth and Grail +components, each with its tests) and the project in `rowan/`, which +`install.sh` loads with Rowan. Up to mcp_server 0.9.0 (2026-10-19) it was +thirty-eight `.gs` file-outs and the loaders that `input` them. Its Grail +component, the Python toolset GemDB serves, loads only through Rowan; the +release's file-out carries Core and Auth only. So, like Grail, it needs a +database started from `extent0.rowan3.dbf` (see `docs/grail.md`). Even so, +Grail remains the opposite in most respects: | | Grail | MCP server | | ---------------------- | ---------------------------------- | --------------------------------- | -| What ships | Python sources + a compiled shim | Smalltalk file-outs | +| What ships | Python sources + a compiled shim | Smalltalk packages (Tonel) | | Platform-specific | yes — one shim per target | no — identical on all three | | Tied to an engine | yes — links `gciualib.o` | no | | Size | ~25 MB | ~480 KB | diff --git a/resources/install-grail.sh b/resources/install-grail.sh index d54edc7..3476ade 100755 --- a/resources/install-grail.sh +++ b/resources/install-grail.sh @@ -67,6 +67,35 @@ run_topaz() { LC_ALL=C topaz -lq -S "$script" < /dev/null } +# --------------------------------------------------------------------------- +# Step 0: the database has Rowan. +# --------------------------------------------------------------------------- +# Grail is a Rowan project, loaded into an image started from the engine's +# bin/extent0.rowan3.dbf. A database GemDB created before 1.9 started from the +# plain extent0.dbf and has no Rowan, and nothing can add it in place, so say +# that in a sentence rather than let the load fail deep inside with "Rowan does +# not understand". GemDB's own database is refused before it starts +# (assertDatabaseHasRowan); this catches an external one too. Only a positive +# "absent" stops here: an inconclusive probe leaves the install to report. +# The script is read into a variable rather than written as a heredoc inside +# $( ): macOS's bash 3.2 mis-parses a quoted heredoc in a command substitution. +read -r -d '' ROWAN_PROBE_GS <<'EOF' || true +login +run +'GEMDB_ROWAN=' , ((System myUserProfile symbolList objectNamed: #Rowan) isNil + ifTrue: ['absent'] ifFalse: ['present']) +% +logout +exit +EOF +ROWAN_PROBE=$(printf '%s\n' "$ROWAN_PROBE_GS" | LC_ALL=C topaz -lq 2>/dev/null || true) +if printf '%s\n' "$ROWAN_PROBE" | grep -q 'GEMDB_ROWAN=absent'; then + echo "ERROR: this database has no Rowan, and Grail installs through Rowan." >&2 + echo " It was created from the engine's extent0.dbf; Grail needs one created from" >&2 + echo " bin/extent0.rowan3.dbf. Recreate the database to install Python support." >&2 + exit 3 +fi + # --------------------------------------------------------------------------- # Step 1: the shared base, installed once per extent, as SystemUser. # --------------------------------------------------------------------------- @@ -92,7 +121,7 @@ if printf '%s\n' "$BASE_PROBE" | grep -q 'GRAIL_BASE=absent'; then # Re-probe rather than assume: install_base.sh writes its marker last, so a # base still absent here means a step failed without a non-zero exit -- - # better caught now than as a SecurityError minutes into install.gs. + # better caught now than as a SecurityError minutes into the Rowan load. BASE_PROBE=$(LC_ALL=C topaz -lq -S "$GRAIL_DIR/scripts/check_base_installed.gs" < /dev/null 2>/dev/null || true) if printf '%s\n' "$BASE_PROBE" | grep -q 'GRAIL_BASE=absent'; then echo "ERROR: install_base.sh reported success but the base marker is still absent." >&2 @@ -103,14 +132,14 @@ fi # --------------------------------------------------------------------------- # Step 2: the per-user install. # --------------------------------------------------------------------------- -# install.gs reads SHIM_LIB_PATH from the session environment and records it in -# the database, so every later session finds the shim without the variable +# The Rowan load's post-load doits read SHIM_LIB_PATH from the session +# environment and record it in the database, so every later session finds the shim without the variable # being set again. That recorded path is why GemDB stages Grail to a stable # directory instead of running it from inside the extension. # # PYTHON_LIB_PATH is deliberately not set. It points Grail at a CPython shared # library on the host for its embedded-FFI backend, and GemDB's whole promise -# is a Python that needs nothing installed on the machine; install.gs treats it +# is a Python that needs nothing installed on the machine; the install treats it # as optional, so leaving it unset simply leaves that backend unconfigured. export SHIM_LIB_PATH="${SHIM_LIB_PATH:-}" export PYTHON_PACKAGE_PATH="${PYTHON_PACKAGE_PATH:-$GRAIL_DIR/src/python}" @@ -124,7 +153,13 @@ fi echo "== Installing Grail as $GEMDB_USER" rm -f "$GRAIL_DIR"/*.out -run_topaz src/smalltalk/install.gs +# Rowan loads the Grail project (rowan/specs/Grail.ston, the Tonel packages in +# src/Grail-*), as Grail's install.sh does. It compiles the whole project as one +# patch set, far past gem.conf's temporary object space, so this one gem gets +# Grail's figure; GEM_TEMPOBJ_CACHE_SIZE is a ceiling it grows into, not a +# reservation. +echo "--- topaz scripts/install_rowan.gs" +LC_ALL=C topaz -lq -C "GEM_TEMPOBJ_CACHE_SIZE=1000000;" -S scripts/install_rowan.gs < /dev/null # --------------------------------------------------------------------------- # Step 3: deploy gemdb, so a fresh session starts clean. @@ -136,7 +171,7 @@ run_topaz src/smalltalk/install.gs # `gemdb.transaction()` then refuses to start, describing pending changes the # user did not make. Deploying here turns that import into a warm bind. # -# After install.gs, never before: install.gs recreates the Python runtime +# After the Rowan load, never before: the load recreates the Python runtime # classes and bumps Grail's runtime generation, which discards any deployment # made under the previous one. echo "== Deploying gemdb" diff --git a/scripts/build-test-extent.sh b/scripts/build-test-extent.sh index bf15f17..b0d9ffe 100755 --- a/scripts/build-test-extent.sh +++ b/scripts/build-test-extent.sh @@ -69,7 +69,8 @@ export GEMSTONE_EXE_CONF="$BUILD/conf" export PATH="$GEMSTONE/bin:$PATH" mkdir -p "$BUILD/conf" "$BUILD/data" "$BUILD/log" "$BUILD/locks" -cp "$GEMSTONE/bin/extent0.dbf" "$BUILD/data/extent0.dbf" +# The Rowan image, as createDatabase uses: Grail installs through Rowan. +cp "$GEMSTONE/bin/extent0.rowan3.dbf" "$BUILD/data/extent0.dbf" chmod 644 "$BUILD/data/extent0.dbf" # Mirrors what src/database.ts writes, so the extent is built by the same diff --git a/scripts/bundle-grail.sh b/scripts/bundle-grail.sh index e8b7009..87409f5 100755 --- a/scripts/bundle-grail.sh +++ b/scripts/bundle-grail.sh @@ -138,9 +138,10 @@ echo "Grail commit: $GRAIL_COMMIT ($GRAIL_DESCRIBE)" # on a new developer's first run. REQUIRED=( install_base.sh - src/smalltalk/install.gs + scripts/install_rowan.gs + rowan/specs/Grail.ston + rowan/project.ston scripts/check_base_installed.gs - scripts/kernel_class_extensions.gs scripts/deployGemdb.gs scripts/deployUnicodedata.gs scripts/set_base_marker.gs @@ -193,6 +194,9 @@ mkdir -p "$DEST" cp -R "$SRC/src" "$DEST/src" rm -rf "$DEST/src/c" cp -R "$SRC/scripts" "$DEST/scripts" +# Grail is a Rowan project: rowan/ holds the project, load spec and component +# that scripts/install_rowan.gs loads, with the packages in src/ (Tonel). +cp -R "$SRC/rowan" "$DEST/rowan" # install_base.sh is at Grail's top level, and GemDB's installer shells out to # it for the once-per-extent SystemUser steps -- so it has to travel with the # payload like the scripts/ directory does. @@ -202,50 +206,46 @@ done find "$DEST" \( -name '*.o' -o -name '*.out' -o -name '__pycache__' \) -exec rm -rf {} + 2>/dev/null || true # --------------------------------------------------------------------------- -# Verify every file the topaz scripts read is actually in the payload. +# Verify every package the Rowan load names is actually in the payload. # --------------------------------------------------------------------------- -# `input ` is how a .gs script pulls in another file, and topaz resolves -# it relative to the working directory -- so a missing target is not caught -# until the install is already running against a live database. Scanning the -# assembled payload for those targets turns that into a packaging error, here, -# where someone is watching. -# -# The match is deliberately narrow in two ways. The `./` prefix is optional, so -# both `input src/x.gs` and `input ./src/x.gs` are seen. And the target must end -# in `.gs`, because prose inside comments occasionally begins a line with the -# word "input" — without that anchor the scan reports words like "rather" as -# missing files, and a check that cries wolf gets ignored. -# -# out/gen/ is skipped: resources/install-grail.sh generates it at install time. -echo "Checking that every 'input' target is present" +# Rowan loads the packages its components list, from src/ in Tonel form, and +# runs each component's pre-/post-load doits -- so a package or doit missing +# from the payload is not caught until the install is already running against +# a live database. Checking the component files' own lists here turns that into +# a packaging error, where someone is watching. (Before Grail became a Rowan +# project this scanned install.gs's `input` lines instead.) +echo "Checking that every Rowan package and doit is present" CHECKED=0 MISSING=0 -while IFS= read -r target; do - case "$target" in - out/*) continue ;; - esac - CHECKED=$((CHECKED + 1)) - if [ ! -e "$DEST/$target" ]; then - echo " MISSING: $target" >&2 - MISSING=$((MISSING + 1)) - fi -done < <(find "$DEST" -name '*.gs' -exec \ - sed -nE 's|^[[:space:]]*input[[:space:]]+(\./)?([A-Za-z0-9_./-]+\.gs).*|\2|p' {} + | sort -u) +for component in "$DEST"/rowan/components/*.ston; do + while IFS= read -r package; do + CHECKED=$((CHECKED + 1)) + if [ ! -f "$DEST/src/$package/package.st" ]; then + echo " MISSING package: src/$package" >&2 + MISSING=$((MISSING + 1)) + fi + done < <(awk '/#packageNames/,/\]/' "$component" | grep -oE "'[A-Za-z0-9-]+'" | tr -d "'") + while IFS= read -r doit; do + if [ ! -f "$DEST/rowan/components/$doit.st" ]; then + echo " MISSING doit: rowan/components/$doit.st" >&2 + MISSING=$((MISSING + 1)) + fi + done < <(sed -nE "s/.*#(pre|post)loadDoitName : '([^']+)'.*/\\2/p" "$component") +done if [ "$MISSING" -ne 0 ]; then - echo "ERROR: $MISSING file(s) referenced by Grail's install scripts are not in the payload." >&2 + echo "ERROR: $MISSING package(s) or doit(s) the Rowan load names are not in the payload." >&2 echo " Grail's source tree has changed shape; adjust the copy step above." >&2 exit 1 fi # A scan that silently matches nothing is worse than no scan: it reports success -# for a payload it never looked at. An earlier version of this regex did exactly -# that, and the missing file surfaced minutes into a user's install instead. -if [ "$CHECKED" -lt 100 ]; then - echo "ERROR: only $CHECKED 'input' targets found; Grail files in hundreds." >&2 +# for a payload it never looked at. +if [ "$CHECKED" -lt 10 ]; then + echo "ERROR: only $CHECKED Rowan packages found; Grail loads a dozen." >&2 echo " The scan above is not matching -- fix it rather than trusting this build." >&2 exit 1 fi -echo " $CHECKED referenced files, all present" +echo " $CHECKED Rowan packages, all present" mkdir -p "$DEST/prebuilt/$PLATFORM_KEY" [ -n "$PRESERVED" ] && cp -R "$PRESERVED/." "$DEST/prebuilt/" diff --git a/scripts/bundle-mcp.sh b/scripts/bundle-mcp.sh index 74bd048..0ad9c61 100755 --- a/scripts/bundle-mcp.sh +++ b/scripts/bundle-mcp.sh @@ -107,15 +107,15 @@ ENTRYPOINTS=( # would never have staged it. setup-read-only-user.sh ) -# Loaders inside the wholesale src/ copy that src/mcp.ts's install path drives. -# Not entry points in the shell sense, so no closure applies -- what THEY read -# is checked by the `input` scan further down. -REQUIRED_GS=( - src/core/load.gs - src/tests/load.gs - src/grail/load.gs +# What install.sh's Rowan load reads: mcp_server is a Rowan project (0.9.0 on), +# and install.sh loads it from this directory through rowan/specs/Mcp.ston, +# with the packages in src/ (Tonel). Not entry points in the shell sense, so no +# closure applies -- the packages the components name are checked further down. +REQUIRED_ROWAN=( + rowan/specs/Mcp.ston + rowan/project.ston ) -for item in "${ENTRYPOINTS[@]}" "${REQUIRED_GS[@]}"; do +for item in "${ENTRYPOINTS[@]}" "${REQUIRED_ROWAN[@]}"; do if [ ! -e "$SRC/$item" ]; then echo "ERROR: mcp_server no longer provides $item -- GemDB's installer needs it." >&2 exit 1 @@ -133,11 +133,11 @@ mkdir -p "$DEST" # wholesale: the loaders decide what the source tree contains, and a hand-picked # list of subdirectories silently omits whatever is added next. cp -R "$SRC/src" "$DEST/src" +cp -R "$SRC/rowan" "$DEST/rowan" -# Documents and the top-level loader. Carried for the reader, not driven, so -# there is nothing to derive from them; load.gs's own `input` targets are -# covered by the scan below. -for item in load.gs LICENSE README.md; do +# Documents, carried for the reader and not driven, so there is nothing to +# derive from them. +for item in LICENSE README.md; do if [ -e "$SRC/$item" ]; then cp "$SRC/$item" "$DEST/$item"; fi done @@ -231,35 +231,38 @@ fi echo " $SH_CHECKED referenced scripts, all present" # --------------------------------------------------------------------------- -# Verify every file the loaders read is present. +# Verify every package the Rowan load can name is present. # --------------------------------------------------------------------------- -# Same check bundle-grail.sh makes, and for the same reason: `input ` is -# resolved by topaz against the working directory, so a missing target is not -# caught until the file-in is already running against a live database. -echo "Checking that every 'input' target is present" +# Same check bundle-grail.sh makes, and for the same reason: Rowan reads the +# packages its components list from src/, so a missing one is not caught until +# the load is already running against a live database. Every component is +# checked, not only the ones src/mcp.ts selects, since install.sh's flags +# choose among them. +echo "Checking that every Rowan package is present" CHECKED=0 MISSING=0 -while IFS= read -r target; do - CHECKED=$((CHECKED + 1)) - if [ ! -e "$DEST/$target" ]; then - echo " MISSING: $target" >&2 - MISSING=$((MISSING + 1)) - fi -done < <(find "$DEST" -name '*.gs' -exec \ - sed -nE 's|^[[:space:]]*input[[:space:]]+(\./)?([A-Za-z0-9_./-]+\.gs).*|\2|p' {} + | sort -u) +for component in "$DEST"/rowan/components/*.ston; do + while IFS= read -r package; do + CHECKED=$((CHECKED + 1)) + if [ ! -f "$DEST/src/$package/package.st" ]; then + echo " MISSING package: src/$package" >&2 + MISSING=$((MISSING + 1)) + fi + done < <(awk '/#packageNames/,/\]/' "$component" | grep -oE "'[A-Za-z0-9-]+'" | tr -d "'") +done if [ "$MISSING" -ne 0 ]; then - echo "ERROR: $MISSING file(s) referenced by the MCP loaders are not in the payload." >&2 + echo "ERROR: $MISSING package(s) the Rowan components name are not in the payload." >&2 exit 1 fi # A scan that matches nothing reports success for a payload it never looked at. -# The four loaders together `input` one file per class, so this is dozens. -if [ "$CHECKED" -lt 20 ]; then - echo "ERROR: only $CHECKED 'input' targets found; the MCP loaders file in dozens." >&2 +# Core, Auth and Grail name two packages each. +if [ "$CHECKED" -lt 6 ]; then + echo "ERROR: only $CHECKED Rowan packages found; the components name six." >&2 echo " The scan above is not matching -- fix it rather than trusting this build." >&2 exit 1 fi -echo " $CHECKED referenced files, all present" +echo " $CHECKED Rowan packages, all present" # The stamp is what GemDB compares to decide whether the payload filed into the # database is the one this build ships. No engine in it, unlike Grail's: there @@ -270,4 +273,4 @@ commit=$MCP_COMMIT VERSION echo -echo "Bundled the MCP server $MCP_DESCRIBE ($(find "$DEST/src" -name '*.gs' | wc -l | tr -d ' ') class files)." +echo "Bundled the MCP server $MCP_DESCRIBE ($CHECKED Rowan packages)." diff --git a/scripts/check-vsix.sh b/scripts/check-vsix.sh index 26d6994..efd20bf 100755 --- a/scripts/check-vsix.sh +++ b/scripts/check-vsix.sh @@ -89,11 +89,14 @@ REQUIRED=( "extension/grail/GRAIL_VERSION|the Grail payload" "extension/grail/prebuilt/$PLATFORM_KEY/libcpython_ua.$LIB_EXT|the CPython shim for $PLATFORM_KEY" "extension/grail/scripts/grail.tpz|the Grail scripts the installer drives" + "extension/grail/scripts/install_rowan.gs|the Rowan load the Grail installer runs" + "extension/grail/rowan/specs/Grail.ston|Grail's Rowan load spec" "extension/node_modules/koffi/build/koffi/$KOFFI_KEY/koffi.node|the native FFI addon for $TARGET" "extension/resources/install-grail.sh|the installer the extension runs" "extension/mcp/MCP_VERSION|the MCP server payload" "extension/mcp/install.sh|the MCP installer the extension runs" - "extension/mcp/src/core/McpRouter.gs|the MCP front end the extension forks" + "extension/mcp/src/Mcp-Core/McpRouter.class.st|the MCP front end the extension forks" + "extension/mcp/rowan/specs/Mcp.ston|the MCP server's Rowan load spec" "extension/stats/host.html|the GemDB Stats page, made hostable by bundle-stats.mjs" "extension/stats/flutter_bootstrap.js|what starts GemDB Stats" "extension/stats/main.dart.js|GemDB Stats itself" diff --git a/src/__tests__/createDatabase.test.ts b/src/__tests__/createDatabase.test.ts index 341e9f4..4a25523 100644 --- a/src/__tests__/createDatabase.test.ts +++ b/src/__tests__/createDatabase.test.ts @@ -3,7 +3,7 @@ import * as os from 'os'; import * as path from 'path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { __setSetting } from '../__mocks__/vscode'; -import { createDatabase } from '../database'; +import { IMAGE_RECORD, INITIAL_EXTENT, createDatabase } from '../database'; import { databaseExists, databasePath, extentPath } from '../paths'; // Passes through unless a test hooks it: the extent's chmod is the last step @@ -45,7 +45,7 @@ function makeEngine(withExtent = true): string { fs.mkdirSync(path.join(dir, 'bin')); fs.writeFileSync(path.join(dir, 'sys', 'community.starter.key'), 'key'); fs.writeFileSync(path.join(dir, 'data', 'system.conf'), '# defaults'); - if (withExtent) fs.writeFileSync(path.join(dir, 'bin', 'extent0.dbf'), 'extent'); + if (withExtent) fs.writeFileSync(path.join(dir, 'bin', INITIAL_EXTENT), 'extent'); return dir; } @@ -183,3 +183,17 @@ describe('creating the database', () => { expect(fs.existsSync(staging())).toBe(false); }); }); + +describe('the image the database starts from', () => { + it('is the engine’s Rowan extent, and the database records so', () => { + fs.writeFileSync(path.join(engine, 'bin', 'extent0.dbf'), 'plain'); + + createDatabase(engine); + + expect(INITIAL_EXTENT).toBe('extent0.rowan3.dbf'); + expect(fs.readFileSync(extentPath(), 'utf8')).toBe('extent'); + expect(fs.readFileSync(path.join(databasePath(), 'conf', IMAGE_RECORD), 'utf8').trim()).toBe( + INITIAL_EXTENT, + ); + }); +}); diff --git a/src/__tests__/databaseVersion.test.ts b/src/__tests__/databaseVersion.test.ts index f99f58f..c0b7139 100644 --- a/src/__tests__/databaseVersion.test.ts +++ b/src/__tests__/databaseVersion.test.ts @@ -5,6 +5,9 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import { __setSetting } from '../__mocks__/vscode'; import { DatabaseVersionError, + IMAGE_RECORD, + INITIAL_EXTENT, + assertDatabaseHasRowan, assertDatabaseMatchesEngine, parseRepositoryVersion, repositoryVersion, @@ -144,3 +147,46 @@ describe('the guard in front of a database', () => { engine = ''; }); }); + +// Up to 1.8 GemDB created its database from the engine's plain extent0.dbf, +// which has no Rowan; Grail and the MCP server now install only through Rowan, +// and the engine version is the same, so only the image record tells them apart. +describe('a database created without Rowan', () => { + const recordImage = () => { + fs.mkdirSync(path.join(databasePath(), 'conf'), { recursive: true }); + fs.writeFileSync(path.join(databasePath(), 'conf', IMAGE_RECORD), `${INITIAL_EXTENT}\n`); + }; + + it('is refused, with the folder to delete and a warning', () => { + makeExtent(); + + expect(() => assertDatabaseHasRowan()).toThrow(DatabaseVersionError); + try { + assertDatabaseHasRowan(); + } catch (e) { + const message = (e as Error).message; + expect(message).toMatch(/Rowan/); + expect(message).toContain(databasePath()); + expect(message).toMatch(/delete/i); + } + }); + + it('is not what a database created from the Rowan image is taken for', () => { + makeExtent(); + recordImage(); + + expect(() => assertDatabaseHasRowan()).not.toThrow(); + }); + + it('says nothing before there is a database', () => { + expect(() => assertDatabaseHasRowan()).not.toThrow(); + }); + + it('leaves an external database, whose image is its administrator’s, alone', () => { + makeExtent(); + __setSetting('gemdb.externalDatabase.gemstone', '/opt/gemstone/product'); + + expect(() => assertDatabaseHasRowan()).not.toThrow(); + __setSetting('gemdb.externalDatabase.gemstone', ''); + }); +}); diff --git a/src/__tests__/setup.test.ts b/src/__tests__/setup.test.ts index 9b64a95..11cc8cc 100644 --- a/src/__tests__/setup.test.ts +++ b/src/__tests__/setup.test.ts @@ -39,6 +39,7 @@ vi.mock('../database', async (importOriginal) => { assertRoomForExtent: () => {}, ensureSpaceLimits: () => {}, assertDatabaseMatchesEngine: () => {}, + assertDatabaseHasRowan: () => {}, createDatabase: (enginePath: string, extensionPath?: string) => createDatabase(enginePath, extensionPath), }; diff --git a/src/database.ts b/src/database.ts index 329369f..5bbf131 100644 --- a/src/database.ts +++ b/src/database.ts @@ -2,7 +2,13 @@ import * as crypto from 'crypto'; import * as fs from 'fs'; import * as path from 'path'; import { execFileSync } from 'child_process'; -import { STONE_NAME, databasePasswordPath, engineVersion, rootPath } from './config'; +import { + STONE_NAME, + databasePasswordPath, + engineVersion, + isExternalDatabase, + rootPath, +} from './config'; import { log, logStep } from './log'; import { isOnNfs } from './networkFileSystem'; import { databaseExists, databasePath, enginePath, ensureRootPath, extentPath } from './paths'; @@ -125,7 +131,7 @@ export function createDatabase(enginePath: string): boolean { // the `gemdb` command among them — finds it from the beginning. ensurePasswordFile(path.join(staging, 'conf', 'gemdb.password')); - const stock = path.join(enginePath, 'bin', 'extent0.dbf'); + const stock = path.join(enginePath, 'bin', INITIAL_EXTENT); if (!fs.existsSync(stock)) { throw new Error(`The engine at ${enginePath} has no initial extent at ${stock}.`); } @@ -135,6 +141,9 @@ export function createDatabase(enginePath: string): boolean { // The extent ships read-only in the product tree; the engine must be able to // write to this copy. fs.chmodSync(extent, 0o644); + // Which image the database started from, since nothing in the extent says so + // without a login: see assertDatabaseHasRowan. + fs.writeFileSync(path.join(staging, 'conf', IMAGE_RECORD), `${INITIAL_EXTENT}\n`); // `clearDatabaseDirWithoutExtent` has already removed any `db/` it could, so // a rename that still finds one there is refused the same way. @@ -495,6 +504,45 @@ export function repositoryVersion(enginePath: string): string | undefined { /** Raised when the database on disk was written by a different engine. */ export class DatabaseVersionError extends Error {} +/** + * The image every GemDB database starts from: the engine's extent with Rowan 3 + * in it. Grail and the MCP server are Rowan projects, installed by a Rowan load, + * and Rowan cannot be added to an image afterwards. + */ +export const INITIAL_EXTENT = 'extent0.rowan3.dbf'; + +/** The file under `conf/` that names the image the database started from. */ +export const IMAGE_RECORD = 'image'; + +/** + * Refuse a database created before GemDB started from the Rowan image. + * + * Up to 1.8, GemDB created its database from the engine's plain `extent0.dbf`. + * Grail and the MCP server now install only through Rowan, which that image + * does not have and cannot be given in place, so the install would fail deep + * inside a Rowan load. The engine version is the same, so + * `assertDatabaseMatchesEngine` does not notice. Nothing in the extent says + * which image it came from without logging in, so `createDatabase` records it + * in `conf/image`, and a database without that record is one an earlier GemDB + * created. Checked in the same two places as the engine version, for the same + * reasons: before setup, and before the stone starts, since an extension update + * reaches a start without setup. + * + * Not for an external database, whose image is its administrator's; + * `install-grail.sh` probes for Rowan there and says the same thing. + */ +export function assertDatabaseHasRowan(): void { + if (isExternalDatabase() || !databaseExists()) return; + if (fs.existsSync(path.join(databasePath(), 'conf', IMAGE_RECORD))) return; + throw new DatabaseVersionError( + `The database at ${databasePath()} was created by an earlier GemDB, from an image ` + + 'without Rowan. Python support and the MCP server now install through Rowan, which ' + + 'cannot be added to an existing database, so it has to be recreated: delete ' + + `${databasePath()} and start GemDB again. Anything stored in it is lost, so copy out ` + + 'whatever you still need first.', + ); +} + /** * Refuse to touch a database an older engine wrote. * @@ -585,6 +633,7 @@ export function assertSafeToStart(): void { assertDatabaseIsLocal(); const engine = enginePath(); if (engine) assertDatabaseMatchesEngine(engine, engineVersion()); + assertDatabaseHasRowan(); ensureSpaceLimits(); assertRoomForExtent(); } diff --git a/src/lifecycle.ts b/src/lifecycle.ts index a036dc5..c02bf64 100644 --- a/src/lifecycle.ts +++ b/src/lifecycle.ts @@ -12,6 +12,7 @@ import { ensureDatabaseAccount } from './account'; import { writeCliScripts } from './cli'; import { assertDatabaseIsLocal, + assertDatabaseHasRowan, assertDatabaseMatchesEngine, createDatabase, DatabaseOnNfsError, @@ -170,6 +171,7 @@ async function prepareFiles( // cannot be used by this one, and the engine will not say so until a login // fails. See assertDatabaseMatchesEngine. assertDatabaseMatchesEngine(engine, engineVersion()); + assertDatabaseHasRowan(); progress.report({ message: 'Creating your database…' }); createDatabase(engine); diff --git a/vendor-pins.sh b/vendor-pins.sh index 2dd70d2..2851a45 100644 --- a/vendor-pins.sh +++ b/vendor-pins.sh @@ -11,52 +11,39 @@ # Bumping a pin is a one-line PR to this file; a green CI run on it is the # proof the new upstream commit works. -# Grail: main as of 2026-10-18, 71 commits (31 PRs) on from the previous pin -# (9fb5d23). install.sh and every script resources/install-grail.sh drives are -# byte-identical, there are no deletions or renames, and every name in -# bundle-grail.sh's REQUIRED list is still there. install.gs only adds: one -# runtime input (GrailDynLoadRequest) and test classes, all under src/. The C -# shim's SOURCE changed (numpy, cffi, C-type introspection), so every platform -# rebuilds it; the Makefile adds shim_cffi.o to the default build, the only new -# includes are libc's, and the GCI calls are unchanged and declared in a5's -# headers. The gemdb Python package is unchanged. Grail #1667 (a bare `import -# flask` dirties every fresh session on a deployed stone) is still open. +# Grail: main as of 2026-10-20, 31 commits (8 PRs) on from the previous pin +# (a78e8c0) -- and GRAIL IS NOW A ROWAN PROJECT (#1721, #1722). Its packages are +# Tonel in src/Grail-*, its project is in rowan/, and install.sh loads it with +# Rowan (scripts/install_rowan.gs) instead of filing in src/smalltalk/install.gs, +# which is gone, as is scripts/kernel_class_extensions.gs. It needs an image +# started from the engine's bin/extent0.rowan3.dbf, so GemDB now creates its +# database from that (database.ts), install-grail.sh runs the Rowan load with +# Grail's GEM_TEMPOBJ_CACHE_SIZE and probes for Rowan first, and bundle-grail.sh +# ships rowan/ and checks every package and doit the component names instead of +# scanning `input` lines, of which none remain. The shim and its Makefile are +# as before; the gemdb Python package is unchanged. # -# What users get: class bodies and the object model closer to CPython's -# (#1709, #1685, #1697, #1671, #1704 and more); numpy.random's Cython modules -# loading and nested extension loads (#1665); cffi entry points, `import -# uvicorn`, hashlib.sha224/sha384 (#1676); FastAPI's core tests at 99.8% -# (#1654); and abandoned generators running their `finally` blocks (#1684). -# -# Behaviour changes a user can meet: a class body now runs `raise`, `assert`, -# `match` and conditional `del`/`import` it used to skip, and one that fails -# leaves the name unbound (#1709); bad `__slots__` raise TypeError (#1697); a -# `@property` wins over an instance value of the same name (#1671); a coroutine -# is no longer a GeneratorType (#1672); `x *= y` where both sides decline raises -# instead of storing NotImplemented (#1690); and patching `json.dumps`, -# `math.sqrt` or `os.getcwd` now reaches their callers (#1654). -PINNED_GRAIL_REF=a78e8c03c78bf957acb9119ae10c01567acdda06 +# Also in the range: #1716 moves the bundled third-party packages (flask, +# werkzeug, jinja2, django, ...) to src/python/bundled, searched after sys.path, +# so a user's own copy wins; #1718 binds a class's name only once the class is +# complete; #1719 makes a shape-changing __defaults__ assignment rebind calls; +# #1720 fixes numpy's census cases; #1723 stops a bare `import flask` dirtying +# every fresh session on a deployed stone (#1667, filed by GemDB); and #1725 +# makes a request to a deployed Flask app write nothing (#1229). +PINNED_GRAIL_REF=c8cbc92dfc183fd61f5fd0b7c92b56e82c0b686b -# mcp_server: main as of 2026-09-25, 26 commits (8 PRs) on from the previous pin -# (afa3790). No load.gs changed, and every selector src/mcp.ts sends -- -# toolsetNames:, toolsetOptions:, workerUserId:, serverTitle:, forkOnPort:, -# McpServer defaultToolsetNames -- is still there, as is the "gem session N -# (host pid P)" status line it parses. What moved: -# -# - Upstream CI moved to 4.0.0.a3, the engine GemDB already pins. a3's -# JsonParser refuses trailing whitespace after the outer value, so the server -# now strips it before parsing (#54) -- a pretty-printed or newline-ended -# request body was a -32700 under a3 before this. -# - Breaking upstream, inert here: the worker bootstrap selector gained an -# `instructions:` keyword (#45), because the `initialize` instructions are -# now router config (McpRouter>>serverInstructions:). The router sends that -# selector to its own workers; GemDB never does. GemDB serves the full default -# surface plus Grail, so the stock instructions, which name the transaction -# tools, stay accurate. -# - find_python_senders places each call and reference in an IR-compiled method -# with its own line (#46, #51) instead of one `line ?` per method, and -# eval_python runs as `__main__` (#52). -PINNED_MCP_REF=d836520816d47bc5b8bb43386293f1b9dffbefce +# mcp_server: 0.9.0 (main as of 2026-10-19), 20 commits (9 PRs) on from the +# previous pin (d836520) -- and, like Grail, NOW A ROWAN PROJECT (#57). Its six +# packages are Tonel in src/Mcp-* (components Core, Auth, Grail), its project is +# in rowan/, and install.sh loads it with Rowan by default; the src/*/load.gs +# loaders are gone. The Grail component -- the Python toolset GemDB serves -- +# loads ONLY through Rowan: the release's file-out (--from-gs) carries Core and +# Auth. GemDB's flags are unchanged (--grail --no-auth), bundle-mcp.sh ships +# rowan/ and checks every package the components name, and check-vsix.sh looks +# for the Rowan router class. Also: every forked gem reaches the stone by +# localhost (#63), topaz runs with -i so no .topazini is read (#62), and +# find_python_senders takes its positions from Grail's send sites (#65). +PINNED_MCP_REF=7ab106728285c41f573b5469b8c341dfdad988aa # GemDB Stats: a release asset, not a commit. Its web build needs the Flutter # SDK to compile, which GemDB Code's CI does not carry, so GemDB Stats'