Move to Rowan: start the database from the Rowan image, and install Grail and the MCP server through it

CHANGELOG.md

@@ -10,14 +10,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - **See every session on the database.** Click the Sessions row in the GemDB view, or run **GemDB:
-  Show Database Sessions**, for a tab listing every session on the database, not only this
-  window's: what each one is (a notebook, a GemDB Shell, another window, topaz, the MCP server, the
-  database's own gems), its user, process, how old its view is and how many commits behind, and
-  which one is holding back garbage collection. **Stop…** on a row asks first, as **Stop a Database
-  Session…** does. The tab reads when it opens and when you press Refresh, never on a timer: each
-  read is a brief administrator login, and logins are what the database has few of. When that
-  login is refused — the database has no session left to give — it shows the names and processes
-  any session can see, and still offers to close this window's own notebooks.
+  Show Database Sessions**, for a tab listing every session on the database, not only this window's:
+  what each one is (a notebook, a GemDB Shell, another window, topaz, the MCP server, the database's
+  own gems), its user, process, how old its view is and how many commits behind, and which one is
+  holding back garbage collection. **Stop…** on a row asks first, as **Stop a Database Session…**
+  does. The tab reads when it opens and when you press Refresh, never on a timer: each read is a
+  brief administrator login, and logins are what the database has few of. When that login is refused
+  — the database has no session left to give — it shows the names and processes any session can see,
+  and still offers to close this window's own notebooks.
+
+### Changed
+
+- **Your database now starts from GemStone's Rowan image, so start with a fresh one.** Python
+  support and the MCP server are now Rowan projects and install with Rowan, which a database created
+  by an earlier GemDB Code does not have and cannot be given. Delete `~/GemDB/db` before starting
+  this release; GemDB Code refuses an earlier database with a message saying so, rather than failing
+  partway through installing Python support. A database another administrator runs needs to have
+  been created from `extent0.rowan3.dbf` too.
+- **A newer Python runtime.** Your own copy of a package Grail ships (Flask, Werkzeug, Jinja2,
+  MarkupSafe, ItsDangerous, Blinker, Django, asgiref, sqlparse, twilio), installed in a venv or on
+  `sys.path`, is now imported in preference to the bundled one, which used to win every time. A
+  class statement binds its name only once the class is complete, so a class whose body, metaclass
+  or decorator raises leaves the name as it was. Assigning `__defaults__` or `__kwdefaults__` takes
+  effect even when it adds or removes a default. `import flask` no longer leaves each new session
+  with something to commit, and a request to a Flask app stored in the database writes nothing, so
+  `with gemdb.transaction():` works after either.
+- **A newer MCP server (0.9.0).** Every session it starts reaches the database by `localhost`, and
+  `find_python_senders` reports each call where Python sends it.
 
 ### Fixed
 

CLAUDE.md

@@ -260,6 +260,15 @@ first-install path, and `assertSafeToStart` before the stone starts (the
 an extension update reaches that line without preparing anything — engine
 downloaded, database present, Grail staged, so `isInstalled()` is true.
 
+**The database starts from the Rowan image, and one that did not is refused
+the same way.** Grail and the MCP server became Rowan projects on 2026-10-19,
+and Rowan cannot be added to an image afterwards, so `createDatabase` copies
+`bin/extent0.rowan3.dbf` and records it in `conf/image`.
+`assertDatabaseHasRowan`, beside `assertDatabaseMatchesEngine` in both places,
+refuses a database without that record — one GemDB 1.8 or earlier made from
+the plain `extent0.dbf` — because the engine version is the same, so the
+version guard cannot tell them apart. See `docs/grail.md`.
+
 **Below its free-space threshold the stone stops collecting garbage.** GemDB
 caps the extent at the license's 10240 MB, reserves it on disk, and sets a
 500 MB threshold (`withSpaceLimits` in `database.ts`). The threshold is not a

docs/automation-boundary.md

@@ -220,8 +220,8 @@ commits past holds nothing back, and a prompt then would be noise.
 asked, once per build, and only by someone who asked for the database**
 (`grailFileInAgreed` in `lifecycle.ts`). It used to be automated, behind
 `gemdb.reinstallPythonOnUpdate` (default on), on the reasoning that Grail is
-GemDB's own code. It is not inert: `install.gs` removes Grail's dictionaries and
-rebuilds every runtime class, and committed objects keep pointing at the old
+GemDB's own code. It is not inert: the install (then `install.gs`, now a Rowan
+load) removes Grail's dictionaries and rebuilds every runtime class, and committed objects keep pointing at the old
 ones. Measured 2026-10-15 on a database populated under 1.6.0's Grail
 (`beef473`), Brain Freeze included: after filing in `9fb5d23`, committed
 datetimes, dates, timedeltas, Decimals, tuples, sets, instances of user classes

docs/grail.md

@@ -18,7 +18,23 @@ prepared extent made the first install fast and made every upgrade afterwards
 take a different, less-exercised path, which is precisely backwards: the path
 that has to keep working for the life of the database is the one that should
 run every time. So `createDatabase` always copies the engine's own
-`bin/extent0.dbf`, and `ensureRunning` files Grail in.
+`bin/extent0.rowan3.dbf`, and `ensureRunning` installs Grail into it.
+
+**Grail is a Rowan project, so the database starts from the Rowan image.**
+From Grail #1721 (2026-10-19) Grail is Tonel packages in `src/Grail-*` with a
+Rowan project in `rowan/`, and its installer loads them with Rowan
+(`scripts/install_rowan.gs`) instead of filing in `src/smalltalk/install.gs`.
+Rowan cannot be added to an image afterwards, so GemDB creates its database
+from `extent0.rowan3.dbf` (`INITIAL_EXTENT` in `database.ts`) and records that
+in `conf/image`. A database without the record was made by GemDB 1.8 or earlier
+from the plain `extent0.dbf`, and `assertDatabaseHasRowan` refuses it before
+setup and before the stone starts, as `assertDatabaseMatchesEngine` does for an
+engine change. Moving such a database across is manual: recreate it.
+`install-grail.sh` also probes for Rowan before loading, which is what catches
+an external database. The Rowan load runs in one gem with
+`GEM_TEMPOBJ_CACHE_SIZE=1000000`, Grail's own figure, because it compiles the
+whole project as one patch set. It needs no privilege beyond the two `gemdb`
+holds: measured on 4.0.0.a5, `grail.test.ts` loads it as `gemdb`.
 
 What that costs is minutes of topaz on a first run, worth paying once per user
 and not worth paying once per integration test file — so

docs/mcp-server.md

@@ -10,14 +10,18 @@ MCP-specific ones, which are here.
 
 ## What the payload is
 
-Thirty-eight Smalltalk class file-outs (`.gs`) plus the loaders that `input`
-them. That single fact settles most of the design, and it is worth stating
-plainly because the obvious analogue — Grail — is the opposite in every
-respect:
+A Rowan project: six Tonel packages in `src/Mcp-*` (Core, Auth and Grail
+components, each with its tests) and the project in `rowan/`, which
+`install.sh` loads with Rowan. Up to mcp_server 0.9.0 (2026-10-19) it was
+thirty-eight `.gs` file-outs and the loaders that `input` them. Its Grail
+component, the Python toolset GemDB serves, loads only through Rowan; the
+release's file-out carries Core and Auth only. So, like Grail, it needs a
+database started from `extent0.rowan3.dbf` (see `docs/grail.md`). Even so,
+Grail remains the opposite in most respects:
 
 |                        | Grail                              | MCP server                        |
 | ---------------------- | ---------------------------------- | --------------------------------- |
-| What ships             | Python sources + a compiled shim   | Smalltalk file-outs               |
+| What ships             | Python sources + a compiled shim   | Smalltalk packages (Tonel)        |
 | Platform-specific      | yes — one shim per target          | no — identical on all three       |
 | Tied to an engine      | yes — links `gciualib.o`           | no                                |
 | Size                   | ~25 MB                             | ~480 KB                           |

resources/install-grail.sh

@@ -67,6 +67,35 @@ run_topaz() {
     LC_ALL=C topaz -lq -S "$script" < /dev/null
 }
 
+# ---------------------------------------------------------------------------
+# Step 0: the database has Rowan.
+# ---------------------------------------------------------------------------
+# Grail is a Rowan project, loaded into an image started from the engine's
+# bin/extent0.rowan3.dbf. A database GemDB created before 1.9 started from the
+# plain extent0.dbf and has no Rowan, and nothing can add it in place, so say
+# that in a sentence rather than let the load fail deep inside with "Rowan does
+# not understand". GemDB's own database is refused before it starts
+# (assertDatabaseHasRowan); this catches an external one too. Only a positive
+# "absent" stops here: an inconclusive probe leaves the install to report.
+# The script is read into a variable rather than written as a heredoc inside
+# $( ): macOS's bash 3.2 mis-parses a quoted heredoc in a command substitution.
+read -r -d '' ROWAN_PROBE_GS <<'EOF' || true
+login
+run
+'GEMDB_ROWAN=' , ((System myUserProfile symbolList objectNamed: #Rowan) isNil
+  ifTrue: ['absent'] ifFalse: ['present'])
+%
+logout
+exit
+EOF
+ROWAN_PROBE=$(printf '%s\n' "$ROWAN_PROBE_GS" | LC_ALL=C topaz -lq 2>/dev/null || true)
+if printf '%s\n' "$ROWAN_PROBE" | grep -q 'GEMDB_ROWAN=absent'; then
+    echo "ERROR: this database has no Rowan, and Grail installs through Rowan." >&2
+    echo "       It was created from the engine's extent0.dbf; Grail needs one created from" >&2
+    echo "       bin/extent0.rowan3.dbf. Recreate the database to install Python support." >&2
+    exit 3
+fi
+
 # ---------------------------------------------------------------------------
 # Step 1: the shared base, installed once per extent, as SystemUser.
 # ---------------------------------------------------------------------------
@@ -92,7 +121,7 @@ if printf '%s\n' "$BASE_PROBE" | grep -q 'GRAIL_BASE=absent'; then
 
     # Re-probe rather than assume: install_base.sh writes its marker last, so a
     # base still absent here means a step failed without a non-zero exit --
-    # better caught now than as a SecurityError minutes into install.gs.
+    # better caught now than as a SecurityError minutes into the Rowan load.
     BASE_PROBE=$(LC_ALL=C topaz -lq -S "$GRAIL_DIR/scripts/check_base_installed.gs" < /dev/null 2>/dev/null || true)
     if printf '%s\n' "$BASE_PROBE" | grep -q 'GRAIL_BASE=absent'; then
         echo "ERROR: install_base.sh reported success but the base marker is still absent." >&2
@@ -103,14 +132,14 @@ fi
 # ---------------------------------------------------------------------------
 # Step 2: the per-user install.
 # ---------------------------------------------------------------------------
-# install.gs reads SHIM_LIB_PATH from the session environment and records it in
-# the database, so every later session finds the shim without the variable
+# The Rowan load's post-load doits read SHIM_LIB_PATH from the session
+# environment and record it in the database, so every later session finds the shim without the variable
 # being set again. That recorded path is why GemDB stages Grail to a stable
 # directory instead of running it from inside the extension.
 #
 # PYTHON_LIB_PATH is deliberately not set. It points Grail at a CPython shared
 # library on the host for its embedded-FFI backend, and GemDB's whole promise
-# is a Python that needs nothing installed on the machine; install.gs treats it
+# is a Python that needs nothing installed on the machine; the install treats it
 # as optional, so leaving it unset simply leaves that backend unconfigured.
 export SHIM_LIB_PATH="${SHIM_LIB_PATH:-}"
 export PYTHON_PACKAGE_PATH="${PYTHON_PACKAGE_PATH:-$GRAIL_DIR/src/python}"
@@ -124,7 +153,13 @@ fi
 
 echo "== Installing Grail as $GEMDB_USER"
 rm -f "$GRAIL_DIR"/*.out
-run_topaz src/smalltalk/install.gs
+# Rowan loads the Grail project (rowan/specs/Grail.ston, the Tonel packages in
+# src/Grail-*), as Grail's install.sh does. It compiles the whole project as one
+# patch set, far past gem.conf's temporary object space, so this one gem gets
+# Grail's figure; GEM_TEMPOBJ_CACHE_SIZE is a ceiling it grows into, not a
+# reservation.
+echo "--- topaz scripts/install_rowan.gs"
+LC_ALL=C topaz -lq -C "GEM_TEMPOBJ_CACHE_SIZE=1000000;" -S scripts/install_rowan.gs < /dev/null
 
 # ---------------------------------------------------------------------------
 # Step 3: deploy gemdb, so a fresh session starts clean.
@@ -136,7 +171,7 @@ run_topaz src/smalltalk/install.gs
 # `gemdb.transaction()` then refuses to start, describing pending changes the
 # user did not make. Deploying here turns that import into a warm bind.
 #
-# After install.gs, never before: install.gs recreates the Python runtime
+# After the Rowan load, never before: the load recreates the Python runtime
 # classes and bumps Grail's runtime generation, which discards any deployment
 # made under the previous one.
 echo "== Deploying gemdb"

scripts/build-test-extent.sh

@@ -69,7 +69,8 @@ export GEMSTONE_EXE_CONF="$BUILD/conf"
 export PATH="$GEMSTONE/bin:$PATH"
 mkdir -p "$BUILD/conf" "$BUILD/data" "$BUILD/log" "$BUILD/locks"
 
-cp "$GEMSTONE/bin/extent0.dbf" "$BUILD/data/extent0.dbf"
+# The Rowan image, as createDatabase uses: Grail installs through Rowan.
+cp "$GEMSTONE/bin/extent0.rowan3.dbf" "$BUILD/data/extent0.dbf"
 chmod 644 "$BUILD/data/extent0.dbf"
 
 # Mirrors what src/database.ts writes, so the extent is built by the same

scripts/bundle-grail.sh

@@ -138,9 +138,10 @@ echo "Grail commit: $GRAIL_COMMIT ($GRAIL_DESCRIBE)"
 # on a new developer's first run.
 REQUIRED=(
     install_base.sh
-    src/smalltalk/install.gs
+    scripts/install_rowan.gs
+    rowan/specs/Grail.ston
+    rowan/project.ston
     scripts/check_base_installed.gs
-    scripts/kernel_class_extensions.gs
     scripts/deployGemdb.gs
     scripts/deployUnicodedata.gs
     scripts/set_base_marker.gs
@@ -193,6 +194,9 @@ mkdir -p "$DEST"
 cp -R "$SRC/src" "$DEST/src"
 rm -rf "$DEST/src/c"
 cp -R "$SRC/scripts" "$DEST/scripts"
+# Grail is a Rowan project: rowan/ holds the project, load spec and component
+# that scripts/install_rowan.gs loads, with the packages in src/ (Tonel).
+cp -R "$SRC/rowan" "$DEST/rowan"
 # install_base.sh is at Grail's top level, and GemDB's installer shells out to
 # it for the once-per-extent SystemUser steps -- so it has to travel with the
 # payload like the scripts/ directory does.
@@ -202,50 +206,46 @@ done
 find "$DEST" \( -name '*.o' -o -name '*.out' -o -name '__pycache__' \) -exec rm -rf {} + 2>/dev/null || true
 
 # ---------------------------------------------------------------------------
-# Verify every file the topaz scripts read is actually in the payload.
+# Verify every package the Rowan load names is actually in the payload.
 # ---------------------------------------------------------------------------
-# `input <path>` is how a .gs script pulls in another file, and topaz resolves
-# it relative to the working directory -- so a missing target is not caught
-# until the install is already running against a live database. Scanning the
-# assembled payload for those targets turns that into a packaging error, here,
-# where someone is watching.
-#
-# The match is deliberately narrow in two ways. The `./` prefix is optional, so
-# both `input src/x.gs` and `input ./src/x.gs` are seen. And the target must end
-# in `.gs`, because prose inside comments occasionally begins a line with the
-# word "input" — without that anchor the scan reports words like "rather" as
-# missing files, and a check that cries wolf gets ignored.
-#
-# out/gen/ is skipped: resources/install-grail.sh generates it at install time.
-echo "Checking that every 'input' target is present"
+# Rowan loads the packages its components list, from src/ in Tonel form, and
+# runs each component's pre-/post-load doits -- so a package or doit missing
+# from the payload is not caught until the install is already running against
+# a live database. Checking the component files' own lists here turns that into
+# a packaging error, where someone is watching. (Before Grail became a Rowan
+# project this scanned install.gs's `input` lines instead.)
+echo "Checking that every Rowan package and doit is present"
 CHECKED=0
 MISSING=0
-while IFS= read -r target; do
-    case "$target" in
-        out/*) continue ;;
-    esac
-    CHECKED=$((CHECKED + 1))
-    if [ ! -e "$DEST/$target" ]; then
-        echo "  MISSING: $target" >&2
-        MISSING=$((MISSING + 1))
-    fi
-done < <(find "$DEST" -name '*.gs' -exec \
-    sed -nE 's|^[[:space:]]*input[[:space:]]+(\./)?([A-Za-z0-9_./-]+\.gs).*|\2|p' {} + | sort -u)
+for component in "$DEST"/rowan/components/*.ston; do
+    while IFS= read -r package; do
+        CHECKED=$((CHECKED + 1))
+        if [ ! -f "$DEST/src/$package/package.st" ]; then
+            echo "  MISSING package: src/$package" >&2
+            MISSING=$((MISSING + 1))
+        fi
+    done < <(awk '/#packageNames/,/\]/' "$component" | grep -oE "'[A-Za-z0-9-]+'" | tr -d "'")
+    while IFS= read -r doit; do
+        if [ ! -f "$DEST/rowan/components/$doit.st" ]; then
+            echo "  MISSING doit: rowan/components/$doit.st" >&2
+            MISSING=$((MISSING + 1))
+        fi
+    done < <(sed -nE "s/.*#(pre|post)loadDoitName : '([^']+)'.*/\\2/p" "$component")
+done
 
 if [ "$MISSING" -ne 0 ]; then
-    echo "ERROR: $MISSING file(s) referenced by Grail's install scripts are not in the payload." >&2
+    echo "ERROR: $MISSING package(s) or doit(s) the Rowan load names are not in the payload." >&2
     echo "  Grail's source tree has changed shape; adjust the copy step above." >&2
     exit 1
 fi
 # A scan that silently matches nothing is worse than no scan: it reports success
-# for a payload it never looked at. An earlier version of this regex did exactly
-# that, and the missing file surfaced minutes into a user's install instead.
-if [ "$CHECKED" -lt 100 ]; then
-    echo "ERROR: only $CHECKED 'input' targets found; Grail files in hundreds." >&2
+# for a payload it never looked at.
+if [ "$CHECKED" -lt 10 ]; then
+    echo "ERROR: only $CHECKED Rowan packages found; Grail loads a dozen." >&2
     echo "  The scan above is not matching -- fix it rather than trusting this build." >&2
     exit 1
 fi
-echo "  $CHECKED referenced files, all present"
+echo "  $CHECKED Rowan packages, all present"
 
 mkdir -p "$DEST/prebuilt/$PLATFORM_KEY"
 [ -n "$PRESERVED" ] && cp -R "$PRESERVED/." "$DEST/prebuilt/"