fix(doctor): run the migration checks SM010 and SM011 (#895)

CHANGELOG.md

@@ -16,6 +16,7 @@ All notable changes to this project are documented in this file. The format is b
 
 ### Fixed
 
+- `make doctor` (and `smpy doctor`) now runs the migration checks it documents. It reads `alembic_version` and the table list from `SM_DATABASE_URL` and the heads from the project's `alembic.ini`: `SM010` is an error when the database lacks any head and a warning when it was never migrated or could not be read, and `SM011` reports a module table that a database at every head does not have. A development boot reports `SM011` too once its database is at head (or once the setup wizard has migrated it), and the Doctor screen keeps it on re-run. Doctor now resolves a relative SQLite `SM_DATABASE_URL` against the project root as the app does, so run from `host/` it reads the app's database for `SM033` too. Before, neither code was ever reported. **Behaviour change:** `make doctor` now exits 1 against a dev database that is behind head; run `make migrate` (#895).
 - `background_tasks`: `SM_BG_TASKS_BROKER_URL` and `SM_BG_TASKS_RESULT_BACKEND` are reported as deprecated only when they hold a Redis URL (`redis`, `rediss`), the only values `SM_REDIS_URL` can replace. A production host that runs tasks in-process with `memory://` + `cache+memory://`, or uses an AMQP, SQS or `rpc://` transport, no longer logs a warning on every boot that it cannot act on, and the production localhost-broker error no longer calls those vars deprecated for non-Redis values (#868).
 - `SecurityHeadersMiddleware` no longer overwrites a `Content-Security-Policy` that a response sets. The app policy is sent beside it as a second header and browsers enforce both, so a page can tighten the policy (for example `frame-ancestors 'none'` on a public page) but never relax it. Before, only a policy containing `sandbox` survived and any other was dropped silently. **Behaviour change:** a response that set a looser policy, relying on it being replaced, now has both policies enforced: remove that header, or make it a policy the page can live with alongside the app policy (#870).
 - `alembic check` and SQLite autogenerate no longer report the `file_storage` browse index (`sa.desc(sa.column("created_at"))` instead of `sa.text("created_at DESC")`) or `users_access_token.expires_at` (now `AwareDateTime`) as changed, so the host's `alembic check` on SQLite is clean. `make ci-doctor` now runs `alembic check` against its throwaway SQLite database, and a module index written with `sa.text("col DESC")` is no longer reported as changed on SQLite (#869).

docs/database/migrations.md

@@ -206,20 +206,20 @@ context.configure(
 
 On startup, the framework compares `alembic_version` to **every** head (`resolve_head_revisions` in `simple_module_hosting.migrations`; the history has several, so it never asks for a single head):
 
-- **Development:** prints a warning if the DB is behind. Result stored on `app.state.migration`.
-- **Production:** fails boot with `SM010`. Don't ship a web process that's pointed at an unmigrated DB.
+- **Every environment:** a database behind head fails the boot (`Database is N revision(s) behind … Run: make migrate`), except a never-migrated first run, which boots far enough to serve the `/setup` wizard. The result is stored on `app.state.migration`.
+- `make doctor` reports the same state as `SM010` before you boot. Don't ship a web process that's pointed at an unmigrated DB.
 
 Fix locally with `make migrate`. In production, run migrations before rolling over the web tier.
 
 ## Diagnostic: `SM011`
 
-Fires as a warning when a module's model declares a table that doesn't appear in any Alembic migration. Typical causes:
+Fires as a warning when a database at every head lacks a table a module's `models` declare, so no migration creates it. `make doctor` checks the database `SM_DATABASE_URL` points at; a development boot checks its own database once it is open. Typical causes:
 
 - You added a model but haven't generated a migration for it yet.
 - You renamed a table but the old migration still references the old name.
 - You used `__abstract__ = True` somewhere it shouldn't be.
 
-The dev-mode boot log prints the offending table names. Resolution: run `make migration msg="..."`, review, `make migrate`.
+`make doctor` and the dev-mode boot log print the offending table names; while the database is behind head the check is skipped, since the table may be in a pending revision. Resolution: run `make migration msg="..."`, review, `make migrate`.
 
 ## Cross-module foreign keys
 

docs/database/per-module-base.md

@@ -86,4 +86,4 @@ for t in meta.sorted_tables:
     print(t.name)
 ```
 
-This is also what the boot-time `SM011` check uses — it compares this set against the Alembic history to detect tables that exist in code but not in any migration.
+This is also what the boot-time `SM011` check uses — it compares this set against the migrated database's tables to detect tables that exist in code but that no migration creates.

docs/framework-conventions.md

@@ -626,8 +626,8 @@ Dispatch keys on the **exact** event class (`module.qualname`), not its MRO —
 | SM007 | INFO | Module defines no `register_*` hooks |
 | SM008 | ERROR | Duplicate module name / schema prefix conflict |
 | SM009 | ERROR | Framework package directly imports from a plugin module |
-| SM010 | ERROR | DB revision behind migration head |
-| SM011 | WARNING | Module table not in migration history |
+| SM010 | ERROR or WARNING | DB lacks a migration head (WARNING: never migrated) |
+| SM011 | WARNING | Module table missing from a database at head |
 | SM012 | WARNING | `register_settings` overridden but nothing added to `app.state` |
 | SM013 | WARNING | Locale file missing for a supported locale |
 | SM014 | WARNING | Non-default locale missing keys present in the default |

docs/reference/diagnostic-codes.md

@@ -18,8 +18,8 @@ The framework runs a set of static checks over installed modules at app boot. Th
 | `SM007` | INFO | Module overrides zero `register_*` hooks and has no `on_startup`/`on_shutdown`. | Delete the module if it's vestigial; otherwise ignore. |
 | `SM008` | ERROR | Two modules declare the same `ModuleMeta.name`. Table-prefix collision. | Rename one. Remember: the name is the table-name prefix (all modules share one DB schema) and the Inertia namespace. |
 | `SM009` | ERROR | A file under `framework/*` imports from any package under `modules/*`, including dynamic imports with literal names or fixed f-string package prefixes. | Invert the dependency: have the module register a callback into the framework (see `principal_serializer` pattern). |
-| `SM010` | ERROR | DB `alembic_version` is behind the migration head. | Run `make migrate`. The lifespan migration check raises and aborts boot in every environment before serving. |
-| `SM011` | WARNING | A module's model declares a table that doesn't appear in any Alembic migration. | Run `uv run alembic revision --autogenerate -m "..."`, review, `make migrate`. |
+| `SM010` | ERROR or WARNING | `make doctor` reads `alembic_version` from `SM_DATABASE_URL` and the heads from the project's `alembic.ini` (else `host/alembic.ini`). ERROR when the database lacks any head; WARNING when it was never migrated (a production boot sends that to `/setup`) or could not be read, in which case SM011 is skipped too. A SQLite file that does not exist yet counts as never migrated and is not created. | Run `make migrate`. The lifespan migration check also refuses a behind-head database at boot in every environment. |
+| `SM011` | WARNING | A table in a module's `models` is missing from a database at every head, so no migration creates it. Reported by `make doctor` and, once the database is open, a development boot (also on the Doctor screen). Not checked while the database is behind head. | Generate a revision on the module's branch (`make migration LABEL=<branch_label> msg="..."` in this repo, `make migration msg="..."` in a scaffolded app), review it, `make migrate`. |
 | `SM012` | WARNING | Module overrides `register_settings` but does not assign to `app.state.<module_lower>`. Dev-only. | Either remove the override or move your state assignment into it. |
 | `SM013` | WARNING | Supported locale has no corresponding file in some module's `locales/`. | Add `modules/<name>/<name>/locales/<locale>.json`, or drop the locale from `SM_I18N_SUPPORTED_LOCALES`. |
 | `SM014` | WARNING | Non-default locale is missing keys present in the default (untranslated). | Translate the missing keys. |
@@ -50,7 +50,7 @@ The framework runs a set of static checks over installed modules at app boot. Th
 
 | Context | What runs |
 |---|---|
-| App boot in development | Full structural/page/i18n suite, results logged to stderr. ERRORS abort boot. After the routes are wired, SM031 (unguarded `/api` routes) is added. |
+| App boot in development | Full structural/page/i18n suite, results logged to stderr. ERRORS abort boot. After the routes are wired, SM031 (unguarded `/api` routes) is added; once the database is open and at head, SM011. |
 | App boot in non-development | Strict module discovery (raises on SM001-class failures) + the migration check (SM010) + SM025 (logged as a warning). The page/locale static suite is dev-only. |
 
 Sample dev-mode output:
@@ -60,7 +60,7 @@ running diagnostics …
 
 WARNING SM003  modules/orders/orders/pages/Unused.tsx: no inertia.render("Orders/Unused") call found
 WARNING SM013  locale "es" missing file for module "orders" (expected modules/orders/orders/locales/es.json)
-ERROR   SM010  DB revision 3a1b2c3d4e5f is behind migration head 9f8e7d6c5b4a — run `make migrate`
+ERROR   SM010  Database is behind: missing heads '9f8e7d6c5b4a' — run `make migrate`
 
 1 error, 2 warnings
 ```

docs/superpowers/plans/2026-11-10-doctor-migration-checks.md

@@ -0,0 +1,46 @@
+# Doctor migration checks (#895) — implementation plan
+
+Design: `docs/superpowers/specs/2026-11-10-doctor-migration-checks-design.md`.
+
+Tasks run in order; 1 and 2 are independent of each other.
+
+1. **`MigrationDiagnostics.check_heads` + `module_table_names`** (core)
+   - Files: `framework/core/simple_module_core/diagnostics/_migration.py`,
+     `framework/core/tests/test_diagnostics.py`.
+   - Test first: at heads → `[]`; no current → SM010 WARNING; one of several
+     heads missing → SM010 ERROR naming it; an extra unknown current revision
+     with every head present → `[]`.
+   - Done: targeted tests pass.
+
+2. **`simple_module_db.migration_state.read_migration_state`** (db)
+   - Files: `framework/db/simple_module_db/migration_state.py` (new; also
+     takes over `_database_exists` from `data_compatibility.py`),
+     `framework/db/tests/test_db_migration_state.py` (new).
+   - Test first: no ini → `None`; missing SQLite file → empty current and
+     tables, file not created; unstamped file → empty current, its tables;
+     stamped file → current revisions.
+   - Done: targeted tests pass.
+
+3. **Doctor wiring** (hard: cross-package)
+   - Files: `framework/core/simple_module_core/_migration_doctor.py` (new),
+     `framework/core/simple_module_core/__main__.py`,
+     `framework/core/tests/test_core_migration_doctor.py` (new).
+   - Test first, through `main()` with a temp project root holding an
+     `alembic.ini` and two revisions: DB at heads with a module table missing
+     → SM011, exit 0; DB behind → SM010, exit 1; missing DB → SM010 warning,
+     exit 0; unreachable DB → one warning.
+   - Done: targeted tests + existing doctor tests pass.
+
+4. **Dev boot SM011**
+   - Files: `framework/hosting/simple_module_hosting/_table_coverage.py` (new),
+     `framework/hosting/simple_module_hosting/_lifespan.py`,
+     `framework/hosting/tests/test_hosting_table_coverage.py` (new).
+   - Test first: with a runner set and a table missing, results gain SM011 and
+     `rerun()` keeps it; with no runner nothing runs.
+   - Done: targeted tests pass.
+
+5. **Docs + CHANGELOG**: `docs/reference/diagnostic-codes.md` (SM010/SM011
+   rows, "When diagnostics fire"), `CHANGELOG.md` Fixed.
+
+6. **Full checks**: `make lint`, `make test-py` (PYTEST_WORKERS=8), `make
+   ci-doctor`.

docs/superpowers/specs/2026-11-10-doctor-migration-checks-design.md

@@ -0,0 +1,104 @@
+# Doctor runs the migration checks SM010 and SM011 (#895)
+
+## Goal & scope
+
+`smpy doctor` / `make doctor` never report `SM010` (database behind head) or
+`SM011` (a module table no migration creates), although `CLAUDE.md` and
+`docs/reference/diagnostic-codes.md` promise both. `run_diagnostics()` runs them
+only when it is handed a `migration_state`, and neither caller (doctor in
+`simple_module_core/__main__.py`, the dev boot in
+`simple_module_hosting/_dev_boot.py`) passes one. A new module's table can go
+unmigrated with no warning.
+
+In scope: doctor reports both codes; the development boot (and so the in-app
+Doctor screen) reports `SM011`; tests; docs; CHANGELOG.
+Out of scope: changing what a production boot refuses; `alembic check`-style
+column drift (CI already runs `alembic check`, #869).
+
+## Approach
+
+Chosen: **read the state from the database doctor already points at.**
+Doctor reads `alembic_version` and the table list from `SM_DATABASE_URL`, and
+the heads from the host's `alembic.ini`. When the database is at every head,
+any table a selected module's models declare that the database does not have
+is a table no migration creates: `SM011`. Behind head is `SM010`.
+
+Rejected:
+- *Scan the revision files for `op.create_table(...)`* (what the issue
+  sketched). It misses `rename_table`, batch operations, tables built in
+  helpers and raw SQL, so it would raise false `SM011`s on this repo's own
+  history; the database is the ground truth for "what the migrations create".
+- *Migrate a throwaway SQLite database inside doctor.* Accurate, but slow on
+  every run, and `make ci-doctor` already does exactly this before calling
+  doctor, so the database approach gets the same answer in CI for free.
+
+## Design
+
+**`simple_module_db.migration_state`** (new; db owns Alembic, core may not
+import hosting):
+`async def read_migration_state(database_url, alembic_ini) -> MigrationState | None`
+returns `heads`, `current` (plural `get_current_heads()`, like
+`migration_status`) and `tables` (`inspect(...).get_table_names()`), or `None`
+when there is no `alembic.ini` / no revisions. It never creates a database: a
+SQLite file URL whose file does not exist yields an empty state instead of
+connecting (connecting would create `app.db`).
+
+**`MigrationDiagnostics.check_heads(current, heads)`** (core, new beside the
+existing `check_revision_mismatch`, which stays for `run_diagnostics`
+callers): the same rule the boot uses (`heads - current`):
+- at every head → nothing;
+- never migrated (no `alembic_version` rows) → `SM010` **WARNING**, "run
+  `make migrate`" (a production boot sends this database to `/setup` rather
+  than refusing it, so it is not an error);
+- behind on some branch → `SM010` **ERROR** naming the missing heads (a
+  production boot refuses this).
+`check_table_coverage` (existing) gives `SM011`, run only when at every head.
+
+**Doctor** (`simple_module_core/__main__.py` → a `_migration_doctor.py` helper
+like `_data_doctor.py`): skips silently when `simple_module_db` is not
+installed; resolves `alembic.ini` as the CLI does (project root `alembic.ini`,
+else `host/alembic.ini`); module tables come from `module_all_tables()` over the
+selected modules (the set SM026 already uses). A database it cannot reach (a
+stopped Postgres) gives one `SM010` WARNING saying the checks were skipped,
+not a crash.
+
+**Dev boot**: the diagnostics runner runs at build time, before the database
+is open, so `SM011` is computed in the lifespan instead: in development, once
+`migration_status` reports the database at head, the lifespan reads the
+table list from the app's own engine, prints any `SM011`, and wraps
+`diagnostics_state.runner` so the in-app Doctor screen (which already maps
+`SM010`/`SM011` to its migrations check) shows them on every re-run. `SM010` at
+boot is already enforced by the lifespan's own refusal.
+
+## Decisions made
+
+- Never-migrated is a WARNING, behind is an ERROR (mirrors production boot).
+- Unreachable database: WARNING, migration checks skipped; doctor still exits
+  on the other findings.
+- `SM011` only when at every head; behind head the missing table may simply be
+  in a pending revision.
+- Module tables = tables in each selected module's `models` (as SM026). Tables
+  declared outside `models` are not checked (missed, never falsely reported).
+- No new setting, no new code; docs: `docs/reference/diagnostic-codes.md`
+  (when each fires, that doctor reads `SM_DATABASE_URL`), CHANGELOG Fixed.
+
+## Testing
+
+- `check_heads`: at heads, never migrated, behind (multi-head), extra unknown
+  revision is not "behind".
+- `read_migration_state`: a SQLite file stamped at heads (the test plugin's
+  create-all + stamp), an unstamped one, a missing file is not created.
+- Doctor `main()` end to end: a database at heads with one module table
+  dropped reports `SM011` and exits 0 (warning); a database stamped one
+  revision behind reports `SM010` and exits 1; a missing DB file warns.
+- Dev boot: the lifespan helper adds `SM011` to `diagnostics_state` and
+  `rerun()` keeps it.
+No browser QA (`--skip-browser`): CLI and boot diagnostics only.
+
+## Risks
+
+- `make doctor` on a developer checkout whose dev database is behind will now
+  exit 1. That is the point of SM010, and the message says `make migrate`.
+- `make ci-doctor` migrates first, so CI should stay green; if this repo's own
+  modules have a table no migration creates, doctor will now say so and the
+  fix is a migration (that would be a real bug the check exists to find).

framework/core/simple_module_core/__main__.py

@@ -25,6 +25,7 @@
 from pathlib import Path
 
 from simple_module_core._data_doctor import data_compatibility_diagnostics
+from simple_module_core._migration_doctor import migration_diagnostics
 from simple_module_core.diagnostics import (
     Diagnostic,
     DiagnosticLevel,
@@ -39,7 +40,7 @@
     select_auth_provider,
     topological_sort,
 )
-from simple_module_core.dotenv import find_env_file, parse_dotenv
+from simple_module_core.dotenv import absolutize_sqlite_url, find_env_file, parse_dotenv
 from simple_module_core.exceptions import InvalidModuleError
 
 
@@ -77,8 +78,13 @@ def _load_i18n_settings_from_env() -> tuple[list[str] | None, str]:
 
 
 def _database_url() -> str:
-    """``SM_DATABASE_URL``, defaulting like ``Settings.database_url``."""
-    return os.environ.get("SM_DATABASE_URL", "sqlite+aiosqlite:///./app.db")
+    """``SM_DATABASE_URL``, defaulting and anchored like ``Settings.database_url``.
+
+    A relative SQLite path is resolved against the project root, as the app
+    resolves it, so doctor run from ``host/`` reads the app's database.
+    """
+    url = os.environ.get("SM_DATABASE_URL", "sqlite+aiosqlite:///./app.db")
+    return absolutize_sqlite_url(url, anchor=find_env_file().parent)
 
 
 def _database_dialect() -> str:
@@ -188,6 +194,7 @@ def main() -> int:
         database_dialect=_database_dialect(),
     )
     diagnostics.extend(override_findings)
+    diagnostics.extend(migration_diagnostics(modules, _database_url(), find_env_file().parent))
     diagnostics.extend(data_compatibility_diagnostics(modules, _database_url()))
     # A missing dependency only warns at a dev boot, but production refuses
     # to start with it (GH #501), so doctor reports it as what it will be.