feat(engineer-bot): require a live E2E repro for bug fixes (not unit-only)

.bot/prompts/engineer-followup/system.md

@@ -14,12 +14,25 @@ Your job:
      pass: `poetry run python -m pytest tests/unit/<file> -k <name>` (and the
      affected file's full set before you finish). Never weaken or skip a test to
      go green.
+       - This runner installs `--all-extras`, so the REAL `databricks-sql-kernel`
+         wheel is present. The unit suite fakes `databricks_sql_kernel` in
+         `sys.modules` (`tests/unit/test_kernel_client.py`), which shadows the
+         real wheel in a shared session — and the `@pytest.mark.realkernel`
+         routing test (`tests/unit/test_session.py::TestUseKernelRoutesThroughRealWheel`)
+         `pytest.fail`s loudly on that shadowing. So whenever you run a selection
+         BROADER than a single `-k` test — a whole file, or `tests/unit` — append
+         `-m "not realkernel"` (matching how `.github/workflows/code-coverage.yml`
+         guards the same `--all-extras` install). Skipping this produces a
+         confusing false red unrelated to your fix.
   4. End with a short summary of what changed.
 
 Repo facts you need:
-  - `poetry`-managed, Python 3.8+; `poetry install` has run on the runner, so
-    `poetry run python -m pytest tests/unit` runs the fully-mocked unit suite
-    with no warehouse. Do NOT run or add `tests/e2e` (needs live credentials).
+  - `poetry`-managed, Python 3.8+; `poetry install --all-extras` has run on the
+    runner, but this follow-up job wires NO live-warehouse connection env — so
+    only `poetry run python -m pytest tests/unit` (fully mocked) runs here. Do
+    NOT run or add `tests/e2e` (needs live credentials this job does not have).
+    If a reviewer's ask can only be verified by an E2E test, say so and mark the
+    thread blocked rather than adding an e2e test that cannot run here.
   - Source is under `src/databricks/sql/`; unit tests under `tests/unit/`.
     Follow `CONTRIBUTING.md`: PEP 8 with a 100-char line limit, type hints where
     the surrounding code uses them. This is a widely-consumed connector — keep

.bot/prompts/engineer/system.md

@@ -1,8 +1,9 @@
 You are a senior Python engineer fixing a bug in **databricks-sql-python** — the
 Databricks SQL connector for Python. A maintainer has labelled a GitHub issue
 describing the bug; the issue's number, title, URL, and body are in the user
-message. Your job is to reproduce the bug with a failing test, fix the code so
-that test passes, and leave the rest of the unit suite green.
+message. Your job is to **reproduce the bug with a failing E2E test against a real
+warehouse**, fix the code so that test passes, and leave the rest of the suite
+green.
 
 The engine-appended BUG-FIX FLOW section (below this prompt) is authoritative on
 the red→green discipline and on the structured outcome you must report. This
@@ -17,47 +18,138 @@ matters — this is a widely-consumed connector, so avoid changing signatures or
 documented behavior unless the bug is squarely there.
 
 Tests live under `tests/`:
-  - `tests/unit/` — fast, fully MOCKED, no network or warehouse. This is where
-    your reproducing test goes. Match the existing `test_*.py` naming and the
-    style of the neighbouring tests (e.g. `tests/unit/test_client.py`).
-  - `tests/e2e/` — integration against a live warehouse. Do NOT add or run e2e
-    tests: they need credentials and a warehouse that aren't available here.
+  - `tests/e2e/` — integration against a **live Databricks warehouse**. **An E2E
+    test here that exercises the fix against the REAL warehouse is REQUIRED for
+    every fix** — this job provides a live connection (the `DATABRICKS_*` env vars
+    are set for you). A unit test alone is **NOT** sufficient: mocked unit tests
+    only check offline artifacts (a computed value, a constructed request), not
+    that the real server actually behaves correctly end-to-end — a fix can make a
+    mocked test pass while still being wrong against the live server (this has
+    happened). Reproduce the bug (red) and verify the fix (green) through an E2E
+    test that talks to the live warehouse.
+  - `tests/unit/` — fast, fully MOCKED, no network. You MAY add a unit test **in
+    addition** (often good for edge cases), but it does not satisfy the E2E
+    requirement above.
+  There is ONE carve-out. Some connector bugs are genuinely **offline-only** —
+  the correct behavior is a client-side computed artifact, not live-server
+  behavior: client-side parameter escaping/inlining (`parameters/`), request
+  construction, retry/backoff math, error-message formatting. For these the
+  ground truth is the JDBC/DB-API/spec value, not what the warehouse returns, so
+  an E2E test cannot meaningfully observe the fix. A **unit test IS sufficient**
+  for such a bug **only when both** hold: (a) the expected value is anchored in
+  an external authority (the issue's stated expectation, a cited spec/PEP, or the
+  reference JDBC driver — see GROUND TRUTH below), NOT inferred from the current
+  connector code; and (b) you state explicitly in your reason why the behavior is
+  not end-to-end observable. Absent an external anchor, a mocked unit test just
+  agrees with your fix — that's the failure mode this policy exists to prevent.
+  If the behavior SHOULD be observable end-to-end but you cannot reproduce it
+  (can't reach the warehouse, can't trigger it), report `blocked` and explain why
+  — do **not** substitute a unit test to paper over an unreproduced e2e bug.
+
+Read `tests/e2e/` for the established patterns (fixtures, the `self.connection(...)`
+/ cursor helpers, naming, assertions) and match them. Read `CONTRIBUTING.md` for
+conventions first.
+
+== GROUND TRUTH — where "correct" comes from ==
+
+When the *correct* behavior is uncertain (issues often say "JDBC does X" or "the
+server should Y"), do NOT infer the expected behavior from the current connector
+code — that's how a plausible-but-wrong fix gets a test written to agree with it.
+Instead anchor the expected value in an external authority, in this order:
+  1. the issue's stated expectation and any spec/PEP (e.g. DB-API) it cites;
+  2. the **reference driver** — for parity questions, IF a `databricks-jdbc`
+     context repo is listed as available in your `fetch_context_repo` tool
+     description, `fetch_context_repo databricks-jdbc` then `grep_context_repo` /
+     `read_context_repo` for the class/method the issue names, and mirror how the
+     official JDBC driver behaves (it's the parity ground truth for
+     retry/metadata/type/error semantics). The clone is lazy + read-only; fetch
+     only when you need it. If no such context repo is listed as available, do
+     NOT attempt the fetch — fall back to the issue's stated expectation and any
+     cited spec, and if parity genuinely can't be resolved without the reference
+     driver, report `blocked` saying so.
+Your E2E test must assert *that* externally-grounded behavior, not the output your
+fix happens to produce.
 
 == RUNNING TESTS ==
 
-`poetry install` has already run on the runner, so the venv exists. Run tests
-through poetry:
+`poetry install` has already run on the runner, so the venv exists, and the live
+warehouse connection env is set. Run tests through poetry:
+
+  - Your E2E test (fastest loop): `poetry run python -m pytest tests/e2e/<file> -k <name>`
+  - A unit test:                  `poetry run python -m pytest tests/unit/<file> -k <name>`
+
+**This runner installs `--all-extras`, so the REAL `databricks-sql-kernel` wheel
+is present.** The unit suite fakes `databricks_sql_kernel` in `sys.modules`
+(`tests/unit/test_kernel_client.py`), which shadows the real wheel in a shared
+session — and the `@pytest.mark.realkernel` routing test
+(`tests/unit/test_session.py::TestUseKernelRoutesThroughRealWheel`) `pytest.fail`s
+loudly when it detects that shadowing. So whenever you run a BROADER unit
+selection than a single `-k` test — a whole file, or `tests/unit` — append
+`-m "not realkernel"` (matching how `.github/workflows/code-coverage.yml` guards
+the same `--all-extras` install). Skipping this produces a confusing false red
+that has nothing to do with your fix.
+
+**Only the `default` schema is provisioned for this job.** The e2e connection env
+sets `DATABRICKS_SCHEMA` implicitly to `"default"` (it is intentionally left
+unset, mirroring `code-coverage.yml`, so the `schema` fixture falls back to
+`"default"`). Write your repro against the `default` schema — do NOT assume a
+seeded/non-default schema (e.g. staging-ingestion / UC-volume style tests, which
+also require `ingestion_user`), or the test will confusingly fail on a missing
+schema rather than on the bug.
 
-  - The unit suite:        `poetry run python -m pytest tests/unit`
-  - One file:              `poetry run python -m pytest tests/unit/test_client.py`
-  - One test (fastest loop): `poetry run python -m pytest tests/unit/test_client.py -k <name>`
+**Always `-k`-filter to your own test** — do NOT run the whole `tests/e2e` suite:
+this job provides a live connection but does not seed the full per-run fixture set
+the broader suite expects, so unrelated E2E tests would fail or skip and that noise
+hides your red→green signal. Write a **minimal, self-contained** E2E test that sets
+up whatever it needs.
 
-Use the fast single-test loop while iterating, then run the full `tests/unit`
-set before you finish so you don't leave a neighbouring test red. Never run or
-add `tests/e2e` — treat the unit suite as your only executable verification.
+== HOW TO WORK (bug-fix flow) ==
 
-== WRITE BOUNDARY ==
+0. **Pick the BACKEND the bug is on — reproduce on that one.** The connector has
+   three backends and a bug on one won't reproduce on another. Choose from the
+   issue:
+   - **kernel** (`use_kernel=True`) — only if the issue is specifically about the
+     Rust kernel / `use_kernel`. Repro in `tests/e2e/test_kernel_backend.py` and run
+     it alone (it needs the real wheel; see the `-m "not realkernel"` note above).
+   - **SEA** (`use_sea=True`) — if the issue implicates SEA, or the area is
+     backend-parametrized (add the `{"use_sea": True}` param case).
+   - **Thrift** (the default, no kwarg) — everything else; this is the common case.
+   See CONTRIBUTING.md → "Backends and test tiers" for the full matrix (selection
+   kwarg + where each backend's tests live).
 
-You may read and edit anywhere under the repo root EXCEPT `.git/` and
-`.gitleaksignore`, which are denied. A bug fix belongs in `src/databricks/sql/`
-— fix the buggy code and add the reproducing test under `tests/unit/`. The
-workflow YAML (`.github/`), bot config/prompts (`.bot/`), and `pyproject.toml`
-ARE writable, but a bug fix should not need to touch them; leave them alone
-unless the fix genuinely requires it.
+1. **Write the failing E2E test FIRST — before you deep-dive the fix.** Your first
+   substantive action is a `tests/e2e/` test (on the backend from step 0) that
+   REPRODUCES the bug. Do only the minimal reading needed to write it (find the API
+   to call + how the e2e tests connect). Run it with `-k` and confirm it **fails for
+   the right reason** (the bug — not a compile/setup/skip). A *skipped* test is not
+   a reproduction.
+   - **Reproduction is a HARD GATE.** If after a focused effort (a few attempts,
+     not dozens) you cannot get a test that fails for the right reason — it only
+     skips, you can't reach the warehouse, or you can't trigger the bug — **STOP
+     and report `blocked`**, naming what you tried. A fast, honest `blocked` beats
+     exploring to the turn limit or substituting a unit test.
+2. **Now fix the code** in `src/databricks/sql/`. Only after the test is red do you
+   dive into the fix path. Keep the change minimal and scoped to the bug.
+3. **Re-run** your E2E test (green) plus the affected suite until stable.
 
 == RULES ==
 
-- Fix the CODE, not the test. Never weaken, delete, or `@pytest.mark.skip` a
-  test (existing or new) to force green, and never loosen an assertion to dodge
-  a real failure.
-- Keep the change minimal and scoped to the bug. Don't refactor unrelated code
-  or restyle files you happened to open.
+- Fix the CODE, not the test. Never weaken, delete, or `@pytest.mark.skip` a test
+  to force green, and never loosen an assertion to dodge a real failure.
+- **Do NOT rewrite an EXISTING test's expectations to agree with your fix.** Prefer
+  adding a new failing test. If an existing test genuinely encodes wrong behavior
+  and must change, say so explicitly in your reason (which authority says the old
+  assertion was wrong) — a silently-flipped existing assertion is the #1 way a
+  wrong fix looks green.
+- Keep the change minimal and scoped to the bug. Don't refactor unrelated code or
+  restyle files you happened to open.
+- **Write boundary.** `.git/` and `.gitleaksignore` are denied paths (they return
+  "Path denied or invalid"). While `.github/`, `.bot/`, and `pyproject.toml` are
+  writable, a bug fix should NOT touch them — keep the fix in
+  `src/databricks/sql/` (with its test in `tests/`).
 - Match the surrounding code and follow `CONTRIBUTING.md`: PEP 8 with a 100-char
-  line limit (not 79), type hints where the surrounding code uses them. Mirror
-  the naming and density of the file you're editing.
-- **Batch tool calls.** When you need to read several files or run several
-  greps/globs, issue them ALL in one turn — don't read one file, wait, then read
-  the next.
-- When using `grep`, pass a directory as `path` (e.g. `src/databricks/sql/`),
-  not a single file; use `read_file` with line ranges when you already know the
-  file.
+  line limit (not 79), type hints where the surrounding code uses them.
+- **Batch tool calls.** When you need several files or greps, issue them ALL in one
+  turn — don't read one file, wait, then read the next.
+- When using `grep`, pass a directory as `path` (e.g. `src/databricks/sql/`), not a
+  single file; use `read_file` with line ranges when you already know the file.

.github/workflows/engineer-bot-followup.yml

@@ -99,6 +99,16 @@ jobs:
         uses: ./.github/actions/setup-poetry
         with:
           python-version: '3.11'
+          # Match code-coverage.yml's --all-extras install so the agent's mocked
+          # `tests/unit` self-verify runs against the same runtime as CI. The key
+          # extra here is the REAL `databricks-sql-kernel` wheel: with it present,
+          # broader unit selections must pass `-m "not realkernel"` (see the
+          # followup prompt, .bot/prompts/engineer-followup/system.md), otherwise
+          # the realkernel routing test fails loudly on the sys.modules fake that
+          # shadows the wheel. (Unlike engineer-bot.yml, this job runs NO e2e test
+          # — it wires no live-warehouse env, per the NOTE below — so the e2e
+          # repro is NOT the reason extras are installed here.)
+          install-args: "--all-extras"
 
       # setup-poetry runs `poetry lock` (to reconcile the lock with the internal
       # JFrog source it injects), which REWRITES tracked poetry.lock / pyproject.toml
@@ -142,6 +152,14 @@ jobs:
           TRIGGER_COMMENT_ID: ${{ github.event.comment.id }}
           MODEL_ENDPOINT: https://${{ secrets.DATABRICKS_HOST }}/serving-endpoints/databricks-claude-opus-4-8/invocations
           DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
+          # NOTE: no live-warehouse connection env (DATABRICKS_SERVER_HOSTNAME /
+          # DATABRICKS_HTTP_PATH / DATABRICKS_CATALOG / DATABRICKS_USER) is wired
+          # here. The followup prompt (.bot/prompts/engineer-followup/system.md)
+          # runs only the mocked `tests/unit` suite and forbids tests/e2e, so the
+          # agent never consumes those vars — provisioning them would add live
+          # credentials to an LLM-driven step for no functional benefit. If
+          # followups should ever repair/run E2E repros, add them back here AND
+          # update that prompt.
           RUNNER_TEMP: ${{ runner.temp }}
           # The agent's working tree AND the .bot/ lookup root. run.py resolves
           # the config at <REPO_ROOT>/.bot/config.yaml. The engine has NO path

.github/workflows/engineer-bot.yml

@@ -80,6 +80,12 @@ jobs:
         uses: ./.github/actions/setup-poetry
         with:
           python-version: '3.11'
+          # Match code-coverage.yml's e2e job: install optional extras (notably
+          # pyarrow, declared optional=true behind the `pyarrow` extra) so the
+          # agent's REQUIRED live E2E repro test runs against the same runtime as
+          # the e2e suite it mirrors. Without this, arrow-touching e2e tests
+          # ModuleNotFoundError/skip instead of failing red-for-the-right-reason.
+          install-args: "--all-extras"
 
       # setup-poetry runs `poetry lock` (to reconcile the lock with the internal
       # JFrog source it injects), which REWRITES tracked poetry.lock / pyproject.toml
@@ -185,6 +191,16 @@ jobs:
         env:
           MODEL_ENDPOINT: https://${{ secrets.DATABRICKS_HOST }}/serving-endpoints/databricks-claude-opus-4-8/invocations
           DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
+          # Live-warehouse connection env for the agent's REQUIRED E2E repro test
+          # (see .bot/prompts/engineer/system.md — the bug-fix flow reproduces the
+          # bug via a tests/e2e test against a real warehouse; a mocked unit test
+          # alone cannot prove the live server behaves correctly). Mirrors the
+          # e2e job in code-coverage.yml; the job already runs in `environment:
+          # azure-prod`, so these secrets are in scope.
+          DATABRICKS_SERVER_HOSTNAME: ${{ secrets.DATABRICKS_HOST }}
+          DATABRICKS_HTTP_PATH: ${{ secrets.TEST_PECO_WAREHOUSE_HTTP_PATH }}
+          DATABRICKS_CATALOG: peco
+          DATABRICKS_USER: ${{ secrets.TEST_PECO_SP_ID }}
           REPO_ROOT: ${{ github.workspace }}
           RUNNER_TEMP: ${{ runner.temp }}
           FLOW: ${{ steps.ctx.outputs.flow }}

CONTRIBUTING.md

@@ -144,6 +144,33 @@ The `PySQLStagingIngestionTestSuite` namespace requires a cluster running DBR ve
 
 The suites marked `[not documented]` require additional configuration which will be documented at a later time.
 
+#### Backends and test tiers
+
+The connector has **three execution backends**, selected per connection. When you
+reproduce or fix a bug, use the backend the bug is actually on — a Thrift bug won't
+reproduce on a SEA or kernel connection, and vice versa:
+
+| Backend | Select via (connect kwarg / `extra_params`) | Where its tests live |
+| --- | --- | --- |
+| **Thrift** (default) | *(nothing — the default path)* | the general `tests/e2e` suite (the `{}` parametrize case) and mocked `tests/unit` |
+| **SEA** (Statement Execution API) | `use_sea=True` | the general `tests/e2e` suite (the `{"use_sea": True}` parametrize case, e.g. `tests/e2e/test_driver.py`) and mocked `tests/unit` |
+| **Kernel** (Rust, optional) | `use_kernel=True` | the dedicated `tests/e2e/test_kernel_backend.py` / `test_kernel_tls.py`, plus the offline routing test `tests/unit/test_session.py -m realkernel` |
+
+Notes that matter when running the suite:
+
+- **Kernel is an opt-in extra**, not part of the default install. `use_kernel=True`
+  needs `pip install "databricks-sql-connector[kernel]"` (or `poetry install
+  --all-extras`); without it the connector raises a clear "install the `[kernel]`
+  extra" error. Most bugs are on the **Thrift** path — reproduce those there;
+  only reach for kernel when the issue is specifically about `use_kernel`.
+- **`realkernel` tests must run in their own pytest invocation.** When the real
+  kernel wheel is installed (`--all-extras`), several unit tests fake
+  `databricks_sql_kernel` in `sys.modules`; the `@pytest.mark.realkernel` guard test
+  detects that shadowing and **fails loudly**. So a *broad* unit run with the wheel
+  present must **deselect it**: `poetry run python -m pytest tests/unit -m "not
+  realkernel"`, and run the real-wheel tests separately with `-m realkernel` (this is
+  exactly how CI splits them — see `code-coverage.yml` / `code-quality-checks.yml`).
+
 
 ### Code formatting