Add a sandbox that exercises every ucode configure path end to end - #440
Open
david-siqi-liu wants to merge 1 commit into
Open
Add a sandbox that exercises every ucode configure path end to end#440david-siqi-liu wants to merge 1 commit into
ucode configure path end to end#440david-siqi-liu wants to merge 1 commit into
Conversation
This was referenced Sep 1, 2026
david-siqi-liu
marked this pull request as ready for review
September 1, 2026 20:43
david-siqi-liu
force-pushed
the
david/configure-managed-flow
branch
from
September 1, 2026 23:28
f95af9c to
f6d295b
Compare
david-siqi-liu
force-pushed
the
david/configure-sandbox
branch
from
September 1, 2026 23:29
7e5df91 to
5a346ab
Compare
The configure flow branches on three axes at once: whether the managed-config opt-in and backend are live, whether the workspace publishes a config, and whether the caller is an admin (including the case where the SCIM check cannot say). The unit tests cover each branch with the collaborators stubbed, which is where the branch logic belongs, but nothing exercised a whole run: the state file it leaves behind, the settings it writes, the order the messages come out in, or whether the server was touched at all. Add a sandbox that runs the real CLI in a subprocess per scenario, inside a throwaway HOME, against a fake workspace. Only three seams are faked: the HTTP layer in `ucode.databricks`, the subprocesses that shell out to the databricks CLI or an agent binary, and the questionary prompts, which are driven by a scripted answer queue that fails loudly on an unexpected question. Everything above them (cli, managed_wizard, managed_config, managed_publish, agents/*, config_io, state) runs for real, and any endpoint the fake does not model is reported instead of quietly degrading. The 27 scenarios cover the gates, a published config for each persona, an empty workspace, the draft lifecycle, the pre-v2 migration, and the admin-only sections. `draft/survives-a-launch` is the load-bearing one: it fails if the draft and published slots are ever collapsed back into one, since a launch has to overwrite the fetched snapshot while leaving the admin's authored draft alone. Expectations live in `tests/test_configure_sandbox.py`, which drives the table concurrently and adds about two seconds to the suite. Verified by mutation: dropping the apply of a freshly authored draft fails four scenarios, and making a published write clobber the draft slot fails twelve. Co-authored-by: Isaac <no-reply@databricks.com>
david-siqi-liu
force-pushed
the
david/configure-managed-flow
branch
from
September 2, 2026 00:39
f6d295b to
3709e5b
Compare
david-siqi-liu
force-pushed
the
david/configure-sandbox
branch
from
September 2, 2026 00:39
5a346ab to
e4bd556
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🥞 Stack (AIGTWY-4342 ucode configure)
ucode setupwith aucode configure-centered workflow #433ucode configurepath end to end #440 ⬅ this PRWhat did you change, and why?
The configure flow spans admin detection, authoring, publish, apply, and per-agent file writes, and
the unit tests mock most of that away. This adds a sandbox that runs the real CLI against a fake
Databricks workspace inside a throwaway HOME, so the file writes, the state transitions, and the
printed output under assertion are the real ones.
draft,published,empty,gate,multi,section). Each runsin its own subprocess with a fresh HOME, fresh module state, and fresh caches.
ucode.databricksand the questionary prompts inucode.ui. Everything above them (cli, managed_wizard, managed_config, managed_publish, agents,config_io, state) executes for real.
a flow that reaches an unmodelled endpoint fails loudly instead of degrading quietly.
src/.tests/test_configure_sandbox.py. That is the actualspecification of what each path should do; the rest is plumbing.
How do you know it works?
uv run pytest tests/test_configure_sandbox.pycollects 28 tests: the 27 scenarios plus a guardthat fails if a scenario is added without expectations. Green locally, and it runs as part of the
default suite in CI. To drive one scenario by hand:
python -m tests.sandbox_scenarios <name>.