Skip to content

test: stop config-mutating project tests from sharing a project - #455

Open
alnr wants to merge 2 commits into
masterfrom
fix/test-project-contention
Open

test: stop config-mutating project tests from sharing a project#455
alnr wants to merge 2 commits into
masterfrom
fix/test-project-contention

Conversation

@alnr

@alnr alnr commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Problem

Five test functions in cmd/cloudx/project run in parallel and all write project configuration to the same two shared fixtures, defaultProject and extraProject. Two pairs conflict directly:

Test Writes
TestPatchProject /services/identity/config/selfservice/flows/error = {"ui_url":"https://example.org/error-ui"}
TestPatchKratosConfig /selfservice/flows/error/ui_url = "https://example.com/error-ui"

Those are the same key underneath, with different values, on the same project.

Test Writes
TestPatchPermissionConfig /namespaces = legacy definitions
TestUpdateNamespaceConfig /namespaces = an OPL location

Whichever landed last won, so assertions failed intermittently and read as staging flakiness. This is what took down #451 twice — TestPatchKratosConfig/is_able_to_add_a_key_with_string failing with Not equal, and separately a get opl round-trip finding no OPL on a project that had just been given one.

patch_permission_config_test.go already carried a comment noting that its own subtests share a project. The gap was cross-test contention, which nothing guarded.

Fix

Each config-mutating test now gets its own project via newProject(t), leaving defaultProject and extraProject as read-only fixtures for the tests that only read (get_test.go, create_test.go, use_test.go — all verified to assert existence or the local config file, never mutated config).

Subtests within one function still share that function's project, which is safe as long as each performs a single operation and asserts on the response to it — the invariant the permission test already documented, now stated on the helper.

The project goes in a workspace of its own. My first attempt put it in defaultWorkspaceID and failed with:

The requested action was forbidden because the quota for the feature 'Development Projects' has been exceeded

The development-project quota is per workspace, which is why TestUpdateProject and TestListProject already create a workspace before their projects.

Verification

Ran against live staging, not just compiled:

  • The five affected tests plus the read-only TestGetProject / TestGetServiceConfig: pass.
  • The full cmd/cloudx/project package: pass, 36s (no slower than before — the tests stay parallel, they just no longer collide).
  • go vet and golangci-lint clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JYzGVwAKQ4ormxRHZg1eDu

Five test functions in this package run in parallel and all wrote
project configuration to the same two shared fixtures, defaultProject
and extraProject. Two pairs conflict directly:

  - TestPatchProject replaces /services/identity/config/selfservice/
    flows/error with an object, while TestPatchKratosConfig sets
    /selfservice/flows/error/ui_url — the same key underneath, with a
    different value.
  - TestPatchPermissionConfig replaces /namespaces with legacy
    definitions, while TestUpdateNamespaceConfig sets it to an Ory
    Permission Language location.

Whichever landed last won, so the assertions failed intermittently in
CI and looked like staging flakiness.

Each config-mutating test now gets its own project, leaving
defaultProject and extraProject as read-only fixtures for the tests
that only read. Subtests within one function still share that project,
which is safe as long as each performs a single operation and asserts
on the response to it — the invariant patch_permission_config_test.go
already documented.

The project is created in a workspace of its own because the
development-project quota is per workspace; adding to the shared
workspace fails with "the quota for the feature 'Development Projects'
has been exceeded". TestUpdateProject and TestListProject already
create a workspace for the same reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JYzGVwAKQ4ormxRHZg1eDu
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@alnr, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 10 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a512830f-064a-4100-9f21-1095cbbbb85f

📥 Commits

Reviewing files that changed from the base of the PR and between 6b55b72 and a210f3b.

📒 Files selected for processing (8)
  • cmd/cloudx/project/helper_test.go
  • cmd/cloudx/project/main_test.go
  • cmd/cloudx/project/patch_identity_config_test.go
  • cmd/cloudx/project/patch_oauth2_config_test.go
  • cmd/cloudx/project/patch_permission_config_test.go
  • cmd/cloudx/project/patch_test.go
  • cmd/cloudx/project/update_namespace_config_test.go
  • cmd/cloudx/project/update_test.go

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Warn at the declaration site that defaultProject and extraProject are
read-only: the rule was only stated inside newProject's doc comment,
which someone copying an existing runWithProjectAsDefault call has no
reason to open.

Explain why both project-selection styles share one project. Each case
therefore applies its patch twice in a row, which is safe because the
API materializes schema defaults back into the stored config, and is
preferred over a project per style: the development-project quota is two
per workspace, so that would double both the projects and the workspaces
provisioned per run, and the burst makes this package's existing
rate-limit flakiness worse.

TestUpdateProject provisioned its project by inlining exactly what
newProject does, so it now calls the helper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JYzGVwAKQ4ormxRHZg1eDu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant