fix(frontend): serialize concurrent snapshot quota checks

[gouhongshen]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Related to #27718

What this PR does / why we need it:

CREATE SNAPSHOT checked the tenant quota before acquiring the existing
snapshot lineage-publication barrier. Concurrent transactions could therefore
all read the same old mo_snapshots count, then cross the barrier one by one
and each publish a metadata row. Sequential execution did not expose this
time-of-check/time-of-use race because each later check observed the previous
commit.

This PR moves snapshot quota admission after the stable
mo_feature_registry write barrier and before snapshot timestamp generation
and metadata insertion. The quota read, usage count, and mo_snapshots
publication are now in the same serialized background transaction, which is
held through commit or rolled back on any error.

The independent transaction that owns that barrier is explicitly created as
pessimistic/read-committed. Without that ownership-level mode, an
optimistic/SI CN let concurrent losers reach commit and expose Error 20619: w-w conflict instead of waiting and returning the configured quota-limit
error. Only this internal snapshot transaction is forced to pessimistic/RC;
the user session and service defaults are unchanged. The change reuses the
existing background-executor option and publication lock, with no schema,
lock row, retry fallback, or duplicated quota implementation.

TestIssue27718ConcurrentSnapshotQuota proves #27718 by releasing four
pre-opened tenant SQL connections from one start barrier. Two connections use
CN0 and two use CN1, so a process-local lock cannot satisfy the test. The full
matrix runs under both pessimistic/RC and optimistic/SI and asserts:

• quota 1: exactly one creator succeeds, the other three return the quota
  error, and exactly one candidate metadata row persists;
• quota 2: exactly two creators succeed and exactly two rows persist;
• quota 0: all creators are rejected and no candidate row persists;
• quota -1: all four creators succeed;
• dropping the admitted snapshot frees capacity for a replacement; and
• a rejected over-limit create leaves no metadata residue and does not expose
  a transaction retry/conflict error to the client.

TestDoCreateSnapshot additionally verifies that snapshot creation requests
the pessimistic/RC background executor at the transaction ownership boundary.

Before the original change, the issue's four-session reproduction was run
against a fresh local single-CN service: all four clients returned success and
the final snapshot count was 4 with quota 1. Before the ownership-mode fix,
the expanded two-CN regression passed under pessimistic/RC but failed under
optimistic/SI because three losers returned Error 20619 (HY000): w-w conflict instead of the quota-limit error.

Validation after rebasing the same patch onto base
66f2ac48c403091616c1eb3e3def3e68bbc89266, on local head
77ee66e99d5d1c8231b968c9bb1816f458c0993d:

• .agents/skills/mo-dev/scripts/mo-cgo-test -count=1 -timeout=120s -run '^TestDoCreateSnapshot$' -v ./pkg/frontend
• .agents/skills/mo-dev/scripts/mo-cgo-test -count=1 -timeout=180s -run '^TestIssue27718ConcurrentSnapshotQuota$' -v ./pkg/tests/issues
• .agents/skills/mo-dev/scripts/mo-cgo-test -race -count=1 -timeout=240s -run '^TestIssue27718ConcurrentSnapshotQuota$' -v ./pkg/tests/issues
• golangci-lint run --no-config --enable-only copyloopvar ./pkg/tests/issues/...
• git diff --check origin/main...HEAD
• git range-diff 66f2ac48c403091616c1eb3e3def3e68bbc89266...a7b13d353d4b7f9358deb2f03f06b356c90c1190 66f2ac48c403091616c1eb3e3def3e68bbc89266...77ee66e99d5d1c8231b968c9bb1816f458c0993d (all three commits patch-equivalent)

The tested local head and merged PR head
a7b13d353d4b7f9358deb2f03f06b356c90c1190 have the same stable aggregate
patch-id, 7dd68641c27df24a1632fa09c4231a4e2d4614cf. GitHub merged the PR as squash
commit 0725c8c1151df0d70b29e069f83e54793b65b655.

The SCA-reported Go 1.22+ redundant loop-variable copy was removed without
changing test behavior; the exact copyloopvar check and the full two-mode,
two-CN regression pass on the resulting patch.

No BVT fixture is added because mo-tester session directives switch active
connections but execute SQL commands serially; they cannot release multiple
CREATE SNAPSHOT statements concurrently. Existing snapshot feature-limit BVT
fixtures already cover sequential table/database/account quotas, disabled and
unlimited limits, dynamic updates, and capacity reuse. The authenticated
cluster regression supplies the missing concurrent executable proof.

Residual risk: quota catalog reads now run inside the pre-existing global
snapshot publication critical section, slightly extending its hold time. No
new contention domain is introduced. Forcing the independent internal
transaction to pessimistic/RC makes its distributed lock-wait semantics
independent of the CN service's user-transaction default.

[gouhongshen]

Unrelated CI blockers:

• [Bug]: ARM64 SCA self-hosted runner loses communication during analysis #27742: the arm64 SCA job lost its self-hosted runner during static analysis; the check-run API reports runner communication loss rather than a source diagnostic.
• [Bug]: Ubuntu UT job cascades across packages after cluster and checkpoint timeouts #26519: Ubuntu full-UT runs reproduced the known load-sensitive embedded LogService/HAKeeper connection-reset cascade. On current head, database-clone and foreign-key issue tests failed after control-plane timeouts, while TestIssue27718ConcurrentSnapshotQuota ran later and passed; the earlier exact-head focused race test also passed.
• [Bug]: TestDedupLoadWaiterGetsSuccessfulOwnerValue uses a scheduler sleep as waiter admission #27760: the Ubuntu coverage producer exposed a scheduler-dependent pkg/objectio/TestDedupLoadWaiterGetsSuccessfulOwnerValue test race. Its 10 ms sleep does not prove waiter admission before the owner is released; the PR does not touch pkg/objectio or that synchronization path. The downstream Utils coverage merge then failed explicitly because a required producer had failed.
• [Bug]: TestDiskCacheAsyncCallbacksRemainOrderedAndBounded uses a load-sensitive Flush deadline #27764: the Ubuntu full-UT race shard timed out pkg/fileservice/TestDiskCacheAsyncCallbacksRemainOrderedAndBounded at its fixed five-second Flush deadline while draining 64 writes. The same-head coverage producer passed and exact-head focused race stress passed 100/100; the PR does not touch pkg/fileservice or that synchronization path.

[XuPeng-SH]

Blocking: optimistic/SI leaks write-write conflicts instead of completing quota admission.

The new serialization boundary is owned by the independent background transaction created in doCreateSnapshot. That transaction still inherits the service transaction mode. Optimistic mode defaults to SI in the supported CN configuration, but a loser of the registry-row update conflicts when finishTxn commits the background transaction; the outer CREATE SNAPSHOT does not retry that internal transaction. The result is a user-visible Error 20619 w-w conflict, not the expected quota-limit result.

I reproduced this on exact head 7b56cb0 by running TestIssue27718ConcurrentSnapshotQuota after setting the CN runtime to TxnMode_Optimistic plus TxnIsolation_SI. With quota 1, one create succeeded and the other three returned w-w conflict, failing the existing assertion. On the same head, changing the owning background executor to forcePessimisticRC made the full test pass in 12.07s.

Please make serialization/retry explicit at the transaction that actually owns the barrier and the mo_snapshots insert. Forcing this independent transaction to pessimistic RC is one verified option; an equivalent retry at this ownership boundary is also fine. Please also run the regression in both pessimistic/RC and optimistic/SI modes. Because the implementation claims a distributed catalog barrier and the fixture already starts two CNs, split the concurrent connections across CN0/CN1 so a future process-local implementation cannot pass the test.

[gouhongshen]

Verified and addressed on 8ca10e9. The expanded regression reproduced the objection before the fix: pessimistic/RC passed, while optimistic/SI returned Error 20619 (HY000): w-w conflict from the owning background transaction. doCreateSnapshot now creates that independent transaction with the existing forcePessimisticRC option. The regression runs the full quota/failure-atomicity matrix in both modes and alternates four concurrent connections across CN0/CN1. Normal and race-instrumented runs both pass; TestDoCreateSnapshot also asserts the ownership-level option.

[XuPeng-SH]

Deep re-review completed at exact head 8d82e20a59d31ed894737e806af75a3f8da4be73.

My prior blocker is closed. The independent background transaction that owns both the stable mo_feature_registry publication barrier and the mo_snapshots insert is now explicitly pessimistic/read-committed, so a waiter blocks at the distributed row lock and its subsequent RC statements observe the preceding committed snapshot. Optimistic/SI service defaults no longer leak a commit-time w-w conflict to the user.

The quota read, usage count, timestamp/metadata publication, commit, and rollback paths were traced as one transaction. On every error after BEGIN, finishTxn rolls back before the background executor is closed; rejected creators cannot leave a snapshot row. The regression exercises four pre-opened connections split across two CNs, both pessimistic/RC and optimistic/SI service configurations, quotas 0/1/2/unlimited, capacity reuse, expected user errors, persisted row counts, and a bounded timeout. This is the right proof that neither a process-local lock nor a service-default transaction mode can satisfy the fix accidentally.

Performance tradeoff: the existing global SNAPSHOT publication critical section now includes the quota catalog reads and an unindexed filtered count(*). That extends lock hold time and globally serializes this additional work, but snapshot creation was already serialized by this same barrier and remains a low-frequency control-plane operation. No new contention domain, goroutine, unbounded queue, retry loop, or resource owner is introduced; cancellation/lock timeout and rollback remain bounded by the transaction/session context.

The focused exact-head normal/race and two-CN evidence is sufficient. Per reviewer direction I am not waiting for unrelated remaining CI jobs. No remaining blocking correctness, performance, deadlock, or unhappy-path issue found.

[XuPeng-SH]

Incremental deep re-review completed at exact head d63fbe679540301ae14af1fc2f5cfe441325425e on current main 22d56a30b102b58050a31ff04e4f5187541c0f7b.

The rebased PR aggregate patch-id is identical to the fully reviewed head 8d82e20a59d31ed894737e806af75a3f8da4be73 (7dd68641c27df24a1632fa09c4231a4e2d4614cf). There are no files changed by both the PR and the intervening main range, and git diff --check is clean. The prior cross-CN serialization, pessimistic/RC ownership boundary, quota atomicity, rollback, lock-wait, performance, and unhappy-path conclusions therefore remain valid on this exact head. Per reviewer direction, I am not waiting for rerun CI.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-08-27 23:23 UTC · Rule: main · triggered by rule Automatic queue on approval for main
• 🚫 Left the queue — 2026-08-27 23:31 UTC · at 3625544c602d54d5cf17a7d88cde1a655fa5ae25

This pull request spent 7 minutes 50 seconds in the queue, with no time running CI.

Reason

The pull request #27733 has been manually updated

Requeued — the merge queue status continues in this comment ↓.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-08-28 00:19 UTC · Rule: main · triggered by rule Automatic queue on approval for main
• 🟠 Checks running · in-place
• 🚫 Left the queue — 2026-08-28 00:44 UTC · at a7b13d353d4b7f9358deb2f03f06b356c90c1190

This pull request spent 24 minutes 37 seconds in the queue, with no time running CI.

Reason

The pull request can't be updated

This pull request seems to come from a fork, and Mergify needs the author's permission to update its branch.
The author needs to enable "Allow edits from maintainers" on this pull request, or update the branch manually.

Hint

You should update or rebase your pull request manually. If you do, this pull request will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

Tick the box to put this pull request back in the merge queue (same as @mergifyio queue).

• Requeue this pull request