[objectstack-fleet]

Requested by Bill · project thread

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb

维护者速读

• 改了什么：投递调度器和任务队列共用的轮询循环，空闲时改为一次读出"下一条到期时间"并睡到那时；宿主声明自己是唯一写入方时（云端租户内核），开机不再从 0.5 秒起一倍倍退避。
• 为什么改：全新内核第一分钟里三个循环对着空表白轮询约 30 条语句；同进程写入本来就会立即唤醒循环。现在单写者内核每个循环第一分钟只跑一次。
• 风险与回滚：默认不开单写者，老宿主只多一条"到期时间"读（每段空闲一次）；重试、静默时段、摘要窗口改为准点发出。回滚即撤本 PR，无数据迁移。
• 先看：packages/core/src/dispatch-loop.ts（nextDelay 与防空转判断）、packages/services/service-queue/src/db-queue-adapter.ts（延迟发布改用 wakeAt）、packages/services/service-messaging/src/sql-outbox.ts（nextDueAt 两条读）。
• 两分钟验证：pnpm --filter @objectstack/core exec vitest run --project local src/dispatch-loop.test.ts，预期 15 条全绿，其中"fresh loop ticks ONCE in its first minute"对照组为 6 次。
• 需要你做的：无，席位落地。

Fixes #29919

Clause-②: yes (widening) — measured on the built entry closure: @objectstack/core dist/index.d.ts gains DispatchLoopOptions.nextDueAt, DispatchLoopOptions.soleWriter and DispatchLoop.wakeAt; @objectstack/service-messaging gains optional INotificationOutbox.nextDueAt / IHttpOutbox.nextDueAt, the NextDueOptions / HttpNextDueOptions types, soleWriter on both dispatchers' options and MessagingServicePluginOptions.dispatchSoleWriter; @objectstack/service-queue gains DbQueueAdapterOptions.soleWriter. Every addition is optional; nothing is removed or narrowed.
Metadata-protocol: none

What changes

DispatchLoop wakes at the next due time (card item 1). A loop given nextDueAt() asks for the earliest instant work becomes claimable once per idle streak: after the first empty tick following start(), a tick that claimed work, a wake(), or the arrival of the due time it was sleeping towards. It then never sleeps past that instant. The answer is remembered, not re-read on every ceiling tick, so an idle loop's steady-state cost does not move. A deferred row written in this process later in the streak reaches the loop through the new wakeAt(dueAtMs), which moves the timer and costs no read and no tick.

Two guards keep it from spinning. A due time at or before the start of the tick that just found nothing is dropped, because that work was visible and not claimable by this loop (another node holds its partition, for example). A rejected tick or a rejected read counts as idle and sleeps by the backoff.

The single-writer signal (card item 2). No existing host signal says "I am the only writer", and none is needed in packages/spec: the dispatchers' and the queue's options are TypeScript plugin options. soleWriter (default false) on DispatchLoopOptions, on both dispatchers, as MessagingServicePlugin dispatchSoleWriter, and on DbQueueAdapterOptions (reachable as QueueServicePlugin db.soleWriter) makes an empty tick sleep straight to the next due time or the ceiling instead of re-running the ramp from intervalMs. Item 1 alone does not meet "Done when": without the flag the ramp keeps its shape (the CONTROL legs below read 6 ticks in the first minute), and only the due time cuts it short.

Due-time sources, each read only after an idle tick:

Loop	Read	Sources
NotificationDispatcher	SqlNotificationOutbox.nextDueAt, 2 SELECTs	earliest future next_attempt_at of a pending row (retry, quiet hours, digest window close), oldest in_flight claimed_at + claimTtlMs + 1 (the reaper takes a claim strictly older than the TTL)
HttpDispatcher	SqlHttpOutbox.nextDueAt, 2 SELECTs	earliest future next_retry_at, oldest in_flight claim + TTL + 1
DbQueueAdapter	1 SELECT over the subscribed queues	earliest future scheduled_for

The queue has no lease expiry to read: every write that returns a row to pending clears locked_until, so no lease gates a pending row. The in-flight read uses the reaper's status, claimed_at index; the deferred reads sort the deferred pending rows, and an idle store has none.

A deferred publish() used to skip the wake on purpose, so as not to reset the backoff. It now calls wakeAt, so the row goes out at its scheduled_for even under a minutes-long ceiling.

Docs (head a1a9fbd3, a docs-only commit the seat asked for at review). content/docs/automation/webhooks.mdx: an idle tick costs two reads whatever partitionCount is (the old 1 + partitionCount figure was stale since #28383); a new paragraph on waking at the next due time (a retry is attempted when due, a crashed node's claim is recovered when its TTL passes; the backoff is left only for rows another process wrote); dispatchSoleWriter and the queue's db.soleWriter: what they do, who turns them on (one kernel per tenant database), who leaves them off (a second node or worker on the same database), default off; the crash row of the failure table matches. kernel/cluster.mdx, plugins/packages.mdx and api/environment-routing.mdx were read and nothing in them is made false by this PR.

Statement counts, before and after

Taken on this repo's harnesses, at the client (knex query events on the real ObjectQL + SqlDriver better-sqlite3 harness for the outboxes; engine calls on the counting engine for the queue). The card's figures were taken at the Turso client in cloud, which counts more per read, so the absolute numbers differ. The tick counts are the same.

Loop, fresh kernel, empty store, first minute, 5-minute ceiling	before (main)	after, soleWriter	after, default
NotificationDispatcher	6 ticks × 2 = 12	1 tick × 2 + due read 2 = 4	6 ticks × 2 + 2 = 14
HttpDispatcher	6 × 2 = 12	2 + 2 = 4	14
DbQueueAdapter (one queue, 1 s base)	5 ticks × 1 = 5	1 + 1 = 2	6
Total	29	10	34

Steady state at the ceiling is unchanged: an idle hour at 30 s reads the due time once (steady state costs no extra read pin, 121 ticks, 1 read). Without the flag, an idle stretch costs one extra read at its start. That is the price of "a deferred row goes out within one tick of coming due" on a multi-writer host.

Tests

New pins (fake timers, exact instants)

• packages/core/src/dispatch-loop.test.ts (new, 15): a fresh sole-writer loop ticks once in its first minute, with a CONTROL showing the ramp the harness can see (0, 1, 3, 7, 15, 31 s). soleWriter alone goes to the ceiling. Steady state reads the due time once an hour. A row due at T+20 s is claimed at T+20 s under a 5-minute ceiling. The ramp is cut short only by a due time. Consecutive deferred rows are each on time. wakeAt costs no read and keeps a sooner timer. wake() ticks at once. A rejected tick is idle (one tick per ceiling) and keeps a learned due time without spinning. A due time at or before the empty tick is not chased. A rejected read falls back to the backoff. stop() is final for wakeAt.
• db-queue-idle-backoff.test.ts (+5, 2 moved): a sole-writer queue reads 2 in the first minute (CONTROL 6), a row scheduled before boot runs at exactly +20 s, a deferred in-process publish runs at exactly +20 s with no extra read, and a failing retry is re-attempted at its 15 s backoff. The two service-queue: DbQueueAdapter.claimBatch 按 priority, scheduled_for 排序,而 sys_job_queue 的声明索引不含 priority —— 排序首键落空,恒定 1s 轮询每跳吃一次临时 B-tree 排序 #17612 exact counts move by the one due read of the streak (3601 → 3602, 124 → 125), with a comment saying so.
• dispatcher-idle-backoff.test.ts (+3) and http-dispatcher-idle-backoff.test.ts (+2): one tick in a sole-writer dispatcher's first minute. A deferred delivery written before boot is sent at +20 s. A crashed node's claim is re-sent at claim TTL + 1 ms. Every HTTP retry is attempted at exactly its due time under a 5-minute ceiling.
• dispatcher-idle-cost.integration.test.ts (+3) and http-dispatcher-idle-cost.integration.test.ts (+2), on a real ObjectQL + SqlDriver: nextDueAt on an empty outbox is null in exactly 2 SELECT statements. It answers the earliest deferred row, digest rows included, and ignores a row already due. It answers the oldest claim's expiry when that is sooner.

Ablations: each mutation made through scripts/ablation-replace.mjs, landed on disk, red, then restored with blob == HEAD

All three suites resolve @objectstack/core to its source through their vitest alias, so no dist/ rebuild is involved.

Mutation	Suite	Red
A1: put the ramp back (delete if (this.soleWriter) return this.maxIdleIntervalMs;)	core	3 (ticks ONCE; soleWriter alone; steady state)
A1	queue	1 (sole writer first minute)
A1	messaging	3 (both dispatchers' first minute, and the deferred / claim legs)
A2: ignore the due time in nextDelay	core 5 · queue 2 · messaging 3	every due-time leg
A3: drop the guard that discards a due time at or before the tick's start	core 2	the not-chased leg and the rejected-after-due leg spin
A4: drop the queue's wakeAt on a deferred publish	queue 1	deferred in-process publish

Acceptance notes

• Item 2 lands as soleWriter / dispatchSoleWriter / db.soleWriter. Cloud turns it on for hosted tenant kernels together with the queue's ceiling under cloud#3413; nothing in this repo turns it on. With cloud's 5-minute dispatchMaxIdleIntervalMs, each dispatcher ticks once in the first minute. The queue ticks once only after cloud also raises db.maxIdleIntervalMs (the card says that is cloud's change); at the framework's 30 s default a sole-writer queue ticks at 0 s and 30 s.
• Default hosts: the ramp is unchanged. content/docs/automation/webhooks.mdx is updated in this PR (see Docs above); its stale 1 + partitionCount figure is corrected here.
• No packages/spec key was added. Seat 4's objectql: the lifecycle sweep runs 60 s after every kernel boot, even on a database created seconds earlier, so a fresh tenant kernel pays 130 statements in its first minute #29918 adds no host signal, so nothing is shared with it.

Verification

Local gates and suites (head sha in the os-dev-report)

Head def0361dc (after merging origin/main at dfafae813; no incoming commit touches packages/core, service-messaging or service-queue).

• pnpm --filter @objectstack/core --filter @objectstack/service-messaging --filter @objectstack/service-queue run test: core 134 files / 2989 tests, service-messaging 66 / 812, service-queue 5 / 82, all passed (lock VERDICT command-exit 0).
• run typecheck on the same three: exit 0 (core includes check:test-typecheck).
• pnpm lint: exit 0.
• node scripts/pm/dispatch-gates.mjs --commands: 95 derived families. The 65 code families ran at def0361d (65 × exit 0; the docs commit touches no code), and the 30 families the docs page adds ran at a1a9fbd3 (30 × exit 0). --ran reconciliation: 95 derived, 95 run, 0 NOT-MEASURED. Readings: project file server-seat5/gates-29919-ran-r2.txt.
• Before the merge the same list ran on 2d31b756 with three exit 3 (prerequisites: check-plugin-teardown-shape --self-test on a shallow clone, check:dual-build-cjs-loads and check:i18n without the built CLI closure). On this head all three ran after git fetch of the fixture commit and the gate's own named build: exit 0.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

This PR changes 3 package(s): @objectstack/core, @objectstack/service-messaging, @objectstack/service-queue, touching 43 documentable anchor(s). ⚠️ 1 changed file(s) yielded no anchor (packages/services/service-messaging/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

• content/docs/api/environment-routing.mdx (via HttpDispatcher (symbol, a top-level class))
• content/docs/automation/webhooks.mdx (via DbQueueAdapter (symbol, a top-level class), HttpDispatcher (symbol, a top-level class), HttpDispatcherOptions (symbol, a top-level interface), MessagingServicePlugin (symbol, a top-level class), claimTtlMs (symbol, a field of interface HttpNextDueOptions; a field of interface NextDueOptions), dispatchSoleWriter (symbol, a field of interface MessagingServicePluginOptions), soleWriter (symbol, a field of class DispatchLoop; a field of interface DbQueueAdapterOptions; a field of interface DispatchLoopOptions; a field of interface HttpDispatcherOptions; a field of interface NotificationDispatcherOptions), claimed_at (literal, a string literal in nextDueAt), in_flight (literal, a string literal in nextDueAt), maxIdleIntervalMs (literal, a string literal in HttpDispatcher; a string literal in NotificationDispatcher), next_retry_at (literal, a string literal in nextDueAt), onAttempt (literal, a string literal in HttpDispatcher; a string literal in NotificationDispatcher), soleWriter (literal, a string literal in HttpDispatcher; a string literal in NotificationDispatcher))
• content/docs/kernel/cluster.mdx (via HttpDispatcher (symbol, a top-level class))
• content/docs/plugins/packages.mdx (via HttpDispatcher (symbol, a top-level class), MessagingServicePlugin (symbol, a top-level class))

⛔ 4 release-owned page(s) also name something this change touched. These are read-only:

• content/docs/releases/v14.mdx (via MessagingServicePlugin (symbol, a top-level class))
• content/docs/releases/v17/17-2.mdx (via IHttpOutbox (symbol, a top-level interface), SqlHttpOutbox (symbol, a top-level class))
• content/docs/releases/v17/17-3.mdx (via INotificationOutbox (symbol, a top-level interface))
• content/docs/releases/v17/17-5.mdx (via DbQueueAdapter (symbol, a top-level class), DbQueueAdapterOptions (symbol, a top-level interface), HttpDispatcher (symbol, a top-level class), MessagingServicePlugin (symbol, a top-level class), NotificationDispatcher (symbol, a top-level class), maxIdleIntervalMs (literal, a string literal in HttpDispatcher; a string literal in NotificationDispatcher))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see

• 1 changed file(s) yielded no anchor (packages/services/service-messaging/src/index.ts) — pages documenting those are invisible to this run
• 7 name(s) were too generic to anchor anything (single lowercase words)
• the SDK route bridge reached 61 of 219 client-bound route-ledger rows — the other 158 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 158: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 103 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
• a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 39 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json ff12aed37b35ecf2cc1d8fd9a3f609fa79dfa6e7 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 65b56e032c75904131b69ce3edc796481411694f — the merge of head a1a9fbd308c37468d652a5f60daa7cde68e4c095 into base ff12aed37b35ecf2cc1d8fd9a3f609fa79dfa6e7, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 65b56e032c75904131b69ce3edc796481411694f && git checkout 65b56e032c75904131b69ce3edc796481411694f
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin ff12aed37b35ecf2cc1d8fd9a3f609fa79dfa6e7 a1a9fbd308c37468d652a5f60daa7cde68e4c095 && git checkout -B drift-repro ff12aed37b35ecf2cc1d8fd9a3f609fa79dfa6e7 && git merge --no-ff a1a9fbd308c37468d652a5f60daa7cde68e4c095

node scripts/docs-audit/affected-docs.mjs --json ff12aed37b35ecf2cc1d8fd9a3f609fa79dfa6e7

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs ff12aed37b35ecf2cc1d8fd9a3f609fa79dfa6e7 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

[objectstack-fleet]

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb

席位核对 · ACCEPT · head a1a9fbd3 · 2026-11-05T03:19Z

Head-sha: a1a9fbd

Item	Reading
Shape	Draft, base main, relay pr_create. Body: attribution, signature, ## 维护者速读, Fixes #29919; Clause-②: / Metadata-protocol: at line start.
Acceptance	Item 1: DispatchLoop.nextDueAt reads once per idle streak, with guards against chasing a due time at or before the empty tick and against a rejected read (pins in dispatch-loop.test.ts, 15). Item 2: soleWriter default false on the loop, both dispatchers, MessagingServicePlugin.dispatchSoleWriter and DbQueueAdapterOptions; first-minute count 29 → 10 with CONTROL 6-tick ramps. "No signal in packages/spec" is stated with its reason (TS plugin options).
Gates	Check-runs on a1a9fbd3: 59 success, 10 skipped, 0 failure, 0 pending. Dev's local reconciliation 95/95/0.
Declarations	Clause-②: yes (widening), measured on the built dist/index.d.ts of core, service-messaging and service-queue; every addition optional, nothing removed. Widening → this seat's review is the contract review (TIER_DEFAULT). Metadata-protocol: none; changeset patch.
Docs	automation/webhooks.mdx fixed at the seat's ask (idle cost two reads, not 1 + partitionCount; next-due wake; dispatchSoleWriter / db.soleWriter: who turns them on, default off; crash row). kernel/cluster.mdx, plugins/packages.mdx, api/environment-routing.mdx read: nothing made false. packages.mdx:287's lease promise belongs to #30228.
Out of scope	filed #30228 (queue running rows never reclaimed: the queue has no lease expiry to read, as the body says).
Landing	Not governed (0 of 21 paths), 1081 changed lines (< 3000). Seat lands: pr_ready + automerge_enable. After merge: seat 4 (mailbox #30189) is told of the soleWriter keys; #30228 merges main and builds acceptance 3 on this code.

[github-actions]

⛔ merge queue 构建失败 — 先分诊，再决定要不要重排

队列构建 41961141659 红了。队列跑的是全量套件（PR 侧 CI 只跑 affected 子集），
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

分类：failure —— 按下面的日志分诊。

失败的 job（日志抽取，best effort）：

• Test UI (16/16) — 失败步骤: Run the ui tests (shard 16/16)

  × a read-only package: overlay types can be customized, never 可写; the create-only and closed types are read-only 126ms
  × a read-only package: the view list header reads 可定制, the page list header 只读 15ms
  × typeStateWord: the mapping in every context 5ms
  FAIL   dom  packages/app-shell/src/views/metadata-admin/DirectoryPage.stateVocabulary-29705.test.tsx > All metadata tiles: one state word, in the package context (objectstack#29705) > a read-only pack
    ↳ 失败原因: AssertionError: expected '可定制' to be '只读' // Object.is equality
  FAIL   dom  packages/app-shell/src/views/metadata-admin/DirectoryPage.stateVocabulary-29705.test.tsx > the list header draws the tile word (objectstack#29705) > a read-only package: the view list head
    ↳ 失败原因: (这条 FAIL 之后 12 行内没有可识别的原因行 —— 点进 job 看)
  FAIL   dom  packages/app-shell/src/views/metadata-admin/DirectoryPage.stateVocabulary-29705.test.tsx > the state vocabulary census (objectstack#29705) > typeStateWord: the mapping in every context
    ↳ 失败原因: AssertionError: expected [ Array(4) ] to deeply equal [ 'customizable', 'readOnly', …(2) ]
  

↳ 失败原因 是判读的关键：超时（Test timed out in … / Hook timed out in …）多半是负载/时序，不是本 PR 的回归；
断言（AssertionError: …）才指向真实的行为改变。两者的 FAIL 行长得一模一样，只有这一行能区分。

⚠️ 断言这一侧有一类例外，判据是断言在测什么，不是它是不是 AssertionError。 断言的对象是产品行为（一个值、一个形状、一次拒收）⇒ 照上面读：真实的行为改变，去查，⛔ 不要重排掉；
断言的对象是这次实验自身的有效性前提（跑完的耗时、负载下的先后、任何只在时间预算内才成立的条件）⇒ 它跟超时是同一类，同样对负载敏感，重排一次是合法的判别手段。
识别是机械的：断言的消息或它比较的值本身点名了一段时长、一个时间戳、一个耗时计数。实测过的一对 —— AssertionError: SecurityPlugin.init() ran: expected false to be true 测的是产品行为（真回归）；
AssertionError: this run took over a second, so second-precision stamps could have differed too: expected 1006 to be less than 1000 测的是实验前提：它守护的那条不变式当时是绿的，同一个 head 原样重排一次即成功。
穿着 AssertionError 外衣的时间测量，仍然是时间测量。（⛔ 这只改「怎么读一次红」，不改「哪些测试可以重排」——后者由别处管。）

跨 PR 相同签名（24h，按失败测试文件聚合）：

• packages/app-shell/src/views/metadata-admin/DirectoryPage.stateVocabulary-29705.test.tsx — 24h 窗口内只有本 PR 撞到过，暂不汇总（再有一个不同 PR 撞到就会自动开汇总 issue）。
• ⚠️ 24h 评论账本没读完（超过 5 页仍未读到窗口尽头），所以上面的「不同 PR 数」是下界，不是全量。

历史信号：

• 本 PR 过去 24h 无队列失败记录（首次）。
• 过去 24h 队列共有 1 个失败构建（不含本次）。

分诊清单：

1. 失败测试在本 PR 改动的包里 → 真回归，修 PR。
2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」；已有汇总 issue ⇒ flaky/环境问题实锤，去那张 issue 上谈，修好前重排只会再烧一轮全队列。
3. 两者都不是 → 可能与同组 PR 语义冲突；等前面的 PR 落地或失败出队后再重排一次即可，不要连续重排。

---

Generated by Claude Code · merge-queue-triage workflow (#4859)