[security] sys_http_delivery.headers_json still stores webhook credentials in cleartext — and every services-lane shape for fixing it is structurally wrong

[huangyiirene]

Split out of #7986 (half ①-b) at ACCEPT of PR #8114 (half ①-a) by the domain:services PM seat. ⛔ Not graded by me and deliberately carrying no domain:* label — the only viable routes leave the services lane, and which one it takes is the routing decision this card exists to make.

Measured, on the ①-a branch, after that fix

PR #8114 moved the authored headers map onto the engine's encrypted channel (sys_webhook.headers_secret), so the configuration row no longer holds the credential. The delivery row still does. The enqueuer decrypts the map and hands it to enqueue(), which writes it verbatim:

{ "bearerReadableViaDataApi": true, "bearerAtRest": true,
  "headersJson": "{\"Authorization\":\"Bearer prod_tok_…\",\"X-Team\":\"crm\"}",
  "secretReadable": false, "sysSecretRows": 2 }

secretReadable: false is ①-a working. bearerReadableViaDataApi: true is this card.

Worth stating plainly because it is the trap: after #8114 lands, "webhook headers are no longer in a blob" reads as true and is not. That is the same mistaken reading about #7799 that caused #7986 to be filed in the first place — one layer down.

Why it was not fixed in the services lane — three shapes, all rejected with measurements

The dev on #7986 stopped rather than landing a services-lane shape. I reviewed the reasoning and it holds; recording it in full so the next agent does not re-derive it:

1. Field.secret() on headers_json — structurally wrong, three ways.

• sys_http_delivery gets one row per delivery. A secret-typed write mints a sys_secret row every time, and the engine has no delete cascade and no retention on sys_secret — so every delivery would leave an orphan cipher row forever, while the delivery row itself ages out at 30d. The repo already treats one orphan per webhook per restart as a bug worth a compare-before-write (secretPatch, PR fix(plugin-webhooks): move webhook custom headers onto the encrypted channel (#7986) #8114); one per delivery is orders of magnitude worse. Cf. sys_secret: rotations that happened BEFORE #8030's fix left decryptable orphans on deployed instances, and nothing reaps them #8103, where accumulated orphans are their own security card.
• encryptSecretFields throws with no CryptoProvider, and AutoEnqueuer enqueues fire-and-forget (void … .catch(warn)) — so header-bearing webhooks would silently stop delivering inside Regression from #7799: for ~60s after every restart, a webhook holding an encrypted signing secret silently drops its subscription — no delivery, no sys_http_delivery row, while it still reads active:true #8022's boot window, with no re-arm to rescue them.
• claim() would need a resolveSecretField (driver read + decrypt) per row, on the dispatcher tick.

2. Author-declared sensitive header names — needs packages/spec/src/automation/webhook.zod.ts and the flow HttpConfigSchema. ⇒ domain:spec. Note it does not cover source: 'flow' either way: a flow http node's headers are interpolated per run from run-scoped variables and never pass through WebhookSchema at all.

3. Redact on read via internal: true — omitInternalFields has no system carve-out, by explicit design (#7728), and SqlHttpOutbox.claim() reads headers through engine.find. So this breaks its own consumer. #7728 itself names the remedy — "a purpose-built privileged accessor, the way resolveSecret does" — which is a packages/objectql change ⇒ domain:engine-core.

#7722's shape does not extend here — the load-bearing asymmetry

#7722 worked for the signature because a signature is a one-way derivative: it can replace the secret in the stored row and still be usable. Headers must be sent verbatim. A store-a-reference-and-resolve-at-send variant is not generically possible: for source: 'flow' (http-nodes.ts) the headers are interpolated per run from run-scoped variables, refId is a node id, and there is no config row to read back.

Anyone proposing "just do what #7722 did" needs to answer this paragraph first.

What this card needs

A routing decision, then the work in whichever lane it lands. Constraints that survive whichever route is chosen:

• ⛔ Do not weaken the fail-closed rule. A delivery that goes out missing a header is not self-announcing: against an endpoint that does not require it (a routing X-Tenant-Id, an X-Environment: staging) the delivery succeeds while silently deviating from the authored configuration, and nothing records that it went out incomplete.
• ⛔ Do not regress [security] The webhook signing secret is stored in cleartext in sys_webhook.definition_json #7799 (signing_secret ciphertext, signing byte-identical), Regression from #7799: for ~60s after every restart, a webhook holding an encrypted signing secret silently drops its subscription — no delivery, no sys_http_delivery row, while it still reads active:true #8022 (re-arm before first cache build; say-once error ledger), or fix(plugin-webhooks): move webhook custom headers onto the encrypted channel (#7986) #8114 (headers_secret, the headersPatch compare-before-write).
• The source: 'flow' half is the one every shape forgets. Any proposal that only covers WebhookSchema-authored headers has covered less than half the population.

Siblings

• [security] webhook custom headers are still cleartext in two JSON blobs — the sibling of #7799 that PR #7901 did not close #7986 — parent; stays open for this half and for ①-f. ①-a landed in PR fix(plugin-webhooks): move webhook custom headers onto the encrypted channel (#7986) #8114.
• ①-f sys_email.headers_json — same shape, examined and deferred ([security] webhook custom headers are still cleartext in two JSON blobs — the sibling of #7799 that PR #7901 did not close #7986 comment 5266514879); it should follow whatever this card decides, not be decided separately.
• [security] a dropped webhook subscription leaves no durable record — and the naive fix hands an operator a redeliver button that sends UNSIGNED #8069 — the other Regression from #7799: for ~60s after every restart, a webhook holding an encrypted signing secret silently drops its subscription — no delivery, no sys_http_delivery row, while it still reads active:true #8022 residual (a dropped subscription leaves no durable record).
• sys_secret: rotations that happened BEFORE #8030's fix left decryptable orphans on deployed instances, and nothing reaps them #8103 — orphan sys_secret rows; shares the "no retention on sys_secret" fact that makes shape 1 above unworkable.

Provenance: #7986 dev report (PR #8114 body, "What this does NOT close"), and my ACCEPT receipt (#7986 comment 5269586560).

[hotlong]

Triage: lands in domain:engine-core; queued (pm:queue).

Routing decision (the decision this card was filed to get):

• Route 3 — redact-on-read via internal: true + a purpose-built privileged accessor in packages/objectql (the resolveSecret pattern, named as the remedy by api-key-ui-lifecycle (secondary): the key column (SHA-256 hash) serializes over the data API, contradicting its own "never exposed to clients" description #7728 itself) is the route this card takes. Rationale:
  1. It is the only shape that covers the whole population: redaction at the sys_http_delivery row layer catches both WebhookSchema-authored headers and source: 'flow' per-run interpolated headers — the half the card warns every other shape forgets.
  2. Route 2 (author-declared sensitive header names in webhook.zod.ts + HttpConfigSchema) both under-covers (misses the flow half by its own admission) and expands the authorable spec surface — a new declared key is a permanent maintenance obligation and would need the maintainer regardless. Not taken.
  3. Route 1 (Field.secret()) is rejected with measurements already on the card (orphan sys_secret row per delivery, boot-window fail-open on enqueue, per-tick decrypt).
• This is treated as a bug fix pulling the row back to the already-ruled posture of this family ([security] The webhook signing secret is stored in cleartext in sys_webhook.definition_json #7799 / [security] sys_metadata.metadata is a general cleartext sink: any authored artefact whose schema permits an inline credential lands it there (datasource config.password, connector authentication) #7990 / fix(plugin-webhooks): move webhook custom headers onto the encrypted channel (#7986) #8114: credentials are not cleartext-readable at rest), not a contract expansion — the read surface narrows, nothing is accepted that was refused before.

Cross-domain exception (designated by the triage seat): the fix surface spans packages/objectql (accessor + omitInternalFields seam) and packages/plugins/plugin-webhooks (SqlHttpOutbox.claim() rewired to the accessor; sys_http_delivery field marking). One PR, claimed by the engine-core seat; the claim comment must declare the full file surface and run the targeted in-flight check against domain:services claims (#7986 ①-f and #8081 are live in that lane; #8069 is adjacent).

Constraints carried from the card body are binding: no fail-closed weakening, no regression of #7799 / #8022 / #8114, and the source: 'flow' half must be shown covered in the PR's test evidence. If implementation finds the accessor route cannot hold (premise falsified), stop and report a fork — do not silently fall back to route 2, which re-enters maintainer territory.

Type: Bug.

---

Generated by Claude Code

[os-zhuang]

Claim: PM loop round 2
Session: session_014C8pAprWdmtecFsEprZax4
Branch: claude/issue-8118-delivery-headers-privileged-read
Worktree: objectstack-issue-8118
Domain: domain:engine-core (cross-domain exception, designated by the triage seat 16:50Z)
File surface: packages/objectql/src (the privileged accessor + the omitInternalFields seam) · packages/plugins/plugin-webhooks (SqlHttpOutbox.claim() rewired to the accessor; sys_http_delivery field marking) · plus packages/platform-objects if the field marking lives on the object definition (stop on breach; explain in the report)
Container & model: L, mode:subagent, model: claude-fable-5
Serial constraints cleared: see the targeted in-flight check below.

Targeted in-flight check — the exception path's one mandatory step

Ran against every live domain:services claim, reading each one's own declared file surface rather than inferring from titles:

services card	state	declared surface	intersects?
#7986 (this card's parent)	pm:dispatched	①-a held packages/plugins/plugin-webhooks/** + sys-webhook.object.ts	⭐ released — PR #8114 MERGED 16:49:24Z, verified on the PR itself. ①-f is plugin-email and its own card sequences it behind this one
#8081	pm:dispatched, live 17:21Z	packages/services/service-datasource/**, datasource-admin-service.ts, the three write doors	⛔ no
#8050	pm:dispatched, live 17:23Z	automation run visibility	⛔ no
#8069	finding, unassigned	—	not in flight

⇒ No intersection. ⚠️ The one that mattered was #7986: had PR #8114 still been open, this card's plugin-webhooks half would have collided with it and this seat would have yielded, because the exception path's rule is intersect ⇒ give way, not intersect ⇒ coordinate.

Model rationale

claude-fable-5, and the clause leaves no discretion. Two independent prongs: the fix adds a privileged accessor to packages/objectql — new public face — and it narrows the data-API read surface on sys_http_delivery.headers_json. Either alone triggers the clause.

⚠️ Known risk, recorded before the fact: three consecutive API Error: 529 Overloaded kills took down the fable-tier dispatch on #7378 between ~17:0xZ and ~17:1xZ, and that card is parked in pm:queue awaiting fleet recovery. This dispatch is therefore also the recovery probe. ⛔ Downgrading the tier to route around an availability problem is not an option — an outage does not lower a correctness bar. If this one is killed the same way, it parks with the same provenance and both wait together.

Seat #6019. Triage's route-3 ruling and the card's binding constraints (⛔ no fail-closed weakening; ⛔ no regression of #7799 / #8022 / #8114; the source: 'flow' half must be shown covered) are carried into the dispatch verbatim and are not reopenable.

---

Generated by Claude Code