Skip to content

Three relayfile mount calls are still unbounded after #368, and confirmWrite's deadline cannot interrupt a call #370

Description

@khaliqgant

Follow-up to #368, which bounds the change-log tail reads (listLastNChanges /
getEvents) — the await the deployed readiness sweep was held on.

While diagnosing that I audited every method on RelayfileCloudMountClient that
touches this.#client, mechanically rather than by eye. Three calls are still unbounded
after #368. Filing them together so the next wedge is not another archaeology round.

method file:line SDK call state
readFile :675 readFile bounded (#354)
writeFile :684 readFile, writeFile bounded (#354)
deleteFile :735 deleteFile, readFile bounded (#354)
listTree :783 listTree bounded (#354), one deadline for the whole cursor walk
ensureSubRoot :1117 listTree bounded (#354)
getEvents / getEventHighWatermark :1036 / :1057 getEvents, listLastNChanges bounded by #368
confirmWrite :1069 getOp UNBOUNDED
#assertProviderDeleteAllowed :752 getOp UNBOUNDED
getSyncStatus :1064 getSyncStatus UNBOUNDED

1. confirmWrite — the most likely next wedge

const deadline = Date.now() + (opts.timeoutMs ?? 90_000)
for (;;) {
  const operation = await this.#client.getOp(this.workspaceId, opId)
  ...
}

The deadline is evaluated between calls and can therefore never interrupt one. It
bounds the number of polls, not the poll. A getOp that stops answering makes this loop
wait forever with a 90-second deadline sitting right above it — which reads, to anyone
skimming, as bounded. This is the exact shape of the bug #368 fixes, one call path over.

It is reachable from work the sweep awaits: writeback/linear.ts:174 and
writeback/slack.ts:72 both call mount.confirmWrite(path, { timeoutMs: 90_000 }), and
that 90 000 is the argument being silently defeated.

Fix: give each getOp its own budget via #remainingMs()listTree already models
exactly this for a multi-call operation (:783-813), refusing at zero rather than falling
through to "no budget".

2. #assertProviderDeleteAllowed — same getOp, delete path

Single call, no loop, no deadline. Route it through #bounded().

3. getSyncStatus — unbounded, but no live caller

grep -rn 'getSyncStatus(' src --include='*.ts' finds no caller outside the client itself,
so nothing can hang on it today. Worth bounding anyway before something does, and it is the
cheapest of the three: the SDK's GetSyncStatusOptions does carry a signal, so this
one gets real transport cancellation rather than an abandoned wait.

Note on what "bounded" buys, per call

Not uniform, and worth stating so nobody reads the table as a promise of cancellation:

  • readFile, listTree, ensureSubRoot, getEvents (cursor), getSyncStatus — the SDK
    option type carries a signal, so the abort cancels the request.
  • listLastNChanges — its ProactiveRequestContext is { workspaceId, token? } with no
    signal field, so fix(mount): bound the relayfile change feed so a hung tail read cannot wedge the readiness sweep #368 can only abandon the wait. The socket and the SDK's retry
    loop stay live. Getting real cancellation there needs a signal on
    ProactiveRequestContext upstream in @relayfile/sdk.

Either way the rejection unwinds the sweep and releases the discovery lease, which is what
stops the next cycle coalescing onto a wedged runOnce().

Separately: an unbounded release retry against a dead host node

Not a mount defect, but found in the same /evidence read and currently generating
essentially all of the deployed daemon's log volume. Three agents retried once per second,
forever:

RelayError 503 rawCode=agent_host_unavailable retryable=true
"Agent \"ar-1540-...\" has no live host node; cannot dispatch release"

27 identical cycles in 15.7 minutes, zero progress, on a node whose lastHeartbeatAt in
/evidence's nodeIdentity.ownerPrecheck is 2.9 days old (online: false,
nodeStatus: "offline"). isAgentAlreadyGoneOnRelease already terminalises a 404 (#365);
a 503 agent_host_unavailable against a node offline for days needs the same treatment, or
a cap. It is timer-driven so it does not block the sweep — but it is an unbounded retry
against a permanently unavailable dependency.

Diagnosis and measurements:
factory-cloud#55.
Filed by wedge-layer2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions