Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 25 additions & 2 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

34 changes: 34 additions & 0 deletions services/cloud-agent-next/DEBUG.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,40 @@ Wrangler will show the `PUT` requests. The uploaded archive contains:

The internal `getWrapperLogs` path also discovers these sandbox-side files directly by scanning `/tmp/kilocode-wrapper-*.log` and the Kilo CLI log directory.

## Control-plane Diagnostics

Control-plane (`workspace_*`) sessions use verbose structured wrapper diagnostics, not the legacy raw wrapper/Kilo tarball. Records include heartbeat attempts, feed freshness, control socket and request outcomes, event-send metadata, task phases, and retirement causes. They exclude prompts, assistant/tool content, raw errors, credentials, and URLs.

The wrapper uploads JSON batches to the existing R2 bucket every five seconds and when a batch fills. Shutdown attempts a final flush within the existing shutdown deadline. R2 keys are:

```text
logs/control/<sandboxId>/<allocationId>/<wrapperInstanceId>/<batchId>.json
```

`sandboxId` is the logical SandboxControl ID, not the `workspace_*` session ID or physical provider allocation name. Use Worker logs to correlate these IDs. Each allocation/wrapper has separate immutable batches; sort them by the batch `sequence` and record `timestamp`, not the random batch ID. Check `droppedRecords` and `droppedTerminalRecords` for buffer overflow or rejected diagnostic records.

List and download these JSON batches directly from R2 using local tooling and the key prefix above. Uploaded batches remain available after the container disappears, subject to the bucket's retention policy. The legacy `getWrapperLogs` live-file reader and tarball retrieval do not read these JSON archives.

Worker/DO diagnostics remain in Cloudflare logs/Axiom, not these wrapper archives. Successful `message.part.delta` forwarding is summarized in heartbeat counters and peak queue/RPC/total forwarding times instead of per-frame Worker logs. These counters and peaks reset when the DO is reconstructed. Failure and lifecycle records remain verbose, and wrapper archive logging is unchanged. Upload result markers on wrapper stderr distinguish HTTP rejection, network failure, timeout, and acceptance. An upload-only grant expires four hours after allocation launch and is not renewed; runtime credential revocation does not revoke it. Grant expiry does not delete archives. R2 retention remains governed by external bucket policy, not the session/report cleanup jobs.

Uploads are best effort: an abrupt kill or network failure can lose unuploaded records. The buffer holds 512 records and each batch holds up to 128 records or 256 KiB. A recorded WebSocket send is a local handoff, not proof that a session DO applied the event; correlate it with the Worker forwarding and durable message-transition logs.

### Shared-worktree Lifecycle

`worktreeId` identifies the shared checkout and chat group; `sessionId` is the Cloud Agent chat ID and `kiloSessionId` is the Kilo session ID. Worker `worktree_chat_*` events cover admission, progress, reconciliation, settlement, and the result, correlating the source and resulting chats with the existing worktree. Their durations are phase-local; a reconciliation-pending result describes required recovery, while the separate reconciliation and settlement records report persistence outcomes. Wrapper attachment does not receive an explicit worktree ID, so join its chat IDs with Worker records rather than treating a credential `scopeId` or directory as the worktree identity.

Worker `worktree_ownership` records distinguish `exclusive`, `shared`, and `unresolved` decisions and identify the evidence used. Unresolved ownership is not proof of sharing or permission to destroy a sandbox. `worktree_runtime_cleanup` records the cleanup strategy, failure stage, and confirmed journal flags. `worktree_cleanup_location` confirms resources cleaned at one runtime location; it is not overall deletion completion. Only `worktree_deletion` with `result=completed` or `result=replayed` confirms the complete deletion request.

Wrapper `control.request` attachment summaries separate `workspaceAction` (reuse or bootstrap) from `sessionResolution` (existing, restored, or created chat). Worktree deletion records include the first fence/drain, preparation/deletion outcome, stage, and session count. These records contain IDs and fixed outcomes, not repository paths, credentials, or session content.

### Accepted-message Reconciliation

For `runtime_unhealthy`, correlate Worker `accepted_reconciliation` and `session_sync` records by `messageId`, expected wrapper identity, and lifecycle epoch. Reconciliation records distinguish healthy, superseded, and unhealthy decisions. The unhealthy decision is logged before failure/cleanup starts; `session_message_committed` remains the durable message-state confirmation. `session_interrupt_failed` identifies the separate explicit-abort path that can produce the same public failure reason.

`session_sync` identifies its trigger (`accepted_alarm` or `pending_interactions`), failed stage, timeout, observed physical/connection state, and expected versus observed wrapper identity. It records safe response codes and validation counts, never raw errors or response payloads. Compare `receivedQuestionCount`/`receivedPermissionCount` with `questionCount`/`permissionCount` to see root scoping, and check `interactionSnapshotApplied` for revision-fenced snapshots. A successful sync means the snapshot was fetched and processed, not necessarily that accepted work is still active; the reconciliation decision applies the activity rule afterward.

Wrapper `control.request` records for `session.sync` separate status, question, and permission reads from the overall `sync_result`. `nativeStatus`, `syncStatus`, and `ownedTask` show the task-owned busy override. Pending-query flags are frozen when the shared request signal aborts, so a query that ignores cancellation is still visible. Missing counts mean no successful result was available; they are not zero counts. The three reads retain their original shared deadline and parallel execution.

## Interpreting Common States

- Worker queueing succeeds, but no wrapper logs appear:
Expand Down
2 changes: 1 addition & 1 deletion services/cloud-agent-next/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
},
"devDependencies": {
"@cloudflare/containers": "0.3.7",
"@cloudflare/vitest-pool-workers": "catalog:",
"@cloudflare/vitest-pool-workers": "0.18.6",
"@kilocode/sdk": "7.4.20",
"@types/jsonwebtoken": "catalog:",
"@types/node": "catalog:",
Expand Down
Loading
Loading