diff --git a/contracts/agents-api/environments.md b/contracts/agents-api/environments.md index d06e33d50..784445e50 100644 --- a/contracts/agents-api/environments.md +++ b/contracts/agents-api/environments.md @@ -86,7 +86,7 @@ A `pending` Environment is accepted: the Session's first input waits for prepara **Files.** [Environment files](./environment-files.md#where-files-work) owns the files routes while no Session is attached. -**Lifecycle without a Session.** An unattached Environment provisions and initializes without a Harness, and its lifetime depends on the Provider. With checkpoint support it suspends after the deployment's idle timeout and expires when snapshot retention ends; [Suspension policy](./sandbox-deployment.md#suspension-policy) owns both values. When a node's full retained slots block waiting placement, [capacity pressure](../../docs/sandbox-provider.md#cold-replacement-after-checkpoint-retention) can expire an idle suspended one earlier. Without checkpoint support it cannot be suspended, so it expires after the idle timeout without activity, counted from its last activity: the end of its initialization, a detach or later activity. A failed Environment releases its compute at once. On any Provider it also expires after confirmed compute loss, including compute released while a Session held it once that Session is deleted, because Core never replaces it on new compute: nothing identifies the work to replay. A [sandbox reset](./sandbox-deployment.md#reset) expires it as well. An expired Environment cannot be attached. An ended (`expired` or `failed`) unattached Environment releases its placement and workspace. +**Lifecycle without a Session.** An unattached Environment provisions and initializes without a Harness, and its lifetime depends on the Provider. With checkpoint support it suspends after the deployment's idle timeout and expires when snapshot retention ends; [Suspension policy](./sandbox-deployment.md#suspension-policy) owns both values. When a node's full retained slots block waiting placement, [capacity pressure](../../docs/sandbox-provider.md#cold-replacement-after-checkpoint-retention) can expire an idle suspended one earlier. Without checkpoint support it cannot be suspended, so it expires after the idle timeout without activity, counted from its last activity: the end of its initialization, a detach or later activity. An Environment whose compute is not running with a completed bootstrap within the [start deadline](../../docs/sandbox-provider.md#timeouts-and-limits) fails. A failed Environment releases its compute at once. On any Provider it also expires after confirmed compute loss, including compute released while a Session held it once that Session is deleted, because Core never replaces it on new compute: nothing identifies the work to replay. A [sandbox reset](./sandbox-deployment.md#reset) expires it as well. An expired Environment cannot be attached. An ended (`expired` or `failed`) unattached Environment releases its placement and workspace. ## Placements diff --git a/contracts/agents-api/zh/environments.md b/contracts/agents-api/zh/environments.md index 29ace25e7..19588d74e 100644 --- a/contracts/agents-api/zh/environments.md +++ b/contracts/agents-api/zh/environments.md @@ -1,7 +1,7 @@ --- title: "Environment 与模板" source: contracts/agents-api/environments.md -source_hash: effa1e3ba384fbb3272d26c5c1d56c9d7e2116f7f2bfa2ecce6cfe1a2a266552 +source_hash: f3f1be076e2b7d65a66fc33614d410237a44b723519d640a25ef90b69f1b8a35 --- Environment 是 Session 的执行资源，包括 Harness 运行所在的机器、工作区以及能力。Session 通过其 `environment` 配置创建 Environment，或挂载由 `POST /agents/environments` 预先创建的 Environment（[独立 Environment](#standalone-environments)）。Environment 模板是 Session 创建时解析的可复用准备配置。本契约涵盖这两类资源、两种放置方式、输入接纳、能力准备、Skills、Plugins 和 MCP 连接来源。 @@ -88,7 +88,7 @@ Core 在 Session 创建事务中创建 Environment 记录；Session upsert 会 **文件。** 没有挂载 Session 时的文件路由由 [Environment files](environment-files.md#where-files-work) 规定。 -**无 Session 时的生命周期。** 未挂载的 Environment 无需 Harness 即可供给和初始化，其存续时间取决于沙箱提供商。支持检查点时，它在部署的空闲超时后挂起，并在快照保留期结束时过期，两者的取值见[挂起策略](sandbox-deployment.md#suspension-policy)。节点的 retained slot 已满并阻塞等待中的放置时，[容量压力](../../../docs/zh/sandbox-provider.md#cold-replacement-after-checkpoint-retention)可以让已挂起的空闲 Environment 提前过期。不支持检查点时它无法挂起，因此在空闲超时内没有活动后过期，空闲时间从最近一次活动起算：初始化完成、卸载或之后的活动。失败的 Environment 会立即释放其计算。无论哪种提供商，确认计算丢失后它也会过期（包括 Session 持有期间被释放的计算，在该 Session 被删除后），因为 Core 从不在新计算上替换它：没有任何信息能标识要重放的工作。[沙箱重置](sandbox-deployment.md#reset)同样会让它过期。已过期的 Environment 不能挂载。已结束（`expired` 或 `failed`）的未挂载 Environment 会释放其放置和工作区。 +**无 Session 时的生命周期。** 未挂载的 Environment 无需 Harness 即可供给和初始化，其存续时间取决于沙箱提供商。支持检查点时，它在部署的空闲超时后挂起，并在快照保留期结束时过期，两者的取值见[挂起策略](sandbox-deployment.md#suspension-policy)。节点的 retained slot 已满并阻塞等待中的放置时，[容量压力](../../../docs/zh/sandbox-provider.md#cold-replacement-after-checkpoint-retention)可以让已挂起的空闲 Environment 提前过期。不支持检查点时它无法挂起，因此在空闲超时内没有活动后过期，空闲时间从最近一次活动起算：初始化完成、卸载或之后的活动。计算在[启动期限](../../../docs/zh/sandbox-provider.md#timeouts-and-limits)内没有运行并完成 bootstrap 的 Environment 会失败。失败的 Environment 会立即释放其计算。无论哪种提供商，确认计算丢失后它也会过期（包括 Session 持有期间被释放的计算，在该 Session 被删除后），因为 Core 从不在新计算上替换它：没有任何信息能标识要重放的工作。[沙箱重置](sandbox-deployment.md#reset)同样会让它过期。已过期的 Environment 不能挂载。已结束（`expired` 或 `failed`）的未挂载 Environment 会释放其放置和工作区。 ## 放置方式 {#placements} diff --git a/docs/sandbox-provider.md b/docs/sandbox-provider.md index 6b4227d19..739fd9743 100644 --- a/docs/sandbox-provider.md +++ b/docs/sandbox-provider.md @@ -171,7 +171,7 @@ An adapter implements none of this, but it relies on it. The code-level mechanic - Core commits the allocation, its daemon credential digest and the Session binding before it calls `Create`. Only a fresh allocation receipt permits `Create`; retries and a Core restart observe the same `Reference` and never replay `Create` or rotate the credential. - Each node has one serial lifecycle worker, so calls for one node never run concurrently; a direct Provider's allocations share one worker. A stuck Provider on one node never stalls another. - Core never infers bootstrap completion from an engine or provider name; the adapter qualifies it through `Info.BootstrapComplete`. -- Core calls `Kill` only after it has revoked the device's authority, settled pending input and requested cancellation. Temporary provider outages, unknown `Create` results and stopped compute never prove a permanent failure. +- Core calls `Kill` only after it has revoked the device's authority, settled pending input and requested cancellation. Temporary provider outages, unknown `Create` results and stopped compute never prove a permanent failure. The start deadline in [Timeouts and limits](#timeouts-and-limits) bounds the wait for compute that never reached running: it ends the Environment without claiming that `Create` failed. - Core releases an allocation only after confirmed cleanup of its compute and volumes and proof that the original `Create` settled. Until then it keeps cleanup ownership, also after an absence observation, and bounded scans continue to catch late resources without another `Create`. This applies after public Session deletion too. - Core never releases an allocation on an offline node, a timeout or an empty read, and takes no lifecycle action on a node that is not [online](../contracts/agents-api/node-generation-protocol.md#connection). Retained ownership needs exact provider evidence: a socket path, a missing instance or an empty listing never proves cleanup or authorizes a replacement. - Placement is automatic and callers cannot choose a node. An active allocation keeps its original node, even while it is offline. After confirmed release, eligible retained Sessions can reserve compatible capacity again, including on another node. Source teardown must be confirmed before active capacity is released, and unknown operations keep their reservations. The [deployment contract](../contracts/agents-api/sandbox-deployment.md#generation-ownership-and-rollout) owns waiting, ordering and generation selection. @@ -308,6 +308,7 @@ Every other observation keeps ownership and its diagnostic: a `paused`, `running | Snapshot retention | Checkpoint-capable allocations, set by the [suspension policy](../contracts/agents-api/sandbox-deployment.md#suspension-policy) | When retention ends Core cleans up the snapshot and, for a qualified Session, replaces the compute through [cold replacement](#cold-replacement-after-checkpoint-retention) | | 15 s | Inactivity of an idle node allocation before capacity-pressure suspension | Core can suspend it ahead of the idle timeout when compatible waiting demand exists | | 2 min | Inactivity of an idle allocation on a node with no free active slot and a free retained slot | Core can suspend it ahead of the idle timeout without waiting demand, so the next Session finds a free slot | +| 10 min | Time from an allocation's creation until Core observes its compute running with a completed bootstrap | Core fails the Environment, then calls `Kill` and releases the allocation as for any cleanup. An unsettled `Create` keeps its cleanup ownership until it settles | | 50 MiB plus 32 bytes (`MaxCommandInputBytes`) | `Command.Stdin` | The command is invalid | The online window and the other node timers are in the [sandbox node protocol](../contracts/agents-api/node-generation-protocol.md#timeouts-and-limits). diff --git a/docs/zh/sandbox-provider.md b/docs/zh/sandbox-provider.md index 96845ce7d..5aa74e251 100644 --- a/docs/zh/sandbox-provider.md +++ b/docs/zh/sandbox-provider.md @@ -1,7 +1,7 @@ --- title: "沙箱提供商协议" source: docs/sandbox-provider.md -source_hash: 0bb4f94a1520a4f03901d206b9f60458ea8725f52b960dcfd31861325d81f83e +source_hash: b4a7da9ff24dd556a34b2a32c51b534b0dbac8464f840c493f019748fae3792f --- 沙箱提供商协议连接 Core 与为 Core 管理的（`openai_hosted`）Environment 提供计算资源的 adapter，并由该 adapter 运行启动 Runtime daemon 的有界引导流程。本页面向 adapter 作者：说明提供商必须实现什么，以及可以依赖 Core 做什么。microsandbox、E2B 和 OpenSandbox 实现了该协议。 @@ -173,7 +173,7 @@ adapter 不实现这些行为，但依赖它们。代码层面的机制见 [Core - Core 在调用 `Create` 之前提交 allocation、其 daemon 凭据摘要和 Session 绑定。只有新的 allocation receipt 才允许 `Create`；重试和 Core 重启只观察同一个 `Reference`，不重放 `Create`，也不轮换凭据。 - 每个 node 有一个串行 lifecycle worker，同一 node 的调用从不并发；direct 提供商的 allocation 共用一个 worker。某个 node 上卡住的提供商不会阻塞其他 node。 - Core 从不根据引擎或提供商名称推断引导完成；由 adapter 通过 `Info.BootstrapComplete` 声明。 -- Core 只在撤销 device 权限、结清待处理 input 并请求取消之后才调用 `Kill`。临时提供商故障、未知的 `Create` 结果和已停止的计算资源都不证明永久失败。 +- Core 只在撤销 device 权限、结清待处理 input 并请求取消之后才调用 `Kill`。临时提供商故障、未知的 `Create` 结果和已停止的计算资源都不证明永久失败。[超时与限制](#timeouts-and-limits)中的启动期限限定了等待从未进入运行状态的计算资源的时间：它结束该 Environment，但并不声明 `Create` 已失败。 - Core 只有在确认清理 allocation 的计算资源与 volume，并证明原始 `Create` 已结算后才释放 allocation。在此之前它保留清理归属，包括观察到资源不存在之后，有界扫描继续捕获迟到的资源，不再次 `Create`。公开删除 Session 之后同样适用。 - Core 从不在离线 node、超时或空读取时释放 allocation，也不对不[在线](../../contracts/agents-api/zh/node-generation-protocol.md#connection)的 node 执行 lifecycle 操作。保留的归属需要精确的提供商证据：socket 路径、实例缺失或空列表都不证明已清理，也不授权替代。 - Placement 是自动的，调用方不能选择 node。活动 allocation 即使离线也保留原 node。确认释放后，符合条件的保留 Session 可以再次预约兼容容量，包括在其他 node 上。释放 active 容量前必须确认 source 已拆除，未知操作保留其预约。等待、排序和 generation 选择由[部署契约](../../contracts/agents-api/zh/sandbox-deployment.md#generation-ownership-and-rollout)负责。 @@ -310,6 +310,7 @@ Core 在释放 allocation 和 placement 归属前，确认原始创建已结算 | 快照保留期 | 支持检查点的 allocation，由[暂停策略](../../contracts/agents-api/zh/sandbox-deployment.md#suspension-policy)设定 | 保留期结束时 Core 清理快照，并对符合条件的 Session 通过[冷替换](#cold-replacement-after-checkpoint-retention)替换计算资源 | | 15 s | 空闲 node allocation 在容量压力暂停之前的无活动时间 | 存在兼容的等待需求时，Core 可以早于空闲超时暂停它 | | 2 min | 所在 node 没有空闲 active slot 但有空闲 retained slot 时，空闲 allocation 的无活动时间 | Core 可以早于空闲超时、在没有等待需求时暂停它，让下一个 Session 找到空闲 slot | +| 10 min | 从 allocation 创建到 Core 观测到其计算运行且引导完成的时间 | Core 使 Environment 失败，然后像任何清理一样调用 `Kill` 并释放 allocation。未结算的 `Create` 在结算之前保留清理归属 | | 50 MiB 加 32 字节（`MaxCommandInputBytes`） | `Command.Stdin` | 命令无效 | 在线窗口和其他节点计时器见[沙箱节点协议](../../contracts/agents-api/zh/node-generation-protocol.md#timeouts-and-limits)。 diff --git a/services/core/internal/deployment/allocations.go b/services/core/internal/deployment/allocations.go index 56e5c7569..5e737ec12 100644 --- a/services/core/internal/deployment/allocations.go +++ b/services/core/internal/deployment/allocations.go @@ -178,6 +178,18 @@ func (e *ExecutionOperations) ReleaseIdleUnattached(ctx context.Context, owner A return e.cleanup(ctx, owner, idleRelease) } +// ReleaseUnstarted requests the cleanup of an allocation whose compute has not +// reached running with a completed bootstrap within ComputeStartDeadline of +// its creation, by the database clock. It fails the Environment, standalone or +// Session-owned. Any other allocation is ErrAllocationConflict. +func (e *ExecutionOperations) ReleaseUnstarted(ctx context.Context, owner Allocation) (Allocation, error) { + return e.cleanup(ctx, owner, startDeadline) +} + +// ComputeStartDeadline bounds the wait for compute to start, because a Provider +// can leave it pending or failed forever and nothing else ends the allocation. +const ComputeStartDeadline = 10 * time.Minute + // cleanupEvidence is the provider proof a cleanup commit consumes. type cleanupEvidence int @@ -186,6 +198,7 @@ const ( creationAbsent computeLost idleRelease + startDeadline ) func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, evidence cleanupEvidence) (Allocation, error) { @@ -205,6 +218,11 @@ func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, evi return err } } + if evidence == startDeadline { + if err := checkStartDeadline(tx, current); err != nil { + return err + } + } if lost && (current.State != "running" || current.ComputePhase != "running" || current.ComputeRevision != owner.ComputeRevision || !current.CreateSettled) { return ErrAllocationConflict } @@ -239,7 +257,9 @@ func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, evi if !retain && lost && revoked.AttachedSessionID != "" { return ErrAllocationConflict } - if !retain && revoked.AttachedSessionID != "" { + // A standalone Environment fails only at the start deadline; its other + // cleanups end it in RequestCleanup. + if !retain && (revoked.AttachedSessionID != "" || evidence == startDeadline) { err = sessions.TerminateEnvironment(ctx, tx, revoked.Expired, sessions.ProvisioningFailureReason, sessions.ProvisioningFailure{Category: sessions.ProvisioningComputeReleased}.Detail()) } } @@ -247,7 +267,7 @@ func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, evi return err } pending, err := tx.RequestCleanup(current) - if err != nil || evidence == cleanupRequested || evidence == idleRelease { + if err != nil || evidence == cleanupRequested || evidence == idleRelease || evidence == startDeadline { result = pending return err } @@ -264,6 +284,23 @@ func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, evi return result, nil } +// checkStartDeadline admits only an allocation that never reached running: +// ObserveRunning is the only writer of the running state, and a suspended or +// restored allocation keeps it. +func checkStartDeadline(tx AllocationCleanupTx, current Allocation) error { + if current.State != "creating" { + return ErrAllocationConflict + } + activity, err := tx.LoadActivity(current) + if err != nil { + return err + } + if activity.ObservedAt.Sub(current.CreatedAt) < ComputeStartDeadline { + return ErrAllocationConflict + } + return nil +} + func checkIdleUnattached(ctx context.Context, tx AllocationCleanupTx, current Allocation) error { if current.State != "running" || current.ComputePhase != "disabled" || current.AttachedSessionID != "" { return ErrNotIdle diff --git a/services/core/internal/execution/runtime_lifecycle.go b/services/core/internal/execution/runtime_lifecycle.go index 2bc9fd470..104bca84e 100644 --- a/services/core/internal/execution/runtime_lifecycle.go +++ b/services/core/internal/execution/runtime_lifecycle.go @@ -428,14 +428,27 @@ func (r *runtimeLifecycle) observe(ctx context.Context, owner deployment.Allocat if err := r.lease.CheckOwnership(ctx); err != nil { return err } + // The start deadline bounds the wait whatever the Provider reports; the + // database clock decides, and a refusal leaves the allocation as is. + if owner.State == "creating" && time.Since(owner.CreatedAt) >= deployment.ComputeStartDeadline { + released, err := r.deployment.ReleaseUnstarted(ctx, owner) + if err == nil { + return r.observe(ctx, released) + } + if !errors.Is(err, deployment.ErrAllocationConflict) { + return err + } + } info, err := provider.GetInfo(ctx, runtimeReference(owner)) cleanup := owner.Ended || owner.Expired || owner.State == "cleanup_pending" - // Resource drift disqualifies execution, not cleanup already authorized by - // Core. Kill independently verifies ownership before removing anything. - invalidCleanup := cleanup && errors.Is(err, sandbox.ErrInvalid) && !errors.Is(err, sandbox.ErrOwnership) - if err != nil && !errors.Is(err, sandbox.ErrNotFound) && !invalidCleanup { + // An unreadable or drifted Provider state disqualifies execution, not + // cleanup already authorized by Core: Kill independently verifies + // ownership before removing anything. + if err != nil && !errors.Is(err, sandbox.ErrNotFound) && (!cleanup || errors.Is(err, sandbox.ErrOwnership)) { return err } + // A configuration rejection still reports the Reference and settlement. + invalidCleanup := cleanup && errors.Is(err, sandbox.ErrInvalid) running := err == nil && info.Reference == runtimeReference(owner) && info.ProviderID != "" && info.State == "running" if (err == nil || invalidCleanup && info.CreateSettled) && info.Reference != runtimeReference(owner) { return sandbox.ErrOwnership diff --git a/services/core/internal/execution/runtime_workspace_settlement_test.go b/services/core/internal/execution/runtime_workspace_settlement_test.go index 2eeb97193..105ee0fd4 100644 --- a/services/core/internal/execution/runtime_workspace_settlement_test.go +++ b/services/core/internal/execution/runtime_workspace_settlement_test.go @@ -224,13 +224,10 @@ func TestDeletedUnknownWorkspaceCreationRequiresOwnedSettlementReceipt(t *testin t.Fatal(err) } observeErr := lifecycle.observe(ctx, allocation) - if observation == "unknown" && !errors.Is(observeErr, sandbox.ErrComputeUnconfirmed) { - t.Fatal("unknown observation changed semantics", observeErr) - } if observation == "foreign" && !errors.Is(observeErr, sandbox.ErrOwnership) { t.Fatal("foreign settlement accepted", observeErr) } - if (observation == "absent" || observation == "settled") && observeErr != nil { + if observation != "foreign" && observeErr != nil { t.Fatal(observeErr) } if _, err = lifecycle.workspaces.DeleteBatch(ctx, ""); err != nil { diff --git a/services/core/tests/integration/compute_start_deadline_test.go b/services/core/tests/integration/compute_start_deadline_test.go new file mode 100644 index 000000000..498c147da --- /dev/null +++ b/services/core/tests/integration/compute_start_deadline_test.go @@ -0,0 +1,95 @@ +package integration + +import ( + "context" + "strings" + "testing" + + "github.com/google/uuid" + + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" +) + +// failedComputeProvider settles Create but its compute is natively failed, so +// it never reaches running with a completed bootstrap. +type failedComputeProvider struct{ lifecycleProvider } + +func (p *failedComputeProvider) Create(_ context.Context, b sandbox.Bootstrap) (sandbox.Info, error) { + p.mu.Lock() + defer p.mu.Unlock() + p.creates++ + info := sandbox.Info{Reference: b.Reference, ProviderID: b.AllocationID, State: "failed", CreateSettled: true} + p.resources[b.AllocationID] = info + return info, nil +} + +func backdateAllocations(t *testing.T, s *Store) { + t.Helper() + if _, err := s.pool.Exec(t.Context(), "UPDATE runtime_allocations SET created_at = created_at - interval '11 minutes'"); err != nil { + t.Fatal(err) + } +} + +func TestComputeThatNeverStartsFailsStandaloneEnvironmentAtDeadline(t *testing.T) { + s, key := configuredStore(t) + p := &failedComputeProvider{lifecycleProvider{resources: map[string]sandbox.Info{}}} + w, _ := managedWorker(t, s, key, p) + request := publicPoster(t, s, uuid.NewString(), "start-deadline") + environment := createStandalone(t, request) + reconcile := func() { + for range 4 { + if err := w.ReconcileManagedRuntimes(t.Context()); err != nil { + t.Fatal(err) + } + } + } + reconcile() + _, body := request("GET", "/agents/environments/"+environment, "") + if p.kills != 0 || strings.Contains(body, `"failed"`) { + t.Fatalf("before the deadline: kills=%d environment=%s", p.kills, body) + } + backdateAllocations(t, s) + p.unavailable = true // The deadline holds whatever the Provider reports. + reconcile() + _, body = request("GET", "/agents/environments/"+environment, "") + var state string + if err := s.pool.QueryRow(t.Context(), "SELECT state FROM runtime_allocations WHERE environment_id=$1", environment).Scan(&state); err != nil { + t.Fatal(err) + } + if p.kills == 0 || state != "released" || !strings.Contains(body, `"failed"`) { + t.Fatalf("after the deadline: kills=%d allocation=%s environment=%s", p.kills, state, body) + } +} + +func TestRunningAllocationIsNeverFailedByStartDeadline(t *testing.T) { + s, key := configuredStore(t) + p := &lifecycleProvider{resources: map[string]sandbox.Info{}} + w, _ := managedWorker(t, s, key, p) + request := publicPoster(t, s, uuid.NewString(), "start-deadline") + environment := createStandalone(t, request) + for range 3 { + if err := w.ReconcileManagedRuntimes(t.Context()); err != nil { + t.Fatal(err) + } + } + var state string + if err := s.pool.QueryRow(t.Context(), "SELECT state FROM runtime_allocations WHERE environment_id=$1", environment).Scan(&state); err != nil || state != "running" { + t.Fatal("allocation not running", state, err) + } + backdateAllocations(t, s) + p.mu.Lock() + for id, info := range p.resources { + info.State = "stopped" + p.resources[id] = info + } + p.mu.Unlock() + for range 3 { + if err := w.ReconcileManagedRuntimes(t.Context()); err != nil { + t.Fatal(err) + } + } + _, body := request("GET", "/agents/environments/"+environment, "") + if p.kills != 0 || strings.Contains(body, `"failed"`) { + t.Fatalf("running allocation was failed: kills=%d environment=%s", p.kills, body) + } +} diff --git a/services/core/tests/integration/runtime_configuration_cleanup_test.go b/services/core/tests/integration/runtime_configuration_cleanup_test.go index e72ff26fe..ad621a2e3 100644 --- a/services/core/tests/integration/runtime_configuration_cleanup_test.go +++ b/services/core/tests/integration/runtime_configuration_cleanup_test.go @@ -78,7 +78,7 @@ func TestManagedRuntimeConfigurationCleanup(t *testing.T) { {name: "unknown creation stays retained", loseCreate: true, inspectionError: sandbox.ErrInvalid, wantKill: true}, {name: "foreign settlement rejected", loseCreate: true, settleInspection: true, foreign: true, inspectionError: sandbox.ErrInvalid}, {name: "foreign ownership rejected", loseCreate: true, inspectionError: sandbox.ErrOwnership}, - {name: "unavailable observation stays retained", loseCreate: true, inspectionError: sandbox.ErrComputeUnconfirmed}, + {name: "unavailable observation stays retained", loseCreate: true, inspectionError: sandbox.ErrComputeUnconfirmed, wantKill: true}, {name: "kill ownership rejection stays retained", inspectionError: sandbox.ErrInvalid, killError: sandbox.ErrOwnership, wantSettled: true, wantKill: true}, {name: "kill unavailable stays retained", inspectionError: sandbox.ErrInvalid, killError: sandbox.ErrComputeUnconfirmed, wantSettled: true, wantKill: true}, } {