Skip to content

[langfun/env] Suspend and resume sandbox services across backend outages. - #754

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_964499236
Open

[langfun/env] Suspend and resume sandbox services across backend outages.#754
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_964499236

Conversation

@copybara-service

Copy link
Copy Markdown

[langfun/env] Suspend and resume sandbox services across backend outages.

A sandbox service that depends on a remote backend had no recovery path when
that backend went away:

  • The errors raised while connecting to a backend are typically neither
    EnvironmentError nor SandboxStateError, so they escaped the outage
    retry loop of BaseSandboxService and crashed the hosting process.
  • Once an outage outlasted outage_grace_period, the service called the
    terminal shutdown(). acquire() then kept raising
    SandboxServiceOutageError for the rest of the process lifetime, even
    after the backend came back, while the process itself stayed up.

BaseSandboxService now:

  • exposes a transient_outage_errors hook, so a backend-specific subclass
    can declare the errors it raises while establishing its connection. Those
    errors enter the existing outage retry loop, which remains bounded by
    outage_grace_period.
  • adds Status.SUSPENDED plus a suspend() / _resume() pair. Giving up
    past the grace period now releases all resources but leaves the service
    resumable, and the next acquire() brings it back into service with a
    fresh outage budget. If the backend is still unreachable, the service
    falls back to SUSPENDED so that a later acquire() can try again.
    shutdown() stays terminal.

A suspended service still reports is_online == False, so readiness signals
are unchanged.

…ges.

A sandbox service that depends on a remote backend had no recovery path when
that backend went away:

*   The errors raised while connecting to a backend are typically neither
    `EnvironmentError` nor `SandboxStateError`, so they escaped the outage
    retry loop of `BaseSandboxService` and crashed the hosting process.
*   Once an outage outlasted `outage_grace_period`, the service called the
    terminal `shutdown()`. `acquire()` then kept raising
    `SandboxServiceOutageError` for the rest of the process lifetime, even
    after the backend came back, while the process itself stayed up.

`BaseSandboxService` now:

*   exposes a `transient_outage_errors` hook, so a backend-specific subclass
    can declare the errors it raises while establishing its connection. Those
    errors enter the existing outage retry loop, which remains bounded by
    `outage_grace_period`.
*   adds `Status.SUSPENDED` plus a `suspend()` / `_resume()` pair. Giving up
    past the grace period now releases all resources but leaves the service
    resumable, and the next `acquire()` brings it back into service with a
    fresh outage budget. If the backend is still unreachable, the service
    falls back to `SUSPENDED` so that a later `acquire()` can try again.
    `shutdown()` stays terminal.

A suspended service still reports `is_online == False`, so readiness signals
are unchanged.

PiperOrigin-RevId: 964499236
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant