[pimlock]

Summary

Spike 02b of the egress usage monitoring RFC. A new TCP connection middleware hook with a STATS mode. A middleware service sees a preflight for every admitted upstream connection, gets cumulative byte counts while it is open, and can deny or close it. Stacked on NVIDIA#4359 (base 1c982faca). This is spike-quality code: it works end to end, but it skips docs, gateway tests, and parts of the contract design.

Read first: examples/supervisor-middleware-egress-usage/README.md. It covers the example service, configuring limits in the policy, and running the smoke suite.

Compare with the OCSF-based approach in #5 (spike 01b).

Context

Network policy decides where a sandbox can connect, but not how much it uses an allowed endpoint. An agent allowed to reach a package proxy can download its whole catalog. Several sandboxes can also use one writable path as a message board. The RFC asks for usage accounting per sandbox and policy key, plus a way to act on it.

The RFC's running example: sandboxes reach artifactory.internal:443 under the policy key pkg_proxy, which allows GET and PUT. Two detections cover the two shapes of misuse:

• D2, volume: one sandbox's bytes for one policy key pass a threshold within a window. The reaction cuts the transfer, denies the key, or stops the sandbox.
• D1, fleet writers: more than 5 distinct sandboxes send PUT to one host within 5 minutes. The reaction denies writes for those sandboxes.

This spike targets D2 by extending the existing middleware model (NVIDIA#4359) from HTTP content to TCP connections. Enforcement then happens in the data path rather than through a policy update.

How this spike solves it

flowchart LR
  subgraph SUP["Sandbox supervisor"]
    P["Network policy<br/>admission"] --> H["TCP hook stage<br/>preflight, stats, close<br/>(new)"]
    H --> R["Relays + metered stream<br/>(new)"]
  end
  H <-- "TcpHooks.EvaluateConnection<br/>bidi stream per connection" --> M["Egress-usage service<br/>windows per sandbox + key<br/>(example, operator-run)"]
  POL["Sandbox policy<br/>network_middlewares.config"] -. "limits" .-> M
  GW["Gateway"] -- "ValidateConfig on policy set" --> M
  R --> U["Upstream<br/>(tls: skip, raw TCP, HTTP)"]

Loading

1. Preflight: after network policy admits a connection and before the upstream dial, the supervisor opens one TcpHooks.EvaluateConnection stream per matching stage. The preflight carries host, port, inspection kind, policy key, sandbox id, and the policy's middleware config. The service answers continue, inspect with STATS at an interval (1–60 s), or close, which denies. It covers CONNECT (including tls: skip and raw TCP), transparent TCP, and the forward proxy.
2. Count: a metered wrapper on the upstream stream counts bytes actually sent and received. The stage sends cumulative bytes_out and bytes_in each interval, only when they changed, and an end event with final totals.
3. Decide: the example service sums deltas into a sliding window per sandbox and policy key. Limits come from the policy's middleware config, whose shape this example service defines (OpenShell passes it through opaquely): max_bytes, window, interval, block_for, direction, and per-key overrides. The gateway calls ValidateConfig on policy set, so invalid limits are rejected up front.
4. React: over the budget, the service sends close on every open connection for that key. Pending reads and writes abort within milliseconds. New connections for the key are denied at preflight for block_for. Other keys keep working. Denials and cuts emit OCSF Network Activity events with reason egress_budget_exceeded.
5. Fail closed: a preflight error, timeout, or unavailable service denies the connection, and a broken stream closes it. fail_open is rejected for TCP stages.

Strengths

• Enforcement in the data path. A close reaches the relay in milliseconds. It needs no config delivery, no API credential, and no policy rewrite.
• Narrow, self-recovering blocks. Only the tripped sandbox and policy key are blocked, and only for block_for.
• Admission control. The preflight can say no before any bytes move, which makes it a natural path toward core budgets.
• Shorter intervals. 1 s is allowed. At 10 MB/s the smoke run leaked about 8–9 MB past a 20 MB threshold.
• Catches split transfers. End totals count, so 200 MB split across 100 short connections was blocked after 0.1 MB in the eval.
• Limits in the policy. Budgets change with openshell policy set, per policy key, without redeploying the service.
• Content-blind. It works the same for tls: skip, raw TCP, and inspected HTTP, and never sees payloads.

Weaknesses

• A new public extension contract. It adds a service, an operation, a phase, and a capability, plus semantics for end delivery and backpressure.
• On the hot path. Each matched connection costs one synchronous round trip and keeps a stream open for its whole life. The spike caps observed connections at 4,096.
• Fail-closed only. A service outage denies all matched egress, and the service must be up before the gateway starts. Operators need to run it highly available.
• Opt-in per policy. A sandbox whose policy has no matching network_middlewares entry is not observed.
• Fleet detection is the service's job. D1 needs cross-sandbox state in an HA, stateful service. It also needs a headers-only HTTP binding, and on hosts that mix inspected and tls: skip endpoints, port selectors. D1 is not implemented here.
• No data for other uses. Counts go only to the service, not to SIEM or observability sinks as they do in 01b.

Changes

• Proto:
  • TcpHooks.EvaluateConnection bidi stream, operation TCP_CONNECTION, phase PRE_CONNECT, and capability openshell.supervisor-middleware.tcp-hooks.
  • Events: preflight, stats, end. Results: preflight result, close.
• Supervisor middleware: tcp_hooks.rs holds the metered stream, the per-stage tasks, and the session.
• Supervisor network: hook call sites in proxy.rs, and OCSF deny and close events in proxy/tcp_hook.rs.
• Uninspectable gate: TCP-only services don't require traffic to be inspectable. The static tls: skip conflict check moved to gateway validation (validate_tcp_hook_rules).
• examples/supervisor-middleware-egress-usage/:
  • The service: usage.rs (windows and reactions) and config.rs (policy limits).
  • policy.yaml, a host-run upstream, and smoke.sh.

Results

Check (smoke, 20 MB / 60 s, 1 s interval)	Result
200 MB over tls: skip at 10 MB/s	cut after 2.8 s, about 28 MB through
New pkg_proxy connection after the trip	denied; other_svc still allowed
40 × 1 MB raw TCP connections	connection 21 denied
policy set with max_bytes: lots	rejected by ValidateConfig
policy set lowers other_svc from 50 MB to 10 MB	the same 15 MB transfer that passed before is now cut

In the eval harness (20 MiB / 60 s, 5 s interval), 02b blocked 9 of 10 abuse scenarios with no false positives across 5 benign scenarios. The 200 MB burst was cut 3.0 s after the threshold with 30 MB through. The miss is D1, which is not implemented.

Testing

• smoke.sh --test-suite on the Docker driver
• Unit tests: openshell-core, openshell-supervisor-middleware, openshell-policy, openshell-supervisor-network, openshell-server middleware, and the example's config and window tests
• Clippy and fmt (pre-commit)
• Gateway tests for validate_tcp_hook_rules. The old tls: skip conflict tests are ignored or removed.
• Docs
• E2E suites