Repository navigation
Description
User Story
I'm retiring e2e/rust (about 70 test files). Not every file should move: some duplicate tests already on testkit, some are benchmarks, and some test the environment instead of the product. I need a per-file decision and a batch order so each migration PR stays small.
Description
Sort every file in e2e/rust/tests/ into keep, drop or merge, and assign kept files a destination and a batch. Judge each file with three questions:
- Does a ported test already assert this?
- Does it test product behavior, or the test environment?
- Would a regression here surprise us in production?
If the answer to the third is no, drop it.
Starting point, from file names and a skim of a few files (not yet from reading every test body):
- Conformance candidates (every driver must pass): policy and network enforcement (
landlock,bypass_detection,no_proxy,policy_activation,live_policy_update,proxy_egress_pipeline,forward_proxy_*,transparent_tcp,websocket_conformance), sandbox API (sandbox_labels,sandbox_templates,ssh_host_identity,port_forward,upload_create,rootfs_tar,ephemeral_cleanup,listener_lifetime), providers and credentials (provider_*,credential_gating,stable_external_placeholders,provider_token_exchange,service_bearer_passthrough,settings_management),workspace_lifecycle. - Feature or driver candidates:
oidc_pkce,additional_ca,config_push,local_driver_token_restart,otlp_relay,ocsf_*,mcp_sessionless,default_image; thekubernetes_*,workspace_namespace_*andcredential_driversfiles;podman_*,user_namespaces;docker_*,gateway_start;vm_*. - Possible duplicates to diff before porting:
sandbox_lifecycle.rs: conformancelifecyclealready covers restart of a stopped sandbox and delete while stopped. The canonical-main, detached-main, exec-recovery and large-output tests don't appear to be covered.host_gateway_alias.rsandpodman_host_gateway.rs: their first test checks the samehost.openshell.internalcontract as thehost-gatewaysuite from test(tmachine): prepare runner environments for the e2e/rust migration #5277.host_gateway_alias.rsalso covers EOF handling and provider bindings, which are separate.podman_host_gateway.rsis not a pure duplicate. It also asserts thathost.openshell.internalstays reachable aftersandbox stopandsandbox start, which thehost-gatewaysuite doesn't cover. That contract is driver-agnostic, so port it to thehost-gatewaysuite (or conformancelifecycle) on every driver that supports stop/start. Its other assertion, the policy-DNS relay in/etc/resolv.conf(nameserver 127.0.0.53), is Podman-specific: keep it as a Podman driver test. Don't drop the file as a duplicate until both have a home.
user_namespaces.rsandpodman_oci_identity.rs: overlap the ports from test(e2e): migrate the features and drivers suites to testkit #5137.
- Drop or keep out of CI:
live_internet_traffic_perfandinternet_network_perf(opt-in benchmarks that use the live Internet);docker_preflight(checksopenshell doctoroutput without Docker, so it needs an environment with no Docker). - Needs a decision on the setup: the three
*_corporate_proxy.rsfiles (about 2,400 lines, mostly environment setup),gpu,core_dump_hardening,edge_tunnel_e2e. - Heavy and last:
kubernetes_shared_supervisor*,kubernetes_ha_rebalancing. These are slow, and the flake census shows Kubernetes lifecycle races, so port them last or only the ones guarding recent bugs.
Batches and their dependencies are in the "Migrating e2e/rust" table in #5281: a credential_gating pilot, then Podman, Docker and the common driver-agnostic files (in parallel, since they touch different crates), then gateway reconfiguration, then Kubernetes. Record each file's batch in that table's terms.
Context
- Depends on test(tmachine): prepare runner environments for the e2e/rust migration #5277 (Python image,
host.openshell.internalon k3s) for tests that use them. Helpers come from the harness-helpers issue; runner timeouts and parallelism from the runner-knobs issue. - Each batch deletes what it ports, so
e2e/rustshrinks as the work lands.
Part of #5281.
Definition of Done
- Every file in
e2e/rust/tests/has a recorded decision (port, merge, drop, keep on host) with a reason. - Each kept file has a destination family and a batch.
- Overlaps above are resolved by reading the test bodies.
Activity
matthewgrossman commentedon Nov 7, 2026
I read every test body in
e2e/rust/tests/and recorded a decision, destination and PR for each. No E2E tests were run for this plan.Result: 222 entries (tests, unit tests and two helper/module entries) across 70 files.
credential_gatingtests; #4606: 3upload_createtests)Milestones. Milestone 1, these 9 PRs, deletes the
e2e/rusttests and harness. Deleting the host-lane wrapper scripts (e2e/rust/e2e-*.sh,e2e/support/*) is a later, separate milestone that needs a KVM host provisioner. Every environment tmachine can host runs on tmachine; host lanes remain only for what it can't (the VM driver and GPU).How environments reach tests
An environment (tmachine, or for VM and GPU the existing host-lane wrapper script) publishes
OPENSHELL_TEST_<SERVICE>_*variables. Atests/support/<service>helper reads them and fails, never skips, if one is missing (tests/support/keycloakworks this way today). Which tests run where is set by the rows in.github/workflows/e2e-suites.yml; until the script-deletion milestone, the VM and GPU wrapper scripts choose their own suites. Each PR adds the helpers and environment variants its tests need, and the last PR to touch a legacy file deletes it.Decisions needed
OPENSHELL_TEST_GATEWAY_STOPandOPENSHELL_TEST_GATEWAY_START, andtests/support/gateway_controlruns them. The restart test goes infeatures/gateway-restart, not conformance, because it also checks that the container or VM survives. Blocks part of PR6.e2e-vm.shruns the VM suites on the existing KVM lane.tests/support/keycloak. The alternative is an OIDC-only mode in testkit. Blocks PR7.PRs
Each PR has a collapsed section below with its test-by-test table, helpers, deleted files, defaults and Definition of Done. Each section can be filed as its own issue or handed to an agent as is.
Order: PR1 and PR4 now; PR2 and PR3 after #5318; then PR5; then PR6 and PR7; then PR8 and PR9. #5295 (per-test timeouts and test groups) matters only for PR8 and PR9. Whichever PR merges last also deletes the
e2e/rusttest crate (tests, harness,Cargo.toml); the wrapper scripts stay until the script-deletion milestone.Mapping to the batches in #5281: PR1–PR4 are "Common", PR5 is "Podman" and "Docker" combined (most of those tests check behavior both engines share), PR6 and PR7 are "Gateway reconfiguration", PR8 is "Kubernetes", PR9 is "Host-only".
Every file, its decision and its PR
additional_ca.rsfeatures/additional-ca,drivers/*bypass_detection.rsconfig_push.rsfeatures/config-pushcore_dump_hardening.rscredential_drivers.rsdrivers/kubernetescredential_gating.rsdefault_image.rsdocker_corporate_proxy.rsopenshell-gatewaycrate test,features/corporate-proxydocker_preflight.rsopenshell-clicrate testdriver_config_volume.rsfeatures/driver-configedge_tunnel_e2e.rsopenshell-servercrate testephemeral_cleanup.rsdrivers/*forward_proxy_graphql_l7.rsforward_proxy_jsonrpc_l7.rsforward_proxy_l7_bypass.rsgateway_start.rsfeatures/gateway-restartgpu.rsgpu/device_selection.rsfeatures/gpu-cdi,tests/support/gpugpu/workloads.rsfeatures/gpuhost_gateway_alias.rsfeatures/host-gatewayinternet_network_perf.rskubernetes_corporate_proxy.rsfeatures/corporate-proxy, driver unit testkubernetes_ha_rebalancing.rsdrivers/kuberneteskubernetes_runtime_reconcile.rsdrivers/kuberneteskubernetes_shared_supervisor.rsdrivers/kuberneteskubernetes_shared_supervisor_ha.rsdrivers/kuberneteskubernetes_shared_supervisor_scaling.rsdrivers/kuberneteslandlock.rslistener_lifetime.rslive_internet_traffic_perf.rstasks/scriptlive_policy_update.rslocal_driver_token_restart.rsdrivers/{docker,podman},features/gateway-restartmcp_sessionless.rsno_proxy.rsocsf_console.rsfeatures/ocsfocsf_jsonl_conformance.rsfeatures/ocsfoidc_pkce.rsfeatures/oidc-authotlp_relay.rsfeatures/otlp-relay, conformancepodman_corporate_proxy.rsfeatures/corporate-proxy,drivers/podmanpodman_gateway_start.rsfeatures/gateway-restartpodman_host_gateway.rsfeatures/host-gatewaypodman_oci_identity.rsdrivers/podmanpodman_resource_limits.rsfeatures/resource-limitspolicy_activation.rsfeatures/policy-activationport_forward.rsprovider_auto_create.rsprovider_files.rsprovider_readiness.rsfeatures/provider-readinessprovider_refresh_handles.rsfeatures/provider-refreshprovider_token_exchange.rsfeatures/provider-token-exchangeproxy_egress_pipeline.rsrootfs_tar.rsdrivers/vmsandbox_labels.rssandbox_lifecycle.rsfeatures/cli-attachsandbox_templates.rsservice_bearer_passthrough.rsfeatures/service-exposuresettings_management.rsshared_supervisor/mod.rstests/support/kubessh_host_identity.rsfeatures/gateway-restartstable_external_placeholders.rsfeatures/provider-readinesstransparent_tcp.rsfeatures/transparent-tcpupload_backpressure.rsupload_create.rsuser_namespaces.rsvm_bind_mount.rsdrivers/vmvm_corporate_proxy.rsopenshell-gatewaycrate test,features/corporate-proxyvm_gateway_start.rsfeatures/gateway-restartvm_overlay.rsdrivers/vmwebsocket_conformance.rsworkspace_lifecycle.rsworkspace_namespace_managed.rsdrivers/kubernetes, conformanceworkspace_namespace_operator.rsdrivers/kubernetesOverlaps called out in this issue:
sandbox_lifecycle.rs: the two tests conformance already covers merge into it; the rest port (PR1, PR4).host_gateway_alias.rs/podman_host_gateway.rs: reachability is dropped as a duplicate offeatures/host-gateway. The stop/start re-check moves tofeatures/host-gatewayon every row. The127.0.0.53resolver check is dropped, not kept as a Podman test: the policy-DNS relay is the same on every container driver, and the transparent-TCP tests prove it works.user_namespaces.rs: permanently ignored; dropped (PR8).podman_oci_identity.rs: only the partoci-imagedoesn't check is ported (PR1).PR details
PR1:
test(e2e): port sandbox lifecycle and API tests from e2e/rust to conformanceUser Story
As a maintainer retiring
e2e/rust, I want the driver-agnostic sandbox lifecycle and API tests on testkit, so they run on every conformance row and their legacy files can be deleted.Description
Port, merge or drop the tests below. All conformance destinations are under
tests/suites/conformance/src/. Merges keep every assertion of the tests they combine.sandbox_lifecycle::sandbox_stop_start_preserves_workspacepersistent_workspace.rs(persistent_workspace_survives_stop_start)sandbox connectsession is open", which moves to PR4's attach tests.sandbox_lifecycle::sandbox_can_be_deleted_while_stoppedlifecycle.rs(stopped_sandbox_can_be_deleted)sandbox_lifecycle::sandbox_exec_outlives_startup_recovery_deadlinelifecycle.rsexec; about 35 s per row.sandbox_lifecycle::detached_canonical_main_nonzero_exit_reaches_error+detached_canonical_main_exit_zero_reaches_completedlifecycle.rssandbox_lifecycle::canonical_main_and_exec_receive_declared_environment+canonical_tty_main_uses_sandbox_environmentlifecycle.rs(canonical_main_environment)spec.environmentandspec.tty. TTY case also checksHOME,USER,TERM.sandbox_lifecycle::detached_main_exit_during_provisioning_is_classified_as_workload_resultlifecycle.rsrun_to_completion.sandbox_lifecycle::on_failure_policy_replaces_runtime_and_preserves_workspacelifecycle.rsget()for restart count and phase; don't usewait_ready(it errors on the Error phase).ephemeral_cleanup::detached_ephemeral_{success,failure}_removes_sandbox_and_driver_resourceslifecycle.rsdrivers/*.landlock::hard_requirement_accepts_enriched_device_pathfilesystem_policy.rslandlock::{best_effort,hard_requirement}_rejects_replaced_policy_{path,ancestor}(4 tests)filesystem_policy.rscore_dump_hardening::sandbox_processes_disable_core_dumpsprocess.rsulimit -candulimit -Hcare 0.default_image::sandbox_from_default_imagelifecycle.rs(default_command_sandbox_is_listed_ready)/etc/os-releasecheck (ID,VERSION_ID).default_image::sandbox_from_explicit_nvidia_ubuntu_imageimage.rstestkit::base_image().sandbox_labels::sandbox_labels_are_stored_and_filterablelabels.rslabel_selector; 3 sandboxes; label values unique per run.sandbox_templates::sandbox_create_from_template_uses_reusable_workloadtemplates.rscreated_from_workload_template.name.sandbox_templates::get-after-delete, duplicate name, missing template (3 tests)templates.rs(template_crud_errors)NotFound/AlreadyExists, not text.settings_management::settings_global_override_round_tripsettings.rssandbox_setting_round_trip; (b)global_setting_overrides_sandbox_scope, which records and restores the prior global value.upload_backpressure::large_upload_preserves_content_and_metadata_outside_workdirfile_transfer.rsupload_download_round_trip; (b)large_upload_backpressurekeeps count, checksum and no-retry.podman_oci_identity::podman_supervisor_runs_as_oci_identity_without_capabilitiestests/suites/drivers/podmanoci-imagedoesn't already check.docker_preflight::doctor_check_fails_without_docker+doctor_check_output_shows_docker_labelcrates/openshell-cli/tests/doctor_integration.rsdocker_preflight::doctor_check_passes_with_dockercrates/openshell-cli/tests/doctor_integration.rsdockeronPATH; add a "nodockeronPATH" case.docker_preflight::doctor_check_error_includes_guidancedocker_corporate_proxy::docker_corporate_proxy_rejects_incoherent_configuration+vm_corporate_proxy::vm_corporate_proxy_rejects_incoherent_configurationcrates/openshell-gateway/tests/proxy_config_preflight.rsopenshell-gateway config preflightper driver. Needs approval in #5297.docker_corporate_proxy::proxy_settings_are_inserted_before_nested_driver_tables(unit test)edge_tunnel_e2e::gateway_status_reports_healthyedge_tunnel_e2e::ws_tunnel_status_through_edge_proxyopenshell-servercrate testinternet_network_perf::benchmark_complete_internet_pathlive_internet_traffic_perf::benchmark_live_internet_agent_traffictasks/scriptHelpers this PR adds
CliOutput::message()(replaces the legacycreate_expect_failure).Files deleted
core_dump_hardening.rs,default_image.rs,docker_preflight.rs,edge_tunnel_e2e.rs,ephemeral_cleanup.rs,internet_network_perf.rs,landlock.rs,live_internet_traffic_perf.rs,podman_oci_identity.rs,sandbox_labels.rs,sandbox_templates.rs,settings_management.rs,upload_backpressure.rs.Shared files:
sandbox_lifecycle.rsis deleted by whichever of PR1 and PR4 lands last;docker_corporate_proxy.rsby whichever of PR1 and PR6 lands last.vm_corporate_proxy.rsstays until PR9.Defaults (change in review if needed)
upload_backpressureruns on every conformance row: large uploads are a public contract and the transport differs per driver. If it nears the runner's 2-minute limit, split it further.ocsf_json_enabledfor the whole gateway, which is safe because the runner runs one test at a time. If a family later runs more threads (test(runner): apply per-family nextest config and a thread count to runner rows #5295), give it a nextest test group.Context
Definition of Done
tasks/*.tomland nextest config is removed.drivers/podman).PR2:
test(e2e): port network, L7 and policy-update tests from e2e/rust to conformanceUser Story
As a maintainer retiring
e2e/rust, I want the network enforcement, L7 and live policy-update tests on testkit, so every driver proves the same egress contract and the legacy files can be deleted.Description
All destinations are under
tests/suites/conformance/src/. Merges keep every assertion of the tests they combine.proxy_egress_pipeline::transparent_destination_denials_fail_connect_with_eaccesnetwork.rsbypass_detection(next row).bypass_detection::bypass_attempt_is_rejected_fastnetwork.rs(direct_connect_denials)allowed_ips; assertEACCES/EPERMwithin 8 s each.listener_lifetime::closing_or_exiting_listener_refuses_connections_and_releases_portnetwork.rslistener_lifetime::unrelated_exec_does_not_inherit_broker_preconnect_socketnetwork.rsSandboxBuilder::command.no_proxy::sandbox_reaches_localhost_without_proxy_environmentnetwork.rsforward_proxy_l7_bypass::forward_proxy_{allows,denies}_l7_*_request(2 tests)network_l7.rs(rest_l7_allows_get_denies_post)forward_proxy_l7_bypass::rest_upgrade_is_refused_in_enforce_and_audit_modesnetwork_l7.rsforward_proxy_graphql_l7::graphql_l7_enforces_high_level_and_raw_transparent_pathsnetwork_l7.rsforward_proxy_jsonrpc_l7::jsonrpc_l7_enforces_high_level_and_raw_transparent_pathsnetwork_l7.rsforward_proxy_jsonrpc_l7::jsonrpc_forward_proxy_hard_denies_response_frames_in_default_audit_modenetwork_l7.rsmcp_sessionless::legacy_and_multi_version_profiles_authorize_tools_through_sandboxnetwork_l7/mcp.rsmcp_sessionless::sessionless_discovery_tools_and_subscription_use_request_metadatanetwork_l7/mcp.rsproxy_egress_pipeline::policy_reload_updates_transparent_requests_and_closes_existing_http_streamnetwork_egress.rsproxy_egress_pipeline::ambiguous_policy_update_is_rejected_without_replacing_active_policynetwork_egress.rspolicy list -o json, not text.proxy_egress_pipeline::explicit_allowed_ips_and_implicit_ip_literals_succeed_transparentlynetwork_egress.rsproxy_egress_pipeline::TLS skip relay, middleware redaction, fail-closed and fail-open middleware, first-request overflow, chunked pipeline (6 tests)network_egress.rsproxy_egress_pipeline::http_credentials_are_rewritten_in_transparent_headers_and_bodiescredential_gating.rscredential_gatingmodule from #5318.websocket_conformance::websocket_text_placeholder_is_rewritten_transparentlycredential_gating.rshost_gateway_alias::static_provider_credentials_are_bound_to_profile_endpointscredential_binding.rshost.docker.internalwith an unbound host so the 403 check holds on every driver.port_forward::port_forward_echocli.rsforward start -dplus a cleanup hook that runsforward stop.live_policy_update::l7_append_target_scope_round_trippolicy_update.rslive_policy_update::live_policy_update_round_trip+live_policy_update_from_empty_network_policiespolicy_update.rspolicy get/list -o json; assert no revision stays Pending.live_policy_update::initial_sparse_policy_is_acknowledged_as_loadedpolicy_update.rspolicy list -o jsonstatuses; use a unique sandbox name.Helpers this PR adds
compute_driver()(refactor ofexpect_compute_driver),Sandbox::logs/logs_until,exec_until,sandbox_driver_config.tests/support/host-server: request handler withRequestand chunked bodies,TcpServer, a WebSocket text helper,free_port.policy::set,policy::from_yaml.Files deleted
bypass_detection.rs,forward_proxy_graphql_l7.rs,forward_proxy_jsonrpc_l7.rs,forward_proxy_l7_bypass.rs,listener_lifetime.rs,live_policy_update.rs,mcp_sessionless.rs,no_proxy.rs,port_forward.rs,proxy_egress_pipeline.rs,websocket_conformance.rs.host_gateway_alias.rsstays until PR5 ports its other two tests.Defaults (change in review if needed)
host.openshell.internal, and the legacy privateallowed_ipsranges go away. If a row can't reach the test host, that test keeps a container fixture on that row.nohup … &throughSandbox::exec; if it doesn't survive, it runs as the sandbox's main command. No new testkit API.Known risks
allowed_ipsapplies. If so, the IP-literal test keeps a container fixture.host.openshell.internalassumes every conformance row can reach the test host. Not yet verified on k3s.Context
install_provider,HttpRecorderand thecredential_gatingmodule this PR builds on).Definition of Done
tasks/*.toml(includinge2e:websocket-conformance) and nextest config is removed.PR3:
test(e2e): port provider and workspace tests from e2e/rust to testkitUser Story
As a maintainer retiring
e2e/rust, I want the provider and workspace CRUD tests on testkit, so they run on every conformance row and the legacy files can be deleted.Description
Conformance destinations are under
tests/suites/conformance/src/.provider_files::provider_file_open_update_and_detachproviders.rsprovider_auto_create::auto_created_provider_credential_available_in_sandboxproviders.rsservice_bearer_passthrough::service_bearer_passthrough_preserves_authorization_headertests/suites/features/service-exposureworkspace_lifecycle::workspace_full_crud_lifecycleworkspaces.rs-o jsonfor lists; check the provider's owning workspace.workspace_lifecycle::workspace_terminating_rejects_createsworkspaces.rsHelpers this PR adds
install_provider(..).config(..)(--config),create_providerfor built-in provider types, and a cleanup poll for provider delete (the gateway refuses to delete a provider still attached to a sandbox).Files deleted
provider_auto_create.rs,provider_files.rs,service_bearer_passthrough.rs,workspace_lifecycle.rs.Context
CliOutput::message()).create_providerfrom this PR.Definition of Done
tasks/*.tomland nextest config is removed.features/service-exposurehas rows in.github/workflows/e2e-suites.yml.PR4:
test(e2e): port CLI-driven exec, attach and SSH host identity tests from e2e/rustUser Story
As a maintainer retiring
e2e/rust, I want the tests that drive a long-runningopenshellCLI process on testkit, so attached create, exec streaming, reconnect and SSH host identity stay covered after the legacy harness is gone.Description
Conformance destinations are under
tests/suites/conformance/src/. Merges keep every assertion of the tests they combine.sandbox_lifecycle::sandbox_exec_large_output_is_completeexec_cli.rssandbox_lifecycle::piped_exec_stdin_crosses_grpc_message_limitexec_cli.rssandbox_lifecycle::canonical_main_exit_zero_completes_persistent_sandbox+canonical_main_nonzero_exit_preserves_statuscli.rs(cli_attached_create_streams_main_and_reports_exit)get --output json.sandbox_lifecycle::sandbox_create_with_no_keep_cleans_up_after_tty_command+…_preserves_failure_then_cleans_upcli.rs(cli_no_keep_removes_sandbox_after_main_exit)--nameand track the sandbox instead of parsing "Created sandbox:". Drop the Docker-only diagnostics; testkit failure context replaces them.sandbox_lifecycle::canonical_main_disconnect_reconnect_replays_history_for_same_processattach.rssandbox_lifecycle::canonical_main_exit_255_is_not_retried_as_transport_failureattach.rssandbox_lifecycle::canonical_main_connect_recovers_its_ssh_transporttests/suites/features/cli-attachsandbox_lifecycle::canonical_main_connect_forwards_pid_targeted_termination_and_reaps_sshtests/suites/features/cli-attachssh_host_identity::ssh_host_identity_survives_restarts_and_changes_after_recreationssh.rs(ssh_host_identity_stable_across_stop_start_and_new_on_recreate)ssh, recreate. Keeps the "auth.jsonunreadable" security check. The gateway-restart part moves to PR6.Helpers this PR adds
cli_child, a long-running CLI process with stdin, a line reader, its pid, kill/signal and wait.cli_ptyonly if plainCli::runcan't drive attachedsandbox create -- <cmd>.sshclient, and host procfs access for the pid-termination test.Files deleted
ssh_host_identity.rs.sandbox_lifecycle.rsis deleted by whichever of PR1 and PR4 lands last.Defaults (change in review if needed)
sandbox create -- <cmd>uses plainCli::runfirst;cli_ptyis added only if that fails.features/cli-attachon one Linux row (client-side logic).ssh_host_identityrequires a named gateway and fails with a clear message otherwise.Context
cli_childfrom this PR.Definition of Done
tasks/*.tomland nextest config is removed.features/cli-attachhas one Linux row in.github/workflows/e2e-suites.yml.PR5:
test(e2e): port Docker and Podman driver feature tests from e2e/rustUser Story
As a maintainer retiring
e2e/rust, I want the tests that need a container engine on testkit, running on both the Docker and Podman rows, so container-driver behavior stays covered and the legacy files can be deleted.Description
Most of these tests were gated to one engine in
e2e/rust, but the behavior they check is shared by Docker and Podman. Each new feature package runs on the Docker and Podman rows unless noted.ocsf_console::network_events_are_json_on_supervisor_stderr_onlytests/suites/features/ocsfocsf_jsonl_conformance::supervisor_jsonl_conforms_to_the_native_schematests/suites/features/ocsfOCSF-JSONrecords from supervisor stderr through engine logs. Keep the class, attribution, schema, version, opt-in and clean-stdout checks. Use an in-process host server instead of a container fixture./var/logfile-sink checkdriver_config_volume::sandbox_mounts_existing_driver_config_volumetests/suites/features/driver-configdriver_config_volume::oci_workspace_preparation_skips_nested_volume_ownershiptests/suites/features/driver-configtestkit::base_image()withoutapt, so no Internet. Runs on Podman too.driver_config_volume::sandbox_mounts_enabled_driver_config_bindtests/suites/features/driver-configlocal_driver_token_restart::local_driver_restart_or_stop_respects_main_process_lifecycletests/suites/drivers/{docker,podman}podman_resource_limits::sandbox_resource_limits_are_enforced_via_cgroupstests/suites/features/resource-limitshost_gateway_alias::sandbox_receives_eof_after_closing_http_responsetests/suites/features/host-gateway(response_eof.rs)host_gateway_alias::sandbox_reaches_host_openshell_internal_via_host_gateway_aliasfeatures/host-gatewaychecks reachability more strictly on 5 rows. Needs approval in #5297.podman_host_gateway::podman_sandbox_reaches_host_openshell_internaltests/suites/features/host-gateway(host_server_reachable_after_stop_start)nameserver 127.0.0.53check (the policy-DNS relay is the same on every driver; the transparent-TCP tests prove it works). Needs approval in #5297.transparent_tcp::local_container_native_tcp_uses_policy_dns_and_fails_closedtests/suites/features/transparent-tcptransparent_tcp::rootless_podman_private_dns_pins_do_not_cross_binary_rulestests/suites/features/transparent-tcphost.openshell.internal, since the supervisor can't resolve network aliases there.transparent_tcp::rootless_podman_musl_getaddrinfo_uses_udp_policy_dnsmusl_getaddrinfo_uses_udp_policy_dns, build the probe as a musl binary intests/support, and file a successor to #3009.This PR also adds the "no driver resources left" half of the ephemeral cleanup test that PR1 split off: one test per driver under
tests/suites/drivers/{docker,podman}. Kubernetes and VM get theirs in PR8 and PR9.Helpers this PR adds
tests/support/container-engine:running,wait_running, split stdout/stderrlogs,copy_from,build_image_from_dir. Uses the existingOPENSHELL_TEST_CONTAINER_ENGINE.Files deleted
driver_config_volume.rs,host_gateway_alias.rs,local_driver_token_restart.rs,ocsf_console.rs,ocsf_jsonl_conformance.rs,podman_host_gateway.rs,podman_resource_limits.rs,transparent_tcp.rs.Defaults (change in review if needed)
features/resource-limitsadds the k3s row only after a check that k3s enforces CPU and memory limits. File an issue that the VM driver ignores limits.ocsf_consoleandocsf_jsonlshare onefeatures/ocsfpackage.Context
Definition of Done
PODMAN_CI_TESTS,OPENSHELL_E2E_DOCKER_TEST,podmanE2eFollowUpBinaries, wrapper scripts, nextest groups) is removed..github/workflows/e2e-suites.yml; tests that ran only on one engine now run on both.PR6:
test(e2e): port gateway-configuration tests from e2e/rust to feature suitesUser Story
As a maintainer retiring
e2e/rust, I want the tests that need a specially configured or restartable gateway on testkit, so each one runs on an environment built for it instead of a wrapper script that rewrites gateway config.Description
Each of these tests needs a gateway configured a certain way (OTLP collector, push-mode config, additional CA, corporate proxy) or a gateway the test can restart. The environment sets that up and publishes how to reach it. The test never edits gateway config itself.
otlp_relay::agent_span_reaches_collector_with_attributiontests/suites/features/otlp-relayotlp_relay::span_emitted_before_exit_is_deliveredtests/suites/features/otlp-relayotlp_relay::relay_is_inert_without_collectortests/suites/conformance/src/otlp.rsconfig_push::streamed_policy_update_reaches_the_supervisortests/suites/features/config-pushpolicy_activation::invalid_image_provider_bundle_waits_for_repair_before_launchtests/suites/features/policy-activationtestkit::python_image()so it needs no Internet orapt.gateway_start::docker_…,podman_gateway_start::podman_…,vm_gateway_start::vm_gateway_restart_preserves_running_and_stopped_intent(3 tests)tests/suites/features/gateway-restartlocal_driver_token_restart(with its token-expiry check).additional_ca::additional_ca_trusts_matching_and_public_hosts_and_rejects_mismatchtests/suites/features/additional-ca+tests/suites/drivers/*docker_corporate_proxy::docker_corporate_proxy_routes_approved_tls_egress+trusts_ca_bundle_for_https_proxy,podman_corporate_proxy::same twotests/suites/features/corporate-proxy(approved_egress_uses_authenticated_proxy)no_proxy. Every proxied request proves CA trust. Thehttp://proxy scheme is no longer tested separately.docker_corporate_proxy::docker_corporate_proxy_trusts_intercepted_destination_tlstests/suites/features/corporate-proxy(intercepting_proxy_ca_is_trusted_by_supervisor_and_workload)podman_corporate_proxy)tests/suites/drivers/podman(proxy_auth_secret_removed_with_sandbox)provider_refresh_handles::long_running_process_survives_rotations_and_reconfigure_revokestests/suites/features/provider-refreshprovider_token_exchange::independent grants across sandboxes; A/B/A route recovery (2 tests)tests/suites/features/provider-token-exchangeprovider_readiness::acknowledged_provider_changes_apply_to_fresh_clients_and_revoke_retained_referencestests/suites/features/provider-readinessstable_external_placeholders::persistent rotation and revocation; ordinary revoked assertion (2 tests)tests/suites/features/provider-readinessprovider_readiness; tests stay separate.The Kubernetes (Helm) branch of
additional_camoves with the Kubernetes environment in PR8.Helpers and environment this PR adds
tests/support/gateway_control: readsOPENSHELL_TEST_GATEWAY_STOPandOPENSHELL_TEST_GATEWAY_START(each a command split on whitespace, likeOPENSHELL_TEST_CONTAINER_ENGINE) and fails if either is missing. Depends on decision A in test(e2e): decide which e2e/rust tests to port and in what order #5297.tests/support/host-serverTLS:tls(servers signed by the test's own CA) andtls_env_ca(servers signed by the environment's additional CA).tests/support/corporate_proxyto read it.gateway.config.additional_caand publishesOPENSHELL_TEST_ADDITIONAL_CA_{CERT,KEY}; the test signs leaf certificates for servers it starts).Files deleted
additional_ca.rs,config_push.rs,gateway_start.rs,otlp_relay.rs,podman_corporate_proxy.rs,podman_gateway_start.rs,policy_activation.rs,provider_readiness.rs,provider_refresh_handles.rs,provider_token_exchange.rs,stable_external_placeholders.rs,vm_gateway_start.rs.docker_corporate_proxy.rsis deleted by whichever of PR1 and PR6 lands last.Defaults (change in review if needed)
features/gateway-restart, not conformance: it checks that the container or VM survives, which needs engine or host access. Once every conformance row publishes the stop/start commands, the phase-only half can move to conformance.provider_readinesskeeps its wrong host/port/path denials (they prove the backend was never hit). Revisit after test(conformance): port the credential gating e2e tests #5318.Context
cli_child, SSH host identity), PR5 (container-engine helper) and test(conformance): port the credential gating e2e tests #5318.e2e-vm.shon the existing KVM lane, which publishes the stop/start commands for the gateway it starts.additional_cadrop waits on decision D in test(e2e): decide which e2e/rust tests to port and in what order #5297.Definition of Done
tests/supportcontract table.tasks/*.tomland nextest config is removed..github/workflows/e2e-suites.ymlon the drivers its legacy tests covered.PR7:
test(e2e): port OIDC authorization tests from e2e/rust to features/oidc-authUser Story
As a maintainer retiring
e2e/rust, I want the OIDC role and workspace authorization tests on testkit as a small set of table-driven tests, so they run on a Keycloak environment and the 43 near-identical legacy tests become 13.Description
All 43 tests come from
oidc_pkce.rsand move totests/suites/features/oidc-auth. The browser (PKCE) login runs once; every other test logs in with a password grant throughtests/support/keycloak. Role × action pairs become rows of table tests. Merges keep every assertion.Suggested new tests and the legacy tests each one covers:
pkce_browser_login_persists_sessionlogin_identity,assert_pkce_authorization_url,assert_persisted_login) as one testuser_reports_gateway_validated_identityuser_can_report_gateway_validated_identityplatform_admin_smokeadmin_can_list_sandboxes,admin_can_inspect_gateway,admin_can_list_providers,admin_can_manage_provider,admin_can_create_workspacenon_admins_denied_platform_actions(table)user_cannot_inspect_gateway,user_cannot_create_workspace,user_cannot_delete_workspace,workspace_admin_cannot_{create,delete}_workspace,workspace_admin_cannot_inspect_gatewayworkspace_user_read_accessworkspace_user_can_read_workspace,user_can_list_sandboxes,user_can_list_providers,workspace_admin_can_read_workspacesandbox_create_delete_by_role(table)admin_can_{create,delete}_sandbox,user_can_{create,delete}_sandboxworkspace_user_denied_management(table)user_cannot_{create,delete}_provider,workspace_user_cannot_manage_membersworkspace_admin_manages_providers_and_membersworkspace_admin_can_{create,delete}_provider,workspace_admin_can_{add,remove}_user_member,membership_removal_revokes_workspace_accessworkspace_admin_cannot_grant_admincross_workspace_access_denied(table)workspace_admin_cannot_read_another_workspace,workspace_admin_cannot_manage_another_workspace_members,workspace_user_cannot_read_another_users_workspace,workspace_user_cannot_list_another_workspace_{sandboxes,providers,members},workspace_user_cannot_create_sandbox_in_another_workspace(also checks the other workspace has no new sandbox)workspace_list_hides_other_workspacesworkspace_list_hides_another_users_workspace(parse JSON names; also checkdefaultis hidden)workspace_admin_cannot_manage_another_workspace_providersuser_can_resolve_provider_profile_for_sandboxDropped (each repeats a role × action pair another test asserts; needs approval in #5297):
admin_can_delete_workspace,second_workspace_user_cannot_read_first_users_workspace,workspace_admin_can_create_sandbox,workspace_admin_can_delete_sandbox.All names, providers and workspaces are unique per run and created in a test-owned workspace, not
default.Helpers and environment this PR adds
tests/support/keycloakgains password-grant sessions per user. Keycloak publishes the extra users asOPENSHELL_TEST_KEYCLOAK_USER_*andOPENSHELL_TEST_KEYCLOAK_USER_B_*.tests/support/oidcfor the PKCE flow, and a browser stand-in that records the authorization URL.Files deleted
oidc_pkce.rs.Context
create_provider) and PR4 (cli_child, used for the browser login).Definition of Done
tests/supportcontract table.oidc_pkce.rsis deleted, and every reference to it in wrapper scripts,tasks/*.tomland nextest config is removed.features/oidc-authhas rows in.github/workflows/e2e-suites.ymlon the drivers its legacy lane covered.PR8:
test(e2e): port Kubernetes driver tests from e2e/rust to drivers/kubernetesUser Story
As a maintainer retiring
e2e/rust, I want the Kubernetes-only tests (workspace namespaces, credential backends, HA, shared supervisor) on testkit, so they run against tmachine k3s environments through a Rust Kubernetes client instead ofkubectlin wrapper scripts.Description
Destinations are under
tests/suites/drivers/kubernetes/tests/unless noted. Merges keep every assertion.shared_supervisor/mod.rshelpers and their 7 tests (6 unit tests and 1 host-call test)tests/support/kube(shared_supervisormodule)kubernetes_runtime_reconcile::start_refreshes_stale_bootstrap_imagereconcile.rskubernetes_runtime_reconcile::old_and_new_gateways_can_restart_the_same_sandboxuser_namespaces::sandbox_pod_spec_has_user_namespace_fieldscredential_drivers::provider_credentials_are_stored_in_configured_backendcredential_backends.rsworkspace_namespace_managed::managed_full_lifecycle_with_multiple_sandboxesworkspace_managed.rsworkspace_namespace_managed::managed_namespace_survives_with_remaining_sandboxes,managed_workspace_delete_removes_namespaceworkspace_managed.rsworkspace_namespace_managed::creates namespace with labels, isolates workspaces, supervisor reads client TLS from its bootstrap secret, gateway cannot read workspace secrets, rejects namespace owned by another gateway, stop waits for pod to disappear (6 tests)workspace_managed.rsgateway_canhelper with the operator tests.workspace_namespace_managed::managed_rejects_invalid_dns1123_sandbox_nametests/suites/conformance/src/cli.rsworkspace_namespace_operator::all 6 testsworkspace_operator.rsCARGO_MANIFEST_DIR.kubernetes_ha_rebalancing::sandbox_exec_rebalances_across_gateway_scale_and_rolloutha.rskubernetes_ha_rebalancing::sandbox_file_sync_survives_gateway_pod_rollsha.rskubernetes_shared_supervisor::one_supervisor_isolates_and_outlives_concurrent_sandbox_instancesshared_supervisor.rskubernetes_shared_supervisor_ha::supervisor pod loss moves sandboxes; pool sessions follow their gateway (2 tests)shared_supervisor_ha.rstests/support/kube, notdocker exec.kubernetes_shared_supervisor_scaling::scaling_a_supervisor_pool_up_and_down_keeps_its_sandboxes_runningshared_supervisor_scaling.rskubernetes_corporate_proxy::kubernetes_corporate_proxy_uses_secret_and_never_falls_backtests/suites/features/corporate-proxy(k3s row)This PR also adds the Kubernetes half of the ephemeral cleanup "no driver resources left" test (PR5 adds Docker and Podman), and the Helm branch of
additional_cafrom PR6.Helpers and environment this PR adds
tests/support/kube: a Rustkubeclient (already a workspace dependency) built from published variables, namespace and sandbox-resource lookups,gateway_can(RBAC check), port-forwarding, and the shared-supervisor helpers. Tests never runkubectl.OPENSHELL_TEST_KUBE_{KUBECONFIG,CONTEXT,NAMESPACE,RELEASE,SANDBOX_NAMESPACE,CHART}.OPENSHELL_TEST_GATEWAY_REPLICAS(k3s-ha runs 3; the legacy tests hard-code 2).Files deleted
credential_drivers.rs,kubernetes_ha_rebalancing.rs,kubernetes_runtime_reconcile.rs,kubernetes_shared_supervisor.rs,kubernetes_shared_supervisor_ha.rs,kubernetes_shared_supervisor_scaling.rs,shared_supervisor/mod.rs,user_namespaces.rs,workspace_namespace_managed.rs,workspace_namespace_operator.rs,kubernetes_corporate_proxy.rs.Defaults (change in review if needed)
Context
Definition of Done
tests/supportcontract table.OPENSHELL_E2E_KUBE_TESTtasks, wrapper scripts, nextest groups) is removed.drivers/kuberneteshas k3s and k3s-ha rows in.github/workflows/e2e-suites.yml, with test(runner): apply per-family nextest config and a thread count to runner rows #5295 timeouts for the long tests.PR9:
test(e2e): port VM and GPU tests from e2e/rust to testkitUser Story
As a maintainer retiring
e2e/rust, I want the VM driver and GPU tests on testkit, running on the existing KVM and GPU lanes, so those lanes stop depending on the legacy harness and milestone 1 can finish.Description
vm_overlay::vm_overlaytests/suites/drivers/vmtls.keypermission checks; make thetls.keycheck unconditional or assert absence explicitly. Drop the redundant/sandboxwrite.vm_overlay::vm_identity_matches_status_and_rejects_conflictstests/suites/drivers/vmvm_bind_mount::vm_bind_mounttests/suites/drivers/vmrootfs_tar::sandbox_from_rootfs_tar+sandbox_from_gzipped_rootfs_tartests/suites/drivers/vm.tar.gz). Build the tarball by exportingtestkit::base_image()and appending a marker file, instead ofdocker buildfrom the Internet.gpu/device_selection::default device, each discovered device, all devices, invalid device (4 tests)tests/suites/features/gpu-cdigpu/device_selection::CDI anddocker infoparsing (5 unit tests)tests/support/gpucargo testwithout a GPU.gpu/workloads::gpu_workload_manifest_runs_expected_workloadstests/suites/features/gpu--gpu,--from), so it runs on any GPU row.gpu.rs(module wrapper)vm_corporate_proxy::vm_corporate_proxy_routes_approved_tls_egress+trusts_ca_bundle_for_https_proxytests/suites/features/corporate-proxy(VM row)This PR also adds:
Helpers and environment this PR adds
e2e-vm.sh(the existing KVM lane,e2e-vm-test.yml) runs the testkit suites against the gateway it starts, as it already does for conformance:drivers/vm(including the VM ephemeral-cleanup test), the VM row offeatures/corporate-proxy(the script sets up the proxy and publishes its variables), and the VM row of PR6'sfeatures/gateway-restart. This is the deliverable, not a stopgap.e2e-gpu-test.yaml, throughe2e:docker:gpu) runsfeatures/gpuandfeatures/gpu-cdiin place of thee2e/rustGPU tests.tests/support/gpu(device inventory), a rootfs tarball fixture, and a sandbox status-conditions helper.Files deleted
gpu.rs,gpu/device_selection.rs,gpu/workloads.rs,rootfs_tar.rs,vm_bind_mount.rs,vm_overlay.rs,vm_corporate_proxy.rs.Defaults (change in review if needed)
e2e-gpu-test.yaml) runs only Docker today (e2e:docker:gpuon linux-arm64, linux-amd64 and WSL).e2e:podman:gpuexists intasks/test.toml, but no workflow runs it. The GPU tests port as-is and run on the existing Docker GPU rows. File a follow-up issue for a Podman GPU row.Context
OPENSHELL_TEST_*variables, so the VM rows can move into.github/workflows/e2e-suites.ymlande2e-vm.shcan be deleted.Definition of Done
e2e-vm.shlist, GPU tasks, nextest groups) is removed.e2e-vm.shrunsdrivers/vmand the VM rows offeatures/corporate-proxyandfeatures/gateway-restarton the existing KVM lane.e2e-gpu-test.yaml) runsfeatures/gpuandfeatures/gpu-cdi.e2e/rusttest crate (tests, harness and its unit tests,Cargo.toml) and the wrapper-script steps that ran it. The wrapper scripts themselves stay.