Skip to content

Add retail-cart benchmark: guest add-to-cart flows with the browser REPL - #8

Closed
rgarcia wants to merge 3 commits into
mainfrom
hypeship/retail-cart-benchmark
Closed

rgarcia wants to merge 3 commits into
mainfrom
hypeship/retail-cart-benchmark

Conversation

@rgarcia

@rgarcia rgarcia commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a retail-cart suite and benchmark: one guest add-to-cart task on each of 16 retail and industrial-supply sites with heavy bot protection (Akamai, DataDome, HUMAN, Cloudflare, first-party). Each task starts on the site's home page, then search → product page → add to cart → cart, and asks the agent to report the cart line and subtotal. No login, no checkout.

The agent drives the browser only through Kernel MCP's browser_repl, and ends each task by calling a finalize tool on a small stdio bench MCP server. The outcome enum separates blocked (bot protection) from website_error, kernel_error and failed, so per-site pass rates and block rates come out of the same run. The verifier also records the browser's final URL and title so reported successes can be spot-checked.

Harness changes

Ported from the closed #6, without the xwebmcp-specific parts:

  • arms take toolset_args and agent_kwargs; benchmarks take suite_args
  • bench run --attempts (Harbor --n-attempts), also as a workflow input; the retail-cart workflow defaults to 10
  • site-config no longer creates one shared proxy per recipe. It emits a proxy_create spec; KernelTaskEnvironment creates a uniquely named proxy for each trial before the browser and deletes it after, and bench run sweeps any bench-<run_id>- proxies a crashed trial left behind. Previously every trial with the same recipe shared one ISP proxy object, i.e. one static IP.
  • A trial whose proxy cannot be created fails with ProxyCreationError; the report lists infrastructure failures by arm, exception type and task, and Braintrust metadata carries errorType
  • kernel-mcp-local takes toolsets = ["repl"] to enable browser_repl instead of execute_playwright_code, and logs tool calls at the server (mcp-call-log) for kernel-mcp-verify

Arms

  • codex-repl: Codex gpt-6-luna, stealth browser
  • codex-repl-site-config: same, with the config registry's per-site browser/proxy and guidance; the configurator uses openai/gpt-6-luna, so the benchmark needs only an OpenAI key

Testing

  • ruff check, ruff format --check, pytest (17 passed)
  • bench prepare retail-cart --arms codex-repl
  • local bench run of costco-microfiber-towels on codex-repl: finalized success, 25 REPL calls on the assigned browser, final page the cart
  • site-config with an OpenRouter model on harborfreight and mouser, 2 attempts each: the two mouser trials ran on separate ISP proxies with different egress IPs, both deleted at teardown, none left after the run. Mobile proxy creation currently fails in the test project with an upstream 403, so the harborfreight trials errored before creating a browser

@rgarcia

rgarcia commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Split into #9 (harness changes) and #10 (retail-cart benchmark, stacked on #9).

@rgarcia rgarcia closed this Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant