Repository navigation
Conversation
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
retail-cartsuite and benchmark: one guest add-to-cart task on each of 16 retail and industrial-supply sites with heavy bot protection (Akamai, DataDome, HUMAN, Cloudflare, first-party). Each task starts on the site's home page, then search → product page → add to cart → cart, and asks the agent to report the cart line and subtotal. No login, no checkout.The agent drives the browser only through Kernel MCP's
browser_repl, and ends each task by calling afinalizetool on a small stdiobenchMCP server. The outcome enum separatesblocked(bot protection) fromwebsite_error,kernel_errorandfailed, so per-site pass rates and block rates come out of the same run. The verifier also records the browser's final URL and title so reported successes can be spot-checked.Harness changes
Ported from the closed #6, without the xwebmcp-specific parts:
toolset_argsandagent_kwargs; benchmarks takesuite_argsbench run --attempts(Harbor--n-attempts), also as a workflow input; the retail-cart workflow defaults to 10site-configno longer creates one shared proxy per recipe. It emits aproxy_createspec;KernelTaskEnvironmentcreates a uniquely named proxy for each trial before the browser and deletes it after, andbench runsweeps anybench-<run_id>-proxies a crashed trial left behind. Previously every trial with the same recipe shared one ISP proxy object, i.e. one static IP.ProxyCreationError; the report lists infrastructure failures by arm, exception type and task, and Braintrust metadata carrieserrorTypekernel-mcp-localtakestoolsets = ["repl"]to enablebrowser_replinstead ofexecute_playwright_code, and logs tool calls at the server (mcp-call-log) forkernel-mcp-verifyArms
codex-repl: Codexgpt-6-luna, stealth browsercodex-repl-site-config: same, with the config registry's per-site browser/proxy and guidance; the configurator usesopenai/gpt-6-luna, so the benchmark needs only an OpenAI keyTesting
ruff check,ruff format --check,pytest(17 passed)bench prepare retail-cart --arms codex-replbench runofcostco-microfiber-towelsoncodex-repl: finalizedsuccess, 25 REPL calls on the assigned browser, final page the cartsite-configwith an OpenRouter model on harborfreight and mouser, 2 attempts each: the two mouser trials ran on separate ISP proxies with different egress IPs, both deleted at teardown, none left after the run. Mobile proxy creation currently fails in the test project with an upstream 403, so the harborfreight trials errored before creating a browser