Update BrowserCode model comparison plot - #23
Merged
Conversation
Resyncs official_plots/browser_harness_by_model_{light,dark}.png from
benchmark-x-laminar@45b563b. The committed copies were generated 2026-06-17
and are the stalest of any published copy -- they still show glm-5.1 (79.7%)
as the top open-weight model and predate claude-opus-5, claude-fable-5,
grok-4.5, kimi-k3, the gpt-5.6 family, qwen3.8-max and
deepseek-v4-flash-0731.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resyncs
official_plots/browser_harness_by_model_{light,dark}.png(the"Comparing Models for BrowserCode" plot) from
benchmark-x-laminar@45b563b.
The committed copies were generated 2026-06-17 and are the stalest published
copy of this plot anywhere. They still show
glm-5.1(79.7%) as the topopen-weight model and predate every model added since:
claude-opus-5(87.0%),claude-fable-5(87.0%),grok-4.5(86.3%),kimi-k3(86.0%),gpt-5.6-sol(84.0%),gpt-5.6-lunaxhigh (82.0%) / default (72.4%),gpt-5.6-terra(72.0%),qwen3.8-max(79.0%),minimax-m3(74.0%),deepseek-v4-flash-0731(76.0%)mimo-v2.5/-pro,glm-4.7,glm-5.1,kimi-k2.6,qwen3.6-plus,grok-4.3,gpt-5.4-mini,claude-sonnet-4-6, basedeepseek-v4-flashImage-only change.
generate_plots.pyin this repo producesaccuracy_by_modeland
accuracy_vs_throughput; the three BU Bench plots in the README areimported copies from benchmark-x-laminar, so nothing here regenerates them.
Companion PR for the same plot in the BrowserCode README: browser-use/browsercode#143
Not in this PR
browser_use_framework_by_model_{light,dark}.pnghas also drifted fromupstream, which now has two extra bars (
qwen3.627b29.0%,glm-5.125.0%);the top eight are unchanged. Left out because it is a browser-use-framework
plot and unaffected by the BrowserCode results this PR propagates -- say the
word and I will fold it in.
best_of_frameworks_public_dark.pngdiffers from upstream only in PNGmetadata; the rendered content is byte-identical to the light version's
content. Not worth a churn commit.
Summary by cubic
Refreshes the BrowserCode model comparison plots
official_plots/browser_harness_by_model_{light,dark}.pngfrombenchmark-x-laminar@45b563bto replace the stale 2026-06-17 images. The plots now include new models (e.g.,claude-opus-5,claude-fable-5,grok-4.5,kimi-k3,gpt-5.6-*,qwen3.8-max,deepseek-v4-flash-0731), remove dropped ones, and change open-weight bars to a gray fill with a blue dashed outline.Written for commit 13bac02. Summary will update on new commits.