Measured on the 1 September matrix, verification-001-stripe-express. The workspace .env the agent is given contains the Stripe signing secret it needs. Of six runs, exactly one used it:
| arm |
secret configured on the source |
outcome |
| claude-code-sonnet-5 |
whsec_51KzQmTestSecretFo… — the one in .env |
pass |
| codex-gpt-5.6-no-skills |
not echoed in tool calls |
pass |
| claude-code-sonnet-5-no-skills |
whsec_new, whsec_xxx |
fail |
| codex-gpt-5.4-mini-no-skills |
whsec_xxx |
fail |
| codex-gpt-5.6 |
whsec_test_secret, whsec_replace_me |
fail |
| codex-gpt-5.4-mini |
five different placeholders |
fail |
All four failures reported the task complete.
Why this is a product finding and not only an agent one
A source configured with whsec_xxx is accepted, saved and enabled. Nothing rejects it, nothing warns, and the configuration reads as correct in the API and the dashboard. The first evidence that anything is wrong is genuine provider traffic failing verification — in production, at whatever hour the provider first sends something.
A human copying from a tutorial makes the same mistake; the agent just makes it faster and more confidently. Two things would surface it:
- Reject or warn on an obviously non-credential value at source-verification config time.
whsec_xxx, whsec_replace_me, your_secret_here and friends are a short list and this is the moment the person is looking at the screen.
- Say it in the verification response. A source that has never successfully verified a request is a state we can detect and report, rather than leaving it to be discovered by absence.
Neither is this repository's to fix, which is why this is filed as a finding.
Why it is worth carrying
It is the most discriminating single check in the suite right now — four of six arms, three of them on the identical check — and it is a failure mode a benchmark is uniquely placed to notice, because it needs a real credential, a real source and real signed traffic to show up at all.
Measured on the 1 September matrix,
verification-001-stripe-express. The workspace.envthe agent is given contains the Stripe signing secret it needs. Of six runs, exactly one used it:whsec_51KzQmTestSecretFo…— the one in.envwhsec_new,whsec_xxxwhsec_xxxwhsec_test_secret,whsec_replace_meAll four failures reported the task complete.
Why this is a product finding and not only an agent one
A source configured with
whsec_xxxis accepted, saved and enabled. Nothing rejects it, nothing warns, and the configuration reads as correct in the API and the dashboard. The first evidence that anything is wrong is genuine provider traffic failing verification — in production, at whatever hour the provider first sends something.A human copying from a tutorial makes the same mistake; the agent just makes it faster and more confidently. Two things would surface it:
whsec_xxx,whsec_replace_me,your_secret_hereand friends are a short list and this is the moment the person is looking at the screen.Neither is this repository's to fix, which is why this is filed as a finding.
Why it is worth carrying
It is the most discriminating single check in the suite right now — four of six arms, three of them on the identical check — and it is a failure mode a benchmark is uniquely placed to notice, because it needs a real credential, a real source and real signed traffic to show up at all.