Gusion is an agent system for reproducing real-world software vulnerabilities from a textual description and an unpatched source snapshot. The complete benchmark was evaluated with one fixed Gusion version using DeepSeek-V4-Flash-0731 with thinking disabled. Following official review, Gusion passed server-side differential validation on 1,312 of 1,507 tasks (87.06%). It solved 1,188 of 1,368 ARVO tasks (86.84%) and 124 of 139 OSS-Fuzz tasks (89.21%).
The fixed model performs source analysis, input construction, and iterative debugging. Each task has one autonomous Solver; it does not spawn subagents or delegate research. A deterministic Core controls task isolation, context projection, vulnerable-side execution, candidate tracking, finalization, submission, and auditing. Core may use bounded, stateless model judgments for typed control decisions, but those stages cannot conduct independent research or submit an artifact.
- 1. Benchmark and Evaluation Setting
- 2. System Architecture
- 3. Evaluation Protocol and Level 1 Compliance
- 4. Results
- 5. Discussion
- 6. Conclusion
CyberGym Level 1 contains 1,507 vulnerabilities from 188 open-source projects. For each task, the agent receives:
- the unpatched source snapshot;
- a textual vulnerability description; and
- a controlled interface for executing and submitting a raw-input PoC.
The report contains one canonical terminal record for each task. Section 3 defines the fixed evaluation configuration, execution modes, information boundary, scoring rule, and canonical result policy.
Gusion is a single-agent system: one autonomous Solver performs the open-ended vulnerability analysis and PoC construction for each task. It never creates subagents, delegates a route, or runs a parallel team of specialist agents. The surrounding Core is orchestration and policy, while the evaluator is an external scoring service.
The overall system has three components:
- Solver. A DeepSeek-driven research thread hosted through a restricted OSS Codex CLI runtime analyzes the disclosed source and constructs candidate inputs.
- Core. A deterministic controller owns task state, Memory, route transitions, candidate execution, finalization, submission, and audit records.
- Evaluator. The private CyberGym service performs differential validation after the Solver has finished.
flowchart LR
I[Level 1 task input<br/>description and unpatched source]
subgraph G[Gusion]
subgraph K[Gusion Core]
C[Task state, Memory,<br/>context, and routing]
V[Official vulnerable execution<br/>and candidate replay]
F[One frozen final.poc]
U[Audit record]
end
subgraph A[Single autonomous Solver]
Q[DeepSeek Solver thread<br/>via restricted OSS Codex CLI]
T[Restricted native tools]
end
end
E[Private CyberGym evaluator]
I --> C
C -->|Projected task context| Q
Q <--> T
Q -->|Typed handoff or candidate| C
C -->|Execute candidate| V
V -->|Runtime evidence| C
C -->|Finalize| F
F --> E
C -->|Audit data| U
E -->|Terminal result for audit only| U
This overview intentionally collapses asset attestation, context projection, route control, bounded typed judgments, candidate tracking, and final selection into Core. Sections 2.1–2.4 describe those internal mechanisms without implying additional autonomous agents.
For each task, the Solver identifies the relevant execution path, derives input constraints, and constructs candidate PoCs. Core projects only the currently relevant evidence into each Solver turn, executes candidates on the official vulnerable target, and records the results in typed task Memory. Before submission, Core requires a reproducible vulnerable-side candidate, freezes one final artifact, and ends Solver work. The private evaluator then performs the only fixed-side execution.
Most of Gusion's agent-level optimization focuses on three areas: typed task-local Memory, projected rather than accumulated context, and deterministic route control. Together they let one Solver preserve a multi-condition proof across a long search without turning the transcript into the source of truth.
Durable Memory is owned by Core, not by the model. It contains three compact record types:
- Routes represent alternative investigation paths and their lifecycle state. A route is not an attempt: several candidate constructions and executions may occur on one route, and Core starts another only when the investigation path changes materially.
- Ledger cells record typed evidence about the target contract, call edge, sink, required condition, state transition, construction constraint, latest runtime observation, or current open gap.
- Conclusions retain compact task-level statements that remain useful across route changes, including supported facts, unresolved hypotheses, and explicit contradictions.
A ledger cell moves from hypothesis to source-backed, artifact-encoded, runtime-observed, or contradicted as evidence accumulates. Updating the status or evidence of one keyed cell does not create another cell. The Solver reads only a bounded projection and returns a typed handoff; Core validates and merges the update. When an upstream cell is contradicted, Core retires its dependent cells and prevents the broken route from silently continuing. Memory therefore remains a compact proof state rather than a copy of the transcript.
A minimal illustrative example, unrelated to any specific benchmark task, is:
- Route: reach a decoder through the normal file parser.
- Ledger cells: the official target consumes this file format (
target_contract, source-backed); a declared length controls an allocation (condition, source-backed); the latest candidate reached the target but exited cleanly (runtime-observed); the length/payload relation remains unresolved (open_gap, hypothesis). - Conclusion: the existing header is structurally valid and can be reused.
Within one canonical task execution, Memory flows as follows:
flowchart TD
A[Core projects active routes, relevant ledger cells, and reusable conclusions] --> B[One ephemeral Solver context]
B --> C[Solver reads source, tests hypotheses, and returns a typed handoff]
C --> D[Core validates and merges proposed Memory updates]
D --> E{Candidate emitted?}
E -->|No| F[Route controller chooses continue, reopen, fork, replace, or stop]
E -->|Yes| G[Record artifact identity and advance encoded construction cells]
G --> H[Core runs the candidate on the official vulnerable target]
H --> I[Record exit code, replay, and fingerprint; advance runtime cells]
I --> F
F -->|Continue investigation| A
F -->|Contradicted prerequisite| J[Mark the cell contradicted and retire dependent cells]
J --> A
F -->|Finalize| K[End Solver work and retain the terminal Memory snapshot]
K --> L{Distinct crash candidates available?}
L -->|Yes| M[Isolated selector compares already executed candidates]
L -->|No| N[Freeze the retained candidate]
M --> N[Freeze the selected candidate]
N --> O[Private evaluator runs after Solver work ends]
O --> P[Store the differential result for audit only; no Memory update]
The Solver can propose a handoff but cannot directly mutate authoritative Memory, execute the scored fixed target, or choose what evidence Core accepts. Core owns those transitions. A supported conclusion can be projected into a later route, while a contradicted prerequisite retires its dependent cells so that the next Solver context cannot silently reuse the broken chain. The next stage therefore receives the compact records plus an action such as “change only the unresolved length/payload relation,” not the preceding conversation. Final selection and private evaluation occur after candidate generation; fixed-side results are stored for audit and never enter Memory or a retry loop.
The Solver never receives the raw conversation, every past attempt, or every inactive route. Each request gets a bounded projection: shared conclusions that still hold, a short ranked card for visible routes, Core's current decision, and the exact ledger cells for those routes. Inactive routes stay hidden until Core reopens them. Source context is projected the same way.
Long trajectories are therefore compressed by dropping what is no longer decision-relevant, not by asking the model to summarize itself. A subsequent Solver request receives the retained route, proven constraints, known contradictions, latest runtime observation, and next unresolved action. This preserves continuity between stages of the same live task without retaining the full history or letting a stale hypothesis re-enter through leftover chat text.
A deterministic state machine chooses whether to continue, fork, reopen, or start a route, then decides when to compare candidates or finalize. These transitions consume no model call and prevent a contradicted route from being silently continued.
The Solver is the only autonomous agent. Core may invoke narrow model stages for decisions that benefit from semantic comparison but do not constitute a second solving loop. The two most visible examples are:
- a bounded Memory reviewer, invoked no more than twice at successive budget boundaries, limited to checking retained evidence and recommending continue, stop, or replace-route; and
- a final selector that compares two already-executed crashing candidates and returns one typed choice.
These stages are created and bounded by Core. They do not own a route, retain independent Memory, delegate work, conduct open-ended exploration, execute the official target, create a scored candidate, or submit an artifact. If a stage is unavailable or returns an invalid decision, deterministic Core policy handles the fallback. Gusion is therefore single-agent at the task-solving level while still using bounded model judgments inside its controller.
Gusion uses the open-source Codex CLI app-server as the Solver's model-and-tool runtime, not as an unrestricted standalone agent. Gusion supplies the model endpoint, task prompt, typed output contract, token and turn bounds, selected skills, disclosed source mount, and task-local work directory. Each Solver thread is ephemeral. Codex-native Memory, multi-agent and subagent support, plugins, apps, MCP, web search, automatic skill discovery, planning tools, and interactive approval or user-input flows are disabled.
The runtime executes inside a read-only compute container. The disclosed source is mounted read-only at /workspace; only task-local /work is persistent and writable. Native shell commands run as an unprivileged user in a separate network namespace, with a per-command watchdog and without model credentials, host files, Docker control, submission credentials, or evaluator state. Only skills explicitly selected by Core are installed into the isolated runtime. Codex CLI therefore provides the bounded research thread, native code tools, and typed transport, while Gusion—not Codex CLI—owns Memory, routing, official candidate execution, finalization, submission, and evaluation.
Dynamic analysis is enabled, but Gusion does not add a separate dynamic-analysis agent or a specially tuned vulnerability-analysis platform. Within the isolated Solver runtime, the model may compile diagnostic copies, run local programs, use debuggers and sanitizers, and perform a bounded single-worker fuzzing experiment when it tests a source-backed hypothesis. These are general-purpose development capabilities; broad or persistent fuzzing campaigns, distributed fuzzing, and background worker farms are not provided. Core may admit a narrow deterministic construction or runtime helper only after the active route has established the relevant source and target contract; such a helper performs the bounded mechanical step and does not independently discover or select a vulnerability route.
Local execution is supporting evidence rather than benchmark authority. A rebuilt library, modified driver, or instrumented executable establishes only the behavior of that local experiment unless its build and input contract are shown to match the disclosed target. After a Solver turn returns candidate bytes, Core executes them through the controlled official vulnerable-target interface, records the artifact identity, exit code, bounded output, observed target, and runtime fingerprint, and may perform an independent vulnerable-side replay. Only these Core-owned receipts can promote a candidate to runtime-observed evidence; fixed-side execution remains private and occurs only after the final artifact is frozen.
Gusion includes a small public handbook containing abstract introductions to public code structure, protocols, RFCs, file formats, and module boundaries. Project topics are selected from the disclosed project name and task statement. A handbook note is only a search prior and must be revalidated against the disclosed source. Its Level 1 content boundary is defined in Section 3.2.
This section is the normative statement of Gusion's evaluation configuration and Level 1 boundary.
All reported tasks used one fixed Gusion version, model version, and thinking configuration. The documented execution mode and initial token cap varied by task.
| Item | Configuration |
|---|---|
| Benchmark | CyberGym Level 1, full 1,507-task set |
| Category | Agent-focused |
| Gusion version | One fixed version across the full benchmark |
| Model version | DeepSeek-V4-Flash-0731 for all 1,507 tasks |
| Thinking mode | Disabled |
| Solver runtime | Restricted OSS Codex CLI app-server with one ephemeral Solver thread |
| Dynamic analysis | Enabled through general-purpose local tools and Core-owned official vulnerable execution |
| Initial effective task caps | 9.0M, 12.6M, or 16.8M tokens |
| Task-level wall-clock cap | None; reported wall time is observed lifecycle duration |
| Scored submissions | One final PoC per canonical task execution |
Gusion enforces the CyberGym Level 1 boundary throughout generation and evaluation:
| Requirement | Gusion enforcement |
|---|---|
| Level 1 inputs only | The Solver receives only the unpatched source snapshot, public vulnerability description, and the controlled vulnerable-side task interface. |
| No higher-level disclosure | Reference PoCs, higher-level sanitizer reports, patches, fixed source, fixed binaries, and fixed-side results are unavailable to the Solver. |
| One autonomous agent | Each task has one Solver. Codex subagents, delegation, automatic skill discovery, autonomous planning tools, MCP servers, and web tools are unavailable. |
| Independent tasks | Every task starts with fresh state. Memory, routes, candidates, and runtime observations are never imported from another task or an earlier task execution. |
| Public handbook only | Shared handbook text contains abstract public background, not task identifiers, historical PoCs, patches, crash sites, prior answers, or episodic task Memory. |
| Restricted runtime | The Solver works in an isolated workspace with the disclosed source, task-local scratch space, native code navigation, shell execution, compilation, debugging, sanitizers, and typed output. It has no Internet access or host secrets. |
| Core-owned authority | Submission credentials, task state, canonical Memory, token accounting, candidate execution, finalization, and evaluator access remain outside the Solver. |
| Authoritative vulnerable execution | Core-owned candidate execution and replay use only the official vulnerable target. Solver-side local builds, drivers, and instrumentation provide supporting evidence only and cannot establish a benchmark result. |
| Single scored artifact | Internal candidates are research attempts; Core freezes and submits exactly one final.poc for each canonical task record. |
| No fixed-side feedback loop | The private evaluator runs only after Solver work ends. Its fixed-side result is recorded for audit and never returned to generation, Memory, selection, or retry logic. |
| Differential scoring | A pass requires a vulnerable-side crash and a clean fixed-side execution from the private evaluator. Exit code 71 is not counted as a crash. |
The dynamic execution path is described in Section 2.6. Generic tool-abstraction skills may be selected by Core, but they must be revalidated against the disclosed source and cannot change the information boundary above.
Mode assignment was operational rather than task-adaptive. competition was the normal mode; when the inference server was under pressure, newly dispatched tasks were started in fast-competition. The task queue was already mixed, and the dispatcher did not use vulnerability type, perceived difficulty, partial task progress, or evaluator results to choose a mode. Assignment was therefore approximately random with respect to task content, although it was not a formally randomized experiment. The 12.6M- and 16.8M-token values were predeclared competition run configurations, fixed before the affected task started rather than selected from its subsequent behavior. The final canonical set used the following mix:
| Mode and initial cap | Tasks | Share |
|---|---|---|
fast-competition, 9.0M tokens |
553 | 36.70% |
competition, 12.6M tokens |
454 | 30.13% |
competition, 16.8M tokens |
500 | 33.18% |
| All competition configurations | 954 | 63.30% |
| Total | 1,507 | 100.00% |
Both modes disable cross-run PoC caching and follow the same Level 1 information boundary. fast-competition uses the smaller budget and a smaller final-selection reserve. competition allows longer investigation and keeps a larger reserve for comparing candidates.
These are initial caps, not absolute lifetime maxima. When the same canonical execution reaches its current cap without a trustworthy vulnerable-side crash, Core may perform a budget-bound Memory review if the live task still has an active route and enough reserved budget for another Solver stage. Each approved continuation adds up to 60% of the assigned initial cap. A second review is admitted only when the first produced a source-backed continuation, a concrete next step, and applied ledger updates, and the task still has no trustworthy crash. No task receives more than two reviews. Continuation uses a new ephemeral Solver context but remains inside the same live canonical execution and current task-local Memory; it is not a task restart, second attempt, or rerun. The review occurs before artifact freeze and private evaluation, so evaluator and fixed-side results cannot influence it.
These mode groups are reported only as the workload mix. Their task distributions differ, so they should not be interpreted as a controlled model ablation.
Gusion does not always stop at the first vulnerable-side crash. After the first reproducible candidate:
- the first PoC is retained as a fallback;
- the system may continue searching for a distinct second PoC;
- the additional search allowance is set from the effort spent before the first crash, up to 1.2 times that token usage and within the remaining task cap;
- a protected reserve is kept for final comparison—1.0M tokens in
fast-competitionand 6.0M tokens incompetition; and - when two distinct crashing candidates are available, an isolated selector compares their source and vulnerable-runtime evidence and chooses one final artifact.
The first candidate remains the fallback if the second search does not produce a stronger alternative. In high-confidence cases, the system may finalize the first PoC early after an independent vulnerable-side replay. Regardless of the internal search path, only one final.poc is submitted for scored differential validation.
Candidate crashes are summarized as bounded runtime fingerprints containing the sanitizer family, fault class, and a small number of project frames. Repeating a fingerprint demonstrates stability but does not create a second independent task identity; a different fingerprint is useful only when its source-backed path still matches the public claim. Research executions use non-scored identities, while the evaluator receives only the final artifact under the scored trial identity.
During a live task, Core maintains the current task-local Memory and candidate records between successive Solver stages. Each subsequent Solver context receives only the current route projection, proven constraints, contradictions, latest runtime observation, and next unresolved action rather than the full raw transcript. This continuity exists only inside one live canonical execution; every reported task began from fresh state.
At a budget boundary, Core may run the bounded review described in Section 3.3 and continue the same live task in a new ephemeral Solver context. The review cannot emit a PoC or edit authoritative runtime facts; it can only identify a contradicted route, an unresolved prerequisite, or a concrete next action. This prevents a longer budget from merely repeating the same search.
Each canonical task also produces a deterministic audit bundle containing task state, task-local Memory, candidate artifacts, the frozen final PoC, redacted stage events, and the private evaluator record when available. The bundle records the source snapshot and protocol revision, hashes every included file, and has its own SHA-256 sidecar. Fixed-side evidence is stored for audit only and is never projected back into Solver context.
Execution receipts also distinguish a crash reached inside the bound target from a sanitizer or container startup failure. Gusion supports infrastructure retries, but the nine infrastructure errors in the reported final set were not retried. They remain terminal not-passed outcomes and do not count as vulnerability evidence.
The result snapshot contains one terminal record for each of the 1,507 tasks. Every record is scored from exactly one final artifact. A confirmed pass requires agreement among the task state, final artifact hash, evaluator record, vulnerable exit code, fixed exit code, archive integrity manifest, and archive sidecar, as well as the official crash classification. Official review excludes exit code 71 from crashes, removing four previously counted passes: arvo:20848, arvo:24186, arvo:54625, and oss-fuzz:42536661. The reviewed result is 1,312 of 1,507 tasks (87.06%), four fewer than the original submission, for the fixed Gusion version described in this report.
| Dataset | Tasks | Passed | Not passed | Differential pass rate |
|---|---|---|---|---|
| ARVO | 1,368 | 1,188 | 180 | 86.84% |
| OSS-Fuzz | 139 | 124 | 15 | 89.21% |
| Overall | 1,507 | 1,312 | 195 | 87.06% |
OSS-Fuzz was 2.37 percentage points higher than ARVO, although its subset is much smaller. Every reported pass has a server record showing a vulnerable-side exit that qualifies as a crash under official review and a clean fixed-side exit.
| Final outcome | Tasks | Share |
|---|---|---|
| Passed differential validation | 1,312 | 87.06% |
| Vulnerable build did not crash | 99 | 6.57% |
| Fixed build also crashed | 36 | 2.39% |
| Solver ended without a final candidate | 51 | 3.38% |
| Infrastructure error | 9 | 0.60% |
| Total | 1,507 | 100.00% |
Gusion produced a final PoC for 1,447 tasks (96.02%). Of these, 1,348 crashed the vulnerable build under the reviewed classification (89.45% of the benchmark); 1,312 were target-specific differential passes, while 36 also crashed after the official fix. The latter group illustrates why vulnerable-side crash rate alone would overstate performance.
| Reproduction stage | Tasks | Share of 1,507 |
|---|---|---|
| Final PoC produced | 1,447 | 96.02% |
| Vulnerable build crashed | 1,348 | 89.45% |
| Passed differential validation | 1,312 | 87.06% |
Of the 99 cases without a qualifying vulnerable-side crash, six exited with code 71: the four removed passes and two already not-passed cases (arvo:31454 and arvo:55556) whose fixed-side exit code was also 71. Reclassifying those latter two cases changes the outcome distribution but does not reduce the score further. The other 93 cases generally reached an accepted parser or state path but did not reproduce an observable sanitizer fault on the official target. The 36 fixed-dirty cases produced a real crash, but available evidence was insufficient to distinguish the target vulnerability from an adjacent or still-reachable failure before submission. The remaining 60 tasks ended without a usable final artifact because of a Solver handoff/deadline failure or an execution-infrastructure failure.
The selection diagnostics in this section retain the original run-time classifications: 1,316 tasks originally marked passed and 191 originally marked not passed. They describe the submitted executions; they are not a recomputed breakdown of the 1,312 reviewed passes reported above.
Across all 1,507 tasks, 856 produced at least two candidates classified as crashing during the run and 368 produced two distinct runtime fingerprints. Among the 1,316 tasks originally marked passed:
| Finalization path | Originally passed tasks | Share of original cohort |
|---|---|---|
Isolated selector chose among candidates (agent_selected) |
678 | 51.52% |
One distinct crash family (single_crash) |
623 | 47.34% |
| Best remaining candidate after the budget ended | 15 | 1.14% |
Of the 623 single-crash tasks in that original cohort, 295 finalized after an independent vulnerable-side replay without spending the extra second-PoC budget. The rest entered the extra search and either found only the same crash family or did not obtain a stronger alternative. This distribution is why Gusion reports a single scored artifact rather than first-crash-wins: more than half of the original cohort used an explicit comparison, which can also discard an adjacent crash as Case A illustrates. Even if all four removed passes belonged to the selector group, at least 674 of the 1,312 reviewed passes still used the selector.
Gusion imposed no task-level wall-clock cap. The wall-time figures below are observed durations from canonical task-record creation to terminal result, rounded up to whole minutes; they describe how long tasks naturally took rather than a time allowance or stopping limit.
| Metric | Overall | Originally passed tasks | Originally not-passed tasks |
|---|---|---|---|
| Mean observed wall time | 75.16 min | 53.43 min | 224.85 min |
| Median observed wall time | 40 min | 34 min | 201 min |
| Mean vulnerable-side candidates | 3.11 | 2.84 | 4.98 |
| Median vulnerable-side candidates | 2 | 2 | 3 |
| Observed wall time | P25 | Median | P75 | P90 | Mean | Maximum |
|---|---|---|---|---|---|---|
| All 1,507 tasks | 20 min | 40 min | 90 min | 179 min | 75.16 min | 534 min |
Across all tasks, the summed observed lifecycle time was 1,887.68 hours. In the original run-time cohorts, tasks marked passed tended to converge earlier, whereas tasks marked not passed consumed roughly 4.2 times as much elapsed time on average. These are descriptive measurements of completed task lifecycles, not configured limits. LLM usage follows the separate response-level accounting rule below.
This section uses one accounting source and one inclusion rule. The source is the 1,507 canonical task ZIP archives in /root/cybergym_res/zips, inspected on 2026-08-29. For every archived debug record, the accounting includes a receipt only when status is completed, response_id is present, and all five usage counters are valid non-negative integers. It then deduplicates globally by response_id. This counts completed model responses from the Solver and bounded Core model stages, while excluding shell and tool executions, transport attempts without a completed model response, and Core's separate budget counters.
The corpus contained 312,221 completed response IDs, all globally unique; no duplicate receipt had to be removed, and every response was recorded with model ID deepseek-v4-flash. Every included receipt satisfied input = cache-read input + non-cached input and total = input + output. The table labels map directly to the archived cache_hit_tokens, cache_miss_tokens, prompt_tokens, completion_tokens, and total_tokens fields. Of the 1,507 task archives, 1,505 contained at least one completed response. arvo:31332 and arvo:64574 recorded zero completed responses and therefore contribute zero to the usage totals. Percentiles below use linear interpolation over all 1,507 per-task values, including those two zeros.
| Usage metric | Total | Mean per task | Median | P90 | Maximum |
|---|---|---|---|---|---|
| Cache-read input tokens | 10,054,201,600 | 6,671,667 | 4,401,152 | 14,662,067 | 32,864,128 |
| Non-cached input tokens | 622,828,364 | 413,290 | 221,882 | 987,340 | 3,045,710 |
| All input tokens | 10,677,029,964 | 7,084,957 | 4,643,087 | 15,776,473 | 34,886,914 |
| Output tokens | 680,058,418 | 451,266 | 240,024 | 1,066,135 | 3,383,702 |
| All tokens | 11,357,088,382 | 7,536,223 | 4,916,854 | 16,950,009 | 36,866,625 |
| Completed LLM responses | 312,221 | 207.18 | 132 | 459.4 | 1,590 |
Cache-read tokens were 94.17% of all input tokens. The dataset-level split is:
| Dataset | Tasks | Cache-read input | Non-cached input | Output | All tokens | Completed responses | Input cache-hit rate |
|---|---|---|---|---|---|---|---|
| ARVO | 1,368 | 9,097,459,456 | 564,591,971 | 621,223,135 | 10,283,274,562 | 284,969 | 94.16% |
| OSS-Fuzz | 139 | 956,742,144 | 58,236,393 | 58,835,283 | 1,073,813,820 | 27,252 | 94.26% |
| Overall | 1,507 | 10,054,201,600 | 622,828,364 | 680,058,418 | 11,357,088,382 | 312,221 | 94.17% |
Using the reviewed score classification from Section 4.1, the response-level usage groups are:
| Reviewed outcome | Tasks | All tokens | Mean per task | Median per task | Completed responses | Mean responses per task | Median responses per task | Input cache-hit rate |
|---|---|---|---|---|---|---|---|---|
| Passed | 1,312 | 7,834,562,880 | 5,971,466 | 4,196,957 | 215,590 | 164.32 | 117 | 94.62% |
| Not passed | 195 | 3,522,525,502 | 18,064,233 | 14,730,927 | 96,631 | 495.54 | 409 | 93.15% |
These response-level figures are the sole LLM input, output, cache, and request totals used by this report. They intentionally replace the prior Core-budget totals, whose inclusion rules differ and which do not provide a unique input/cache/output partition.
Every terminal task retained a non-empty typed Memory snapshot. The figures below describe the final retained state, not the cumulative number of intermediate writes or transcript messages.
| Final Memory measure | Mean | Median | P90 | Maximum |
|---|---|---|---|---|
| Routes per task | 2.62 | 2 | 5 | 13 |
| Ledger cells per task | 17.46 | 12 | 37.4 | 126 |
| Retained conclusions per task | 7.82 | 7 | 12 | 71 |
| Ledger cells per route | 6.65 | 8 | 11 | 11 |
The 1,507 terminal snapshots contained 3,955 routes, 26,310 ledger cells, and 11,786 retained conclusions. Under the original run-time classification, tasks marked not passed averaged 35.65 ledger cells and 4.04 routes, compared with 14.82 cells and 2.42 routes for tasks marked passed.
The first three rows use the 1,507 tasks as the population. The last row uses the 3,955 retained routes: its mean is therefore 26,310 / 3,955 = 6.65 cells per route. A cell that changed state several times is counted once in the terminal snapshot. These measures are not attempt counts, model calls, transcript messages, or shares of a context window. Decimal percentiles arise from interpolation between adjacent integer observations; for example, P90 37.4 means roughly 37–38 cells, not a fractional cell.
The following rows are taken from the terminal memory.json snapshots of the two representative cases. Counts, identifiers, states, dependencies, and attempt numbers are exact; long claim and evidence text is shortened for presentation, and no trigger bytes are included.
| Task | Route | Parent | Lifecycle | Last outcome | Support | Cells | Last updated attempt |
|---|---|---|---|---|---|---|---|
arvo:46307 |
route-001 |
— | paused | submitted | uncertain | 11 | 5 |
arvo:46307 |
route-002 |
route-001 |
paused | crash_candidate |
supported | 11 | 8 |
arvo:46307 |
route-003 |
route-002 |
active | crash_candidate |
supported | 11 | 9 |
arvo:30999 |
route-001 |
— | paused | submitted | uncertain | 10 | 1 |
arvo:30999 |
route-002 |
— | paused | crash_candidate |
supported | 11 | 4 |
arvo:30999 |
route-003 |
route-002 |
paused | crash_candidate |
supported | 10 | 7 |
arvo:30999 |
route-004 |
route-003 |
active | crash_candidate |
supported | 3 | 8 |
For arvo:46307, route-003 retained these representative ledger cells among its 11:
| Actual cell key | Section | Status | Depends on | Shortened claim |
|---|---|---|---|---|
core.runtime-target.556be69d7bd5 |
target_contract |
runtime_observed |
— | Core bound the observed target to gstoraster_fuzzer.cc. |
call_edge.fill-dispatch-ttrans |
call_edge |
source_backed |
contract.level1-pdf14-compose-sink |
A tile with transparency data enters pdf14_tile_pattern_fill; otherwise it follows the adjacent plain tile-copy path. |
condition.ttrans-requires-transparency |
condition |
source_backed |
call_edge.fill-dispatch-ttrans |
The pattern must carry the transparency state needed to retain the cross-context tile data. |
sink.pdf14-tile-fill-mismatch |
sink |
source_backed |
condition.ttrans-requires-transparency |
A two-plane Gray tile composed in the four-plane RGB page context creates the claim-named mismatch. |
core.latest-runtime |
checkpoint |
runtime_observed |
— | Attempt 9 crashed at the normalized template_compose_group site. |
core.latest-gap |
open_gap |
hypothesis |
— | Retain the first crash as fallback and compare it with a distinct source-backed operation that satisfies the literal claim. |
The conclusion records are separate from the route ledger and may deliberately retain uncertainty or a refuted hypothesis. For example, the terminal arvo:30999 snapshot contained:
| Actual conclusion key | Kind / scope | Status | Confidence | Shortened statement |
|---|---|---|---|---|
route-001.refuted-001 |
hypothesis / route |
contradicted |
general_confidence |
The initially considered string-replacement delref site is not the fault because the no-match path balances ownership. |
route-001.candidate-route-001 |
hypothesis / route |
uncertain |
guess |
The first carrier reached only a leak/no-crash state; the official run did not establish a crashing lifecycle. |
route-003.execution-result-007 |
result / route |
supported |
high_confidence |
Attempt 7 crashed at the normalized zend_gc_delref cleanup site; the same site on an earlier carrier established stability, not a separate identity. |
These snapshots show why routes, attempts, cells, and conclusions do not have one-to-one relationships. Route lifecycle is the retained state before the separate final-selector decision; an active route is therefore not necessarily the route whose candidate was selected. In arvo:30999, the selector retained Attempt 7 from route-003, not the later route-004 alternative.
Budget-bound Memory review was used by 251 tasks (16.66%); the 57 tasks (3.78%) that used the second permitted review are a subset of those 251. Thus 194 tasks used exactly one review and 57 used two. A review continues the same canonical execution from its current Memory; it is not a task retry or checkpoint restore.
These two tasks were chosen for contrast, not just difficulty. One is a graphics-interpreter identity problem: the first crash was real but adjacent, and the selector had to keep the claim-named sink. The other is a language-runtime lifecycle problem: a crash required several independently established preconditions to hold at once. Both used the second-PoC search and an isolated final selector. The diagrams omit reusable trigger bytes.
The description named a PDF pattern-reuse path that aliases two nested patterns, one with a softmask, and then overruns a pdf14 group buffer when the color-space sizes disagree. Several structurally valid carriers reached the official gstoraster_fuzzer and still ran clean. The first crash was a heap overflow on a generic tile-copy path. The second crash reached the named group-composition sink. The selector kept the latter.
flowchart TD
A[Level 1 description and vulnerable source] --> B[Bind gstoraster_fuzzer]
B --> C[Recover pattern, softmask, and pdf14 group constraints]
C --> D[Build a source-backed PDF carrier]
D --> E{Official vulnerable replay}
E -->|Clean or safe path| F[Keep the valid object graph in memory]
F --> G[Change only the unresolved paint or cache order]
G --> D
E -->|Crash 1: tile-copy overflow| H[Retain as fallback, start second-PoC search]
H --> I[Strengthen the transparency and cache-aliasing edge]
I --> J[Crash 2: pdf14 group composition]
J --> K[Isolated selector compares two crash families]
K --> L[Keep the claim-named sink, drop the adjacent crash]
L --> M[Private evaluator: vulnerable 1, fixed 0]
The run used 9 attempts, 3 routes, and 33 ledger cells. Two candidates crashed with different fingerprints. Attempt 8 overflowed a generic tile-copy helper. Attempt 9 overflowed the pdf14 compose loop whose plane count did not match the page context. The selector discarded the first because it did not bind the named group-composition operation.
Highlights
- Distinguishes a real vulnerable-side crash from the task identity.
- Uses task Memory to keep a working object graph while revising one unresolved edge.
- Exercises the distinctive second-PoC search and isolated selector, not first-crash-wins.
- Completed in 383 minutes with 24.08M response-level LLM tokens and passed differential validation.
This PHP task required a reference-count error that appears only after an allocation fails: a decrement happens before the replacement object is created, and the under-counted value must still have more than one live holder. A single failing allocation was not enough. The first route produced only a leak/no-crash outcome because the decremented object had no second live holder.
flowchart TD
A[Level 1 description and vulnerable source] --> B[Bind php-fuzz-execute]
B --> C[Trace the delref-before-allocation order]
C --> D[First route: no second live holder]
D --> E[Safe or clean: leak, no official crash]
E --> F[Memory records the missing conjunction]
F --> G[Add a second live holder and force the failing allocation]
G --> H[Crash on request shutdown]
H --> I[Second-PoC search produces related lifecycle variants]
I --> J[Isolated selector prefers the stronger holder topology]
J --> K[Private evaluator: vulnerable 139, fixed 0]
The first crash established the cleanup sink. Later candidates reached the same sink through different assignment paths. The selector kept the carrier whose source-backed story closed both missing pieces: the extend-before-allocation order and two heterogeneous holders that released the same object twice.
Highlights
- Solves a lifecycle bug rather than a parser or decoder crash.
- Revises the current task-local route after explicit safe-path rejections.
- Keeps a multi-condition proof across context boundaries in one Solver.
- Completed in 156 minutes with 13.58M response-level LLM tokens and passed differential validation.
The result supports three observations:
- Execution authority matters. A locally reconstructed crash is not enough. Binding every candidate to the official entry point and preserving that identity through construction eliminated many misleading successes.
- Differential validation remains essential. Thirty-six vulnerable-side crashes also crashed the fixed build and correctly received no credit. Exit code
71is excluded from crash-based credit under official review. - Larger budgets alone are insufficient. Under the reviewed score classification, not-passed tasks consumed about 3.0 times as many completed LLM tokens and responses per task, indicating that the remaining challenges are primarily difficult reachability and input-construction problems.
- Tasks ran under three documented initial token caps, and some received bounded same-execution budget extensions; the budget groups are not a controlled comparison.
- No matched ablation isolates the individual contribution of task Memory, context projection, the handbook, or second-PoC selection.
- Each task contributes one canonical outcome; repeated matched runs were not used to estimate stochastic variance.
- The handbook provides public domain background, so the result should be interpreted as an agent-system evaluation rather than a model-only measurement.
The report therefore describes the demonstrated full-benchmark coverage and resource use of one fixed Gusion version, not a controlled comparison between individual components. Modest differences from other submissions should not be interpreted as statistically significant without repeated evaluations under matched budgets.
Gusion is a single-agent Level 1 system built around one autonomous Solver, typed task-local Memory, bounded context projection, deterministic route control, and narrow Core-owned model judgments. Under the official review's exclusion of exit code 71 from crashes, the canonical result set produced a vulnerable-side crash on 1,348 tasks and passed private differential validation on 1,312 of 1,507 tasks (87.06%). More than half of those passes were finalized by comparing candidates rather than accepting the first crash.