[Blog] Infer-forge: Loop and Graph Engineering Around SGLang

[TianyuZhang1214]

Summary

This PR adds “Infer-forge: Loop and Graph Engineering Around SGLang,” a technical blog describing our agent-driven engineering practice for managing inference-system complexity around SGLang. Infer-forge builds from Harness Engineering to long-running Task Loops, then composes independently verifiable Tasks into project-scale Task Graphs.

Infer-forge is an independently developed internal engineering system around SGLang, not an official SGLang or LMSYS component. Its current repository is not open source because the core components are tightly coupled to our internal environment. This article instead makes its construction methodology public as a blueprint for building an environment-specific system with AI coding tools.

What the Article Covers

• Inference as a deployment space: An optimization is meaningful only within a complete deployment point spanning the model, serving scenario, SLO, topology, runtime, and accelerator platform.
• MonoRepo and Harness: A reproducible cross-repository workspace connects kernels, communication libraries, SGLang, deployment, Evaluation, diagnosis, memory, compute resources, and safety boundaries.
• Task Loop: A persistent Task can advance across multiple Agent executions while preserving its contract, evidence, Verification boundaries, and Handoffs.
• Task Graph: Multiple Tasks can be coordinated into large engineering projects and capability-evolution workflows without losing provenance, constraints, or paths that did not ship.
• Lessons learned: The article discusses Task granularity, evidence-driven graph evolution, verifiable Deliverables, and why Context, Harness, Loop, and Graph accumulate rather than replace one another.

Evidence from Practice

• From April to July, the observed peak number of inference-engineering Tasks in flight rose from 2 to 9. The count is based on overlapping Task lifetimes from creation to archive.
• A large DeepSeek-V4-Pro serving optimization project was organized as 38 independently verifiable Task nodes across seven Task Types.
• The project delivered four serving profiles while preserving seven paths that did not enter the final Release, keeping both successful and unsuccessful engineering decisions reproducible and auditable.
• These figures are longitudinal operating observations from one engineer, not a controlled productivity benchmark.

Assets and Validation

• Adds a dedicated blog cover and 16 technical figures.
• Verifies that all local image references resolve correctly.
• Passes git diff --check.
• Passes the complete npm run build.

[BBuf]

Thanks for the detailed write-up. I left five inline comments, mainly around disclosure boundaries, measurement methodology, implementation status, version-specific accuracy, and node-reclaim semantics.

[BBuf]

Could we confirm that all details in Figure 15 are approved for public disclosure? The SVG contains substantially more than the caption—including H20, W4A8/W8A8, C128, Humming/DSpark, PP/CP/TP/DP/EP topologies, rejected paths, and released profiles. The model name being public does not necessarily make these operational details public. If approval is unclear, please anonymize the body, SVG contents, filename, alt text, and caption together.

[TianyuZhang1214]

Thanks for checking. Yes, this disclosure is intentional. The operational details shown in Figure 15 are all covered in the companion DeepSeek-V4-Pro blog PR (#383). Figure 15 reorganizes the same disclosed material—including the evaluated paths, topology choices, and released profiles—into a Task Graph to illustrate how the project was coordinated.

We plan to publish the DeepSeek-V4-Pro article on LMSYS first. Once it is live, we will add its published URL next to Figure 15 so readers can trace every detail back to the companion article and the disclosure boundary is explicit.

[BBuf]

“4.5× engineering concurrency” seems stronger than the metric supports. This is peak tasks in flight, and the median task lifetime also grew from ~10h to ~28h; higher WIP can therefore reflect longer cycle time or queueing rather than more concurrent execution or capacity. Could we rename this to “peak tasks in flight increased from 2 to 9” and document the inclusion/exclusion criteria, time zone, overlap calculation, and the four omitted tasks? If we want to make a capacity or productivity claim, we should also report completion throughput and waiting versus active time.

[TianyuZhang1214]

Thanks for the feedback. We revised the article to describe the metric strictly as peak Tasks in flight, removing the 4.5× engineering concurrency and operating-capacity claims.

We also clarified that the peak is calculated from overlapping creation-to-archive intervals and documented the inclusion criteria: 86 valid archived Tasks were included, three remained open at the cutoff, and one invalid record was excluded.

[BBuf]

Could we distinguish implemented, partially automated, manual, and proposed components? Calling infer-forge “our implementation” makes the diagrams read as current executable behavior, but the article and PR do not link a repository, version, or runnable example. A compact implementation-status table—and clarification of whether this is an internal system, an independent project around SGLang, or an official SGLang/LMSYS component—would prevent readers from over-interpreting the scope.

[BBuf]

Also, is infer-forge currently open source? If not, do you plan to open-source it, and is there an approximate timeline? A repository link or a brief statement about its availability would help set readers’ expectations.

[TianyuZhang1214]

Thanks for raising this. We added a status table clarifying that infer-forge is an independently developed internal system, not an official SGLang or LMSYS component.

Task generation and adaptive Task Graph evolution are implemented: infer-forge derives Tasks from user goals and updates the Graph as Tasks are accepted, abandoned, or redirected. These decisions, Human Gates, and releases remain human-directed.

The repository is not currently open source because it is tightly coupled to our internal environment. Instead, the article makes the transferable construction methodology public so teams can build an environment-specific implementation with AI coding tools.

[BBuf]

This should be versioned and linked so it does not read as a current unresolved bug. The streaming issue was fixed by sgl-project/sglang#23954, with separate non-stream handling in #25298. Suggest: “In an earlier version of the OpenAI-compatible chat benchmark handler, streamed delta.reasoning_content was not included in TTFT and output accounting (fixed in sgl-project/sglang#23954).” This example is also repeated in §8.3; keeping it once would be tighter.

[TianyuZhang1214]

Thanks. We clarified that this was a historical issue and linked the fixes for the streaming and non-streaming paths. We also kept the example only in the Journal section and removed the repetition from §8.3.

[BBuf]

DCGM can report GPU telemetry and process activity, but it cannot by itself establish that a task is finished or that reclaim is safe: a task may be compiling, downloading, blocked on communication, or in a CPU-only phase. Could we describe DCGM as one signal and mention a lease, heartbeat, or owner confirmation before reclaim? Also, please confirm whether the exact six-hour internal policy is intended to be public; otherwise, use a configurable illustrative threshold.

[TianyuZhang1214]

Thanks—agreed. We revised the text and Figure 6 to make clear that GPU telemetry is only one of two periodically recorded signals, not proof of Task completion. Claim cleanup requires fresh, gap-free runtime-idle and GPU-idle history over a configurable policy window; missing or conflicting evidence blocks cleanup. We also clarify that cleanup is explicit, removes only the Registry claim, and Node state is revalidated before reuse. The fixed six-hour wording has been removed.