Proposal: a vllm-worker worker hive
Summary
A worker hive whose bees raise vLLM server instances as cells. Gives hum a high-throughput, continuously-batched data-plane cell type, including multi-node tensor/pipeline parallelism via Ray.
Motivation
hum currently commissions worker hives that are single-process cells (claude-cli, ollama-worker). vLLM is the industry-standard inference engine: continuous batching, paged KV cache, speculative decode, and multi-GPU/multi-node parallelism. Nesting it as a worker hive lets one cell span several GPUs or several machines, which is the pooled-inference direction we want for frontier-scale models.
Hive stencil
- Typology: worker (accepts
chi:"prompt", emits chunk then finish).
- Bee binary:
vllm-worker, launches a vLLM server (or a vLLM worker inside a Ray cluster) as its cell.
- Propensity:
StatelessPerCall (vLLM is stateless; paged KV is per-request). EphemeralPerCall for one-shot.
- Richness: medium (system, content, tools); can accept
Pollen via vLLM's multimodal surface.
- Wire: as a worker it speaks thrum; optionally also the OpenAI
/v1/chat/completions contract as a forager surface.
Egg spec (candidate)
- model path/quant, ctx, max tokens, dtype
- tensor-parallel size, pipeline-parallel size, worker addresses
- continuous-batching knobs, KV cache size, gpu placement
Cell shape
One vLLM process. feed is the prompt JSON, mmm is the token chunk stream, emerged is the server exit, silence cancels. Standard WorkerBee::raise(egg) -> Cell, tended by lifecycle.rs with RSS/CPU sampling.
Integration with humd / Nest
Handshakes a humd over thrum, announces bee:["worker"], becomes nestable. Nest pool enforces max_procs and idle eviction. A Ray cluster can be represented as one logical cell or several, pending the open question below.
Risks
- vLLM targets CUDA/ROCm, not Apple Silicon. Needs a GPU host for meaningful throughput.
- Multi-node requires Ray; that dependency should be optional (single-node path first).
- Paged KV and hum's per-cell pooling need reconciliation (does the Nest own KV, or does vLLM?).
Open questions
- Represent a Ray cluster as a single cell or as N coordinated cells?
- Should this hive also be a forager (wrap vLLM's own OpenAI server) or stay pure worker?
- How does humd express GPU/device placement constraints in the Egg?
Proposal: a
vllm-workerworker hiveSummary
A worker hive whose bees raise vLLM server instances as cells. Gives hum a high-throughput, continuously-batched data-plane cell type, including multi-node tensor/pipeline parallelism via Ray.
Motivation
hum currently commissions worker hives that are single-process cells (
claude-cli,ollama-worker). vLLM is the industry-standard inference engine: continuous batching, paged KV cache, speculative decode, and multi-GPU/multi-node parallelism. Nesting it as a worker hive lets one cell span several GPUs or several machines, which is the pooled-inference direction we want for frontier-scale models.Hive stencil
chi:"prompt", emitschunkthenfinish).vllm-worker, launches a vLLM server (or a vLLM worker inside a Ray cluster) as its cell.StatelessPerCall(vLLM is stateless; paged KV is per-request).EphemeralPerCallfor one-shot.Pollenvia vLLM's multimodal surface./v1/chat/completionscontract as a forager surface.Egg spec (candidate)
Cell shape
One vLLM process.
feedis the prompt JSON,mmmis the token chunk stream,emergedis the server exit,silencecancels. StandardWorkerBee::raise(egg) -> Cell, tended bylifecycle.rswith RSS/CPU sampling.Integration with humd / Nest
Handshakes a humd over thrum, announces
bee:["worker"], becomes nestable. Nest pool enforcesmax_procsand idle eviction. A Ray cluster can be represented as one logical cell or several, pending the open question below.Risks
Open questions