llm-serving-finetune-lab

A hands on lab for the operational side of LLMs: serving (vLLM on GPU with a CPU fallback behind one OpenAI compatible API), fine-tuning (QLoRA on GPU, LoRA on CPU, on a tiny model and dataset), quantisation trade offs measured on CPU, Kubernetes deployment with a Helm chart and HorizontalPodAutoscaler, and load testing with k6 and Locust.

It is sized to run on a laptop with no paid API keys. CI only runs the fast parts (unit tests, a load test smoke run against a stub backend, Helm lint and render); training and benchmarks are scripts you run yourself.

Lab project. Numbers below come from one run on a CPU dev box with a 135M parameter model. They illustrate the trade offs, they are not production benchmarks.

Architecture

flowchart LR
  C[Clients / k6 / Locust] --> SVC[K8s Service]
  SVC --> P1[Pod: lab.server]
  SVC --> P2[Pod: lab.server]
  HPA[HPA: CPU % or llm_requests_in_flight] -.scales.-> P1
  P1 --> SEL{LLM_BACKEND}
  SEL -->|GPU + vllm| V[vllm serve: paged attention, prefix caching, LoRA adapters]
  SEL -->|CPU| T[transformers fallback]
  SEL -->|CI| S[stub backend]
  P1 --> M[/metrics: Prometheus text/]
  FT[finetune/qlora.py] -->|LoRA adapter| V
  Q[quant/compare.py] --> R[(results/*.md)]

Loading

What is in here

Path	Purpose
lab/server.py	FastAPI /v1/completions (OpenAI shape), /health, /metrics (requests, tokens, in flight, latency)
lab/backends.py	vLLM proxy, transformers CPU fallback and stub, picked automatically
lab/vllm_config.py, serving/vllm.sh	Typed, validated vllm serve arguments (dtype, max model length, GPU memory, tensor parallel, AWQ/GPTQ/FP8, prefix caching, LoRA modules); the script falls back to CPU when no GPU is present
finetune/qlora.py	NF4 4 bit QLoRA with bitsandbytes on CUDA, plain LoRA in fp32 on CPU, rank 8 on q_proj/v_proj
quant/compare.py	fp32 vs bf16 vs dynamic int8: size, latency and perplexity
deploy/helm/llm-serving	Deployment (CPU or GPU mode), Service, HPA (autoscaling/v2, CPU plus optional custom metric), probes
loadtest/	k6 ramp scenario with thresholds, Locust user, and a stdlib smoke load generator

Results from one CPU run

Model HuggingFaceTB/SmolLM2-135M-Instruct, 8 vCPU Intel Xeon, PyTorch CPU build, greedy generation of 32 tokens, best p50 of 5.

Quantisation (python quant/compare.py):

Variant	Size (MB)	p50 latency, 32 tokens (ms)	Tokens/s	Perplexity
fp32	538.2	535.8	59.7	13.28
bf16	269.1	709.0	45.1	13.20
int8 dynamic	248.1	506.3	63.2	38.22

Observations:

• bf16 halves memory with no quality loss, but is slower on this CPU, which lacks fast bf16 matmul kernels. On GPUs it is the normal default.
• Dynamic int8 only quantises Linear layers (embeddings stay fp32, so the size drop is modest) and is slightly faster, but perplexity almost triples. Small models are very sensitive to naive per tensor quantisation; this is why weight only methods such as AWQ/GPTQ, or keeping sensitive layers in higher precision, are used in practice.
• Size is the serialized state dict, which stores the tied embedding and output head separately.

Fine-tune (python finetune/qlora.py --epochs 3, CPU so LoRA fp32 rather than 4 bit): 460,800 trainable parameters (0.34% of 135M); training set perplexity went from 34.7 to 29.4 on the 12 example toy dataset. That shows the loop works, not that the model learned anything general: there is no held out set at this size.

Run it

python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"
LLM_BACKEND=stub uvicorn lab.server:app --port 8000          # instant, no model
pip install --index-url https://download.pytorch.org/whl/cpu torch && pip install -e ".[ml]"
LLM_BACKEND=transformers uvicorn lab.server:app --port 8000  # real model on CPU
./serving/vllm.sh                                             # vLLM if a GPU is present, else CPU

curl -s localhost:8000/v1/completions -H 'content-type: application/json' -d '{"prompt":"Ireland is","max_tokens":16}'

python quant/compare.py
python finetune/qlora.py --epochs 3

k6 run -e BASE_URL=http://localhost:8000 loadtest/k6.js
locust -f loadtest/locustfile.py --host http://localhost:8000 --users 30 --spawn-rate 5 --run-time 2m --headless

helm install llm deploy/helm/llm-serving                               # CPU mode
helm install llm deploy/helm/llm-serving --set mode=gpu --set autoscaling.inFlightRequestsPerPod=8

Tests and CI

pytest covers the OpenAI response shape, request validation, metrics counters, backend selection, vLLM argument building and validation, and the perplexity, percentile and table helpers. CI also starts the server with the stub backend and runs 200 concurrent requests, lints and renders the Helm chart in CPU and GPU modes (asserting the HPA, GPU resource and custom metric appear), and builds the Docker image.

Design decisions

• One API, three backends. Clients and load tests do not change between laptop, CI and GPU cluster; only LLM_BACKEND does.
• Validate serving config in code rather than hand editing flags: invalid GPU memory fractions, non power of two tensor parallel sizes or unknown quantisation fail fast.
• Scale on CPU by default, in flight requests optionally. CPU utilisation is a poor signal for GPU inference; the chart can add a Pods metric fed by /metrics through prometheus-adapter.
• Slow readiness, slower liveness. Model loading takes time, so liveness starts late to avoid restart loops.
• Keep CI fast. Training and model downloads are deliberately excluded.

Limitations

• vLLM and QLoRA 4 bit paths need a CUDA GPU and were not exercised in CI.
• The fine-tuning dataset is a toy; there is no evaluation set.
• The CPU server handles one generation at a time per worker (no continuous batching; that is what vLLM is for).
• No auth, rate limiting or streaming responses.

License

MIT