The Inference Box

Every token Harrir's agents produce comes off one NVIDIA T4. Two vLLM servers sit behind an nginx that routes by path: one serves the quantized vision-language model with the two agent adapters applied per request, the other serves embeddings.

Making 16 GB fit

                          chat          embed
gpu-memory-utilization    0.76          0.16
weights loaded            ~7.6 GB       ~1.4 GB
max-model-len             8192          2048

After weights, roughly 1.2 GB is left for the KV cache. KV cache, not weights, is the binding constraint here, which makes the tuning counterintuitive: when you need more concurrency, lower the context length before touching the memory fractions.

Shipping a model like shipping code

Adapter commits are SHA-pinned by an init container before vLLM boots, so a container restart can never silently pick up un-promoted weights. New adapters stage under a temporary name, pass a smoke test, then swap through vLLM's runtime LoRA API with no restart, and the swap is confirmed against the model list rather than assumed.

Six adapter versions have shipped this way, each one gated on a champion-versus-challenger replay against held-out conversations with 95% confidence intervals. A candidate that does not beat the live adapter on tool-routing accuracy does not ship.