Two agents, one T4: self-hosting on a budget

The unglamorous reality of building an AI product outside the API-budget class: your model runs on the GPU you can actually afford. For Harrir that is one NVIDIA T4, 16 GB of Turing-era VRAM, serving a vision-language chat model, two fine-tuned agents and an embedding model at the same time.

Two Dockerized vLLM servers sit behind a host nginx that routes by path segment. /chat serves Qwen3-VL-8B quantized to AWQ 4-bit with the two agent adapters applied per request; /embed serves a 0.6B embedding model. Clients get one base URL and pick behaviour with the model field.

How 16 GB gets divided

                          chat          embed
gpu-memory-utilization    0.76          0.16
weights loaded            ~7.6 GB       ~1.4 GB
max-model-len             8192          2048
max-num-seqs              12            default
CPU / RAM cap             -             2 vCPU / 4 GB

After weights, roughly 1.2 GB remains for the chat KV cache, which is about 8,600 tokens. At the full 8192-token context that is barely more than one concurrent request. KV cache, not weights, is the binding constraint on this box, and the consequence is counterintuitive: when you need more concurrency, lower max-model-len before touching the utilization fractions.

Two flags doing load-bearing work

Running with enforce-eager looks like leaving performance on the table, and it is, deliberately. LoRA-specialized CUDA graphs roughly double cudagraph VRAM, and on this card that overhead comes directly out of KV cache. Eager mode gives up some decode throughput and buys back the headroom that lets requests run at all.

The tempting fix when LoRA feels tight is to lower gpu-memory-utilization to 'free up space'. That does the opposite: it shrinks the pool KV cache is allocated from, so you trade a throughput problem for an out-of-memory one.

Pinning adapters to commits

Each agent has one Hugging Face repo, and versions are commits inside it. Serving flags point at local directories rather than bare repo ids.

--lora-modules support=/models/huggingface/adapters/support-d7c7751 \
               stylist=/models/huggingface/adapters/stylist-8afd31a

An init container downloads each promoted commit into its own directory on the model-cache volume before vLLM starts, guarded by a completion marker so it only re-downloads when a SHA actually changes. A plain restart costs nothing.

The reason for all of that: a bare repo id resolves to HEAD of the default branch. Push an experimental adapter, run a force-recreate three weeks later for an unrelated reason, and production silently starts serving weights that never passed a gate. Pinning the SHA makes container start deterministic.

The train/serve quantization gap

You cannot LoRA a 4-bit model, so adapters train on the bf16 base and serve unmerged on top of the AWQ base. That mismatch is real and it has already cost a release: an adapter that evaluated perfectly in bf16 produced garbage on the quantized serving path.

Two rules came out of that. Evaluate candidates on the exact production quantization, not the training precision. And keep LoRA targets on the attention projections only. Adding the MLP projections was tried and reverted: a bf16-trained LoRA on gate, up and down projections is numerically unstable on an AWQ-4bit base and emits garbage live.

Numbers to plan around

Cold engine init takes about 260 seconds with multi-LoRA on this card. Health checks need a start period generous enough to cover it, or your orchestrator will restart a container that was booting normally, forever.

Time to first token is roughly 0.2 seconds for text and 0.5 to 2 seconds once an image is in the prompt. The embedding server answers short inputs in under 20 milliseconds when warm, but the first call after any restart takes about 40 seconds while FlashInfer JIT-compiles its kernels. That one surprised me in staging and would have looked like an outage in production.

Two build details are worth stealing. The image uses the CUDA devel base rather than runtime, because vLLM, FlashInfer and Triton compile kernels at runtime and need nvcc and headers present. And vLLM is pinned rather than floating: a later release ships a FlashInfer version that crashes paged-KV prefill on Qwen3-VL image inputs, which is a crash you only find by sending a photo to production.

~260s

engine init on a cold recreate

0.92

of 16 GB allocated across both servers

2

adapters sharing one base model

The property all of this buys is boring and worth the effort: clients ask for 'support' or 'stylist' and never learn which checkpoint answered them. Model cutover is a config change instead of a client migration, and the monthly bill stays a number a bootstrapped company can say out loud.