You spin up a local LLM, point one curl request at it, and watch nvidia-smi while it generates. GPU utilization bounces around 20-40%. You paid for a card that can do trillions of operations a second and it is mostly sitting there, waiting.
Then you get ambitious. You wire the model into a Slack bot, a home automation assistant, and a little RAG tool for your notes. Three users hit it at the same time and now requests queue up behind each other like it’s the DMV. Your instinct says the GPU is out of compute. It isn’t. It’s out of something else, and once you know what that something is, you can serve 20 or 50 concurrent chats on the same card that was struggling with three, for barely more cost than serving one.
That something is memory bandwidth, and the trick is batching.
The GPU Isn’t Doing Math, It’s Doing Deliveries
Generating one token with an LLM means multiplying your input against every weight in the model, layer by layer. For a 12B parameter model in 16-bit precision, that’s roughly 24GB of weights that have to move from VRAM into the GPU’s compute cores for every single token you generate.
The part that trips people up: the actual multiplication is fast. Modern GPUs chew through matrix math in microseconds. What’s slow is hauling 24GB of numbers across the memory bus to feed that math, one token at a time, over and over. During single-request decode, your GPU’s compute units spend most of their time waiting on data to arrive, not crunching it. That’s what “memory bandwidth bound” means in practice: the delivery truck is the bottleneck, not the warehouse workers unloading it.
This is why one request rarely maxes out a GPU. You’re paying the full cost of moving every weight through the chip, to produce a single token for a single user. That’s an expensive delivery for one package.
Now the reframe: what if the delivery truck brought packages for 20 people at once? Same trip, same bandwidth cost, twenty times the payload. That’s batch inference. You run the same weights-loading pass across multiple requests’ hidden states at the same time, so the cost of moving those weights gets divided across every token generated for every request in the batch, instead of paid fresh for each one.
Static Batching Is the Old, Annoying Way
The naive version of this idea is static batching: collect N requests, pad them to the same length, run them through the model together as one fixed batch, and don’t start anything new until the whole batch finishes.
That works fine for a research benchmark where every prompt is the same length and nobody cares about latency. It’s miserable for chat. Real conversations have wildly different lengths. One user asks “what’s 2+2” and another pastes in a 3000 token error log. With static batching, the short request finishes generating in a few tokens and then just sits there, wasting a slot, while the model keeps churning through the long one. New requests that arrive mid-batch wait in a queue until the whole cohort completes, even if there’s an empty seat.
It’s like a shuttle bus that won’t leave the airport until every seat is full, then won’t let anyone off until every single passenger reaches their stop, even the ones who wanted the first exit. Efficient on paper, infuriating to actually ride.
Continuous Batching Fixes the Bus Schedule
Continuous batching (you’ll also see it called in-flight batching) evaluates the batch at the level of individual decode steps instead of whole requests. At every step, the server looks at which sequences still have work to do, packs their next-token computation into one pass through the model, and the moment a sequence finishes, its slot opens up for a brand new request immediately, mid-stream, without waiting for its neighbors.
Short requests exit early and free their seat. Long requests keep running. New requests slide into whatever’s open. The GPU stays busy doing batched decode work almost continuously instead of alternating between “processing a full batch” and “waiting for stragglers.” This is what turns batching from a research trick into something you can put behind a real chat endpoint with unpredictable traffic, which is most home lab and small-team use cases.
vLLM popularized this pattern along with PagedAttention, which manages each sequence’s KV cache in fixed-size, non-contiguous blocks (the same idea as OS virtual memory paging) so the server doesn’t have to reserve one giant contiguous memory region per request up front. llama.cpp’s llama-server implements its own version of continuous batching with parallel decode slots, which is the one worth running if your GPU is a single consumer card and not a rack of A100s.
Running It: llama-server With Parallel Slots
llama-server is the OpenAI-compatible HTTP server that ships with llama.cpp. Two flags matter here, straight from the current server documentation:
-np, --parallel N: number of server slots. Default is-1, which means automatic.-cb, --cont-batching: whether to enable continuous batching. It’s enabled by default as of the current release, with-nocb, --no-cont-batchingto turn it off.
So on a recent build you don’t have to explicitly ask for continuous batching, you have to explicitly ask for more slots to take advantage of it. Set your context size and slot count together, because each slot needs its own KV cache carved out of your context budget.
A docker-compose setup for a Gemma 4 12B GGUF, sized for parallel chat traffic:
services: llama-server: image: ghcr.io/ggml-org/llama.cpp:server-cuda container_name: llama-server restart: unless-stopped ports: - "8080:8080" volumes: - ./models:/models deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] command: > -m /models/gemma-4-12b-instruct.Q4_K_M.gguf -c 16384 -np 4 -cb --host 0.0.0.0 --port 8080That reserves 16384 total context tokens and 4 parallel slots, so plan on roughly 4096 tokens of usable context per concurrent conversation if they’re sharing that pool evenly. Bump -np and -c together as your VRAM allows.
CLI equivalent if you’d rather skip compose entirely:
llama-server \ -m ./models/gemma-4-12b-instruct.Q4_K_M.gguf \ -c 16384 \ -np 4 \ -cb \ --host 0.0.0.0 \ --port 8080Proving It: A Concurrent Load Test
Don’t take a blog post’s word for it, measure your own box. Here’s a small concurrent load test that fires N simultaneous completion requests and reports wall clock time, which you can run once against a server started with -np 1 and once against -np 4 (or whatever your card supports) to see the difference for yourself.
import asyncioimport timeimport httpx
SERVER = "http://localhost:8080/v1/chat/completions"CONCURRENCY = 8PROMPT = "Write a three sentence summary of why GPUs are memory bandwidth bound during LLM decode."
async def one_request(client: httpx.AsyncClient, i: int) -> float: start = time.monotonic() resp = await client.post( SERVER, json={ "model": "local", "messages": [{"role": "user", "content": PROMPT}], "max_tokens": 200, "stream": False, }, timeout=120, ) resp.raise_for_status() elapsed = time.monotonic() - start print(f"request {i}: {elapsed:.2f}s") return elapsed
async def main() -> None: async with httpx.AsyncClient() as client: start = time.monotonic() results = await asyncio.gather( *[one_request(client, i) for i in range(CONCURRENCY)] ) total = time.monotonic() - start print(f"\n{CONCURRENCY} requests, total wall clock: {total:.2f}s") print(f"average per-request latency: {sum(results) / len(results):.2f}s")
if __name__ == "__main__": asyncio.run(main())Run python3 load_test.py against a server started with -np 1, note the total wall clock time for 8 concurrent requests, then restart the server with -np 8 (adjust -c to fit your VRAM) and run it again. With -np 1, the requests effectively serialize since only one decode slot exists, so total wall clock scales close to linearly with request count. With enough parallel slots, the requests overlap and total wall clock drops toward roughly the time it takes to serve one request, because the GPU is spending its memory-bandwidth budget across all 8 at once instead of paying it fresh per request. The exact multiplier depends on your GPU, model size, prompt length, and quantization, so run it on your own hardware rather than trusting a number from someone else’s rig.
The Bill You Don’t See Coming: KV Cache
Every active sequence needs a KV cache entry per token per layer, and that cache lives in VRAM right alongside your model weights. More parallel slots means more concurrent sequences means more KV cache, and that memory is not optional or shared away for free.
Two levers fight each other here:
- Context length per slot. A longer
-cgives each conversation more room to breathe, but it also means each slot’s KV cache reservation grows. - Number of slots. More
-npmeans more concurrent users, but you’re dividing (or in some configurations, multiplying) your KV cache budget across more sequences.
If you set -c 32768 -np 8 on a card with 12GB of VRAM after the model weights are loaded, you’ll likely hit an out-of-memory error before the server finishes starting, because that’s 32768 tokens of context reserved per slot across 8 slots. Cut context per conversation, cut slot count, or move to a smaller quantization of the model, in whatever order matches your actual traffic pattern. A home assistant bot that only needs 2000 tokens of context per turn can run far more parallel slots than a coding assistant chewing through 16000 token files.
Quantizing the KV cache itself helps too. llama-server supports -ctk and -ctv to set the K and V cache data types (f16 by default, with q8_0 and other quantized options available), which shrinks the memory footprint per slot at some quality cost. Worth testing before you assume you need more VRAM.
When Batching Doesn’t Help You At All
If you are the only user, and you want the fastest possible time to first token for your one request, batching is not your friend. Running with -np 1 gives that single request the entire GPU’s memory bandwidth to itself for every decode step. Add parallel slots and, even with zero other traffic, some servers reserve KV cache and scheduling overhead per slot that can shave a sliver off best-case single-user latency.
Batching pays off when you have concurrent demand: multiple users, multiple background jobs, an API serving more than one client. If your actual traffic is “me, alone, at 2 AM, wanting the fastest possible answer,” don’t bother provisioning four slots you’ll never fill concurrently. Set -np 1 or -np 2 and get the lowest latency path instead. Over-provisioning slots for phantom concurrency is the inference equivalent of renting a moving truck to pick up a pizza. Technically it holds more, but you’re paying for capacity nobody needed.
Picking Your Server: llama.cpp vs vLLM
For a home lab running one consumer GPU (a 3090, 4090, or similar), llama-server is the pragmatic choice. It runs a huge range of quantized GGUF models, needs no Python environment or CUDA toolkit juggling, and its parallel slot model is simple to reason about: pick -np, pick -c, watch VRAM.
vLLM earns its keep once you have real GPU headroom, meaning a data-center-class card or multiple GPUs, and traffic heavy enough to benefit from PagedAttention’s more sophisticated memory management and features like continuous batching combined with speculative decoding, prefix caching, and tensor parallelism across cards. It’s a heavier piece of software to operate (Python, CUDA versions, more moving parts in the deploy) and that overhead only pays for itself at higher concurrency and scale than most home labs generate.
If you’re running one GPU and serving your household plus a couple of side projects, start with llama-server. Revisit vLLM only when you’ve actually hit its ceiling, not because a benchmark post told you to.
Common Questions
Does batching help if I only have one user?
No, not meaningfully. A single request already gets the GPU’s full memory bandwidth with one parallel slot. Extra slots add scheduling and KV cache overhead without concurrent work to fill them. Set -np 1 for single-user setups chasing the lowest possible time to first token instead of provisioning for concurrency you don’t have.
How much VRAM does each extra parallel slot cost?
Each slot reserves its own KV cache, sized by context length times model layers times attention head dimensions, on top of loaded model weights. A 4096-token slot costs roughly 4x the KV cache of a 1024-token slot for the same model. Cut context per slot or quantize the cache (-ctk q8_0) before assuming you need more VRAM.
Should I use vLLM or llama.cpp for a home lab GPU?
Use llama-server from llama.cpp for a single consumer GPU. It runs quantized GGUF models directly, needs no Python or CUDA toolkit setup, and its -np/-c slot model is simple to size. vLLM’s PagedAttention and advanced scheduling pay off on data-center-class or multi-GPU setups with heavier concurrent traffic than most home labs generate.
What’s the minimum hardware to see a benefit from batching?
Any single GPU running a model that leaves spare VRAM after loading weights, roughly 4GB or more free, can support two or three parallel slots and show a measurable throughput gain under concurrent load. Below that, there isn’t enough headroom for extra KV cache, and you’re better off running one slot at low latency.
Do I need to explicitly enable continuous batching in llama-server?
No, continuous batching is enabled by default in current llama-server builds, controlled by -cb/-nocb. The setting that actually needs configuring is -np to raise the parallel slot count above its automatic default, since that’s what lets more than one sequence share the GPU’s decode pass at once.
