What is vLLM?
vLLM is an open-source library for running large language models as a service. It takes a model you already have, a Llama or a Mistral or Apertus, and turns it into an API that applications can call. It was originally developed in the Sky Computing Lab at UC Berkeley and is now maintained by a community of over 2,000 contributors.
The distinction that matters: vLLM does not train models and it does not provide them. It serves them. If you have decided to run open-weight models on your own hardware instead of calling a hosted API, vLLM is one of the pieces that makes that practical.
The problem it solves
A GPU running a language model spends most of its memory on the KV cache, the intermediate state each in-flight request accumulates as it generates tokens. Naive serving reserves a fixed block of that memory per request, sized for the longest output the request might produce. Most requests are shorter than the worst case, so most of the reserved memory sits unused, and the number of requests the GPU can handle at once is far lower than its capacity would allow.
vLLM's PagedAttention manages that memory in pages instead, the way an operating system manages virtual memory. Requests take the memory they are actually using, and the GPU holds many more of them concurrently. On the benchmark vLLM published against naive HuggingFace serving, this reaches up to 23 times the throughput.
Two related mechanisms matter as much in practice. Continuous batching admits new requests into a running batch as earlier ones finish, rather than waiting for a whole batch to complete. Tensor parallelism splits a model across several GPUs when it does not fit on one.
When vLLM fits
It fits when you are serving open-weight models to real traffic and the GPU bill is a line item you care about. Concurrency is where it earns its place: a single request on an idle GPU runs at roughly the speed any serving stack would give it.
It fits when the prompts cannot leave your jurisdiction. Running the model yourself is the only arrangement in which the text of a prompt never reaches a third party, which is the reason most of the Swiss organizations we work with look at it at all.
When it does not
If you are experimenting on a laptop, a lighter local runner will get you to a first response faster. vLLM is built for a server with datacenter GPUs.
If your traffic is a handful of requests a day, a hosted API is almost certainly cheaper. A GPU you rent by the hour costs the same whether it serves one request or a thousand, and the throughput advantage that justifies vLLM only pays off under load.
If you need a specific proprietary model, the question does not arise. vLLM serves open-weight models; it cannot serve a model nobody will give you the weights to.
vLLM compared with a hosted API
A hosted inference API is a service. vLLM is a component you operate. Choosing it means taking on the model lifecycle, the GPU capacity planning, and the availability of the endpoint, in exchange for control over where the data goes and what the marginal request costs.
The compatibility detail that makes the choice reversible: vLLM exposes an OpenAI-compatible API. An application written against the OpenAI client libraries can point at a vLLM endpoint without code changes, which means the decision can be tested rather than committed to.
What running it yourself involves
The parts that are genuinely straightforward: installing vLLM, loading a model, getting a response from a single GPU.
The parts that are not, and that decide whether it survives contact with production traffic:
- GPU scheduling. Inference pods, batch jobs and everything else on the cluster compete for a scarce resource. Device plugins, resource quotas and priority classes decide who wins, and the defaults do not.
- Capacity and cost. Model weights alone run to tens of gigabytes, before any consideration of how much GPU the traffic profile needs. Sizing this by guess is expensive in both directions.
- Scaling on the right signal. Autoscaling an inference service on CPU utilization does not work. Queue depth and latency targets do.
- Knowing what it costs. Inference latency at p50, p95 and p99, tokens per second, GPU utilization and queue depth are the metrics that tell you whether to scale up, scale down, or change the model.
None of this is exotic. It is ordinary platform engineering, and it is the reason a proof of concept that worked on one GPU often stalls before it reaches users.
Where VSHN fits
VSHN deploys and operates vLLM on Kubernetes and OpenShift in Swiss data centers, on Exoscale, Cloudscale and other Swiss providers. Model weights, prompts, completions and inference logs stay in the chosen jurisdiction, and our sovereignty assessment sets out how that is scored against the EU Cloud Sovereignty Framework.
If you are deciding whether to serve models yourself, the useful first step is usually sizing rather than architecture: how much GPU your model and traffic profile actually need, and what that costs against the hosted alternative.