vLLM vs Ollama for On-Prem LLM Serving: Concurrency, Memory and Security

Once a local model works on a laptop, the next question is how to serve it to a team. The two most common answers are Ollama, the easiest way to run a model on any machine, and vLLM, the throughput-focused server behind many production deployments. vLLM vs Ollama is not a question of which one is better, but of which workload you have: a handful of users and many models, or many users hitting one model.

This guide compares them on what matters for an on-prem deployment in a regulated organization: how each handles concurrent requests, how they use GPU memory, which model formats and hardware they support, and what their security defaults mean for you. It ends with a benchmark harness you can run against both.

The short answer

• Choose Ollama for single-user tools, internal prototypes, Macs, mixed fleets and workloads that switch between several models.
• Choose vLLM when one model serves many concurrent users or services on NVIDIA (or AMD) GPUs and throughput per GPU matters.
• Use both in many organizations: Ollama on developer machines, vLLM on the shared inference server, with the same OpenAI-compatible client code pointing at either.

How each server works

Ollama wraps a llama.cpp-derived inference engine in a model registry, a scheduler and an API. Models are GGUF files pulled by tag (qwen3:32b), loaded on demand and unloaded after five minutes of inactivity by default. Its defaults favour running many models occasionally: it processes one request per model at a time unless you raise OLLAMA_NUM_PARALLEL, queues up to 512 more, and loads up to three models per GPU. The default context window is 4,096 tokens.

vLLM is built for one model under heavy load. Two ideas drive it. PagedAttention stores the KV cache in fixed-size blocks, the way an operating system pages memory, so many sequences share GPU memory without fragmentation. Continuous batching adds new requests to the running batch at every step instead of waiting for a batch to finish. At startup vLLM claims a fixed share of GPU memory (--gpu-memory-utilization, 0.92 by default in the current docs) for the weights plus as much KV cache as fits, and keeps it.

Head-to-head comparison

The table compares the two servers on deployment-relevant behaviour, based on each project's current documentation. It covers defaults, not measured performance.

Memory: why the same model uses different amounts

The weights cost the same on both servers; the difference is the KV cache. For Qwen3-32B at Q4_K_M, our VRAM estimator gives 18.5 GB of weights, 2.0 GB of FP16 KV cache per 8k-token sequence, and about 1 GB of runtime overhead.

Ollama allocates KV cache for each parallel slot. Its FAQ notes that memory scales with OLLAMA_NUM_PARALLEL × context length. These figures are formula-based estimates, not measurements:

Four parallel 8k slots cost exactly as much as one 32k context, so on a 24 GB RTX 4090 you have to choose between concurrency and context length.

vLLM takes the decision away: on a 32 GB RTX 5090 it reserves roughly 29 GB (92%), loads the weights, and turns everything left into a shared KV-cache pool. Short requests use a little of it and long ones use more, so the same card serves many short conversations or a few long ones without reconfiguration. The cost is that nothing else can use that GPU while vLLM runs, and an out-of-memory error at startup usually means --max-model-len is too large for the pool.

Concurrency: where vLLM pulls ahead

Single-user generation is bound by memory bandwidth: every token requires reading all the weights once. That is why our estimator gives the same order of decode speed regardless of server, about 24–37 tokens/s for Qwen3-32B Q4_K_M on an RTX 4090 at 8k (an estimate from bandwidth, not a measurement).

Batching changes the economics. When eight requests share one decode step, the weights are read once and serve eight tokens. vLLM's scheduler is built around this, so aggregate throughput keeps rising with concurrent users until the GPU becomes compute-bound. Ollama can batch parallel slots too once you raise OLLAMA_NUM_PARALLEL, but its fixed per-slot allocation and simpler scheduler are designed for small numbers of users.

We have not yet run the measured comparison on our reference hardware; the harness below produces it on yours. Treat any article that quotes a single "X times faster" figure with care: the ratio depends heavily on concurrency, prompt length and quantization.

Serving the same model on both

Ollama

Install on Linux and pull the model:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3:32b

For a shared server, raise the context and parallelism, keep the model loaded, and use the 8-bit KV cache. With the systemd service, add these to an override (sudo systemctl edit ollama):

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Then restart it and check that the model fits entirely on the GPU:

sudo systemctl restart ollama
ollama ps

vLLM

Install with uv (Python 3.11 or later) and serve the AWQ 4-bit checkpoint with an API key:

uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen3-32B-AWQ \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 32 \
  --api-key "$VLLM_API_KEY"

Both expose the OpenAI chat API, so client code only changes its base URL and key:

from openai import OpenAI

ollama = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")  # key required by the client, ignored by Ollama
vllm = OpenAI(base_url="http://localhost:8000/v1/", api_key="<your VLLM_API_KEY>")

Operations and monitoring

Day-two operations differ as much as performance does.

Monitoring. vLLM serves Prometheus metrics on the same port as its API, at /metrics. Requests running and requests waiting (vllm:num_requests_running, vllm:num_requests_waiting) are the two signals to alert on: a growing queue means you need another GPU or a lower --max-num-seqs. Metric names change between releases, so check them against your installed version before building dashboards.

curl -s http://localhost:8000/metrics | grep -E "num_requests_(running|waiting)"

Ollama has no equivalent metrics endpoint; its visibility comes from the server logs and ollama ps, which shows each loaded model, its memory and whether it runs fully on the GPU.

Upgrades and model changes. Ollama swaps models with a pull and handles several at once, which suits teams that experiment. vLLM runs one model per process, so a model change means a restart, and upgrades deserve a staging test because flags and defaults move between releases (the default memory share rose from 0.9 to 0.92, for example).

Security defaults for regulated deployments

Neither server was designed as a security boundary, and the defaults differ in ways that matter for tax, legal and healthcare data.

• Ollama has no authentication. It binds to 127.0.0.1:11434, which is safe on a workstation. Setting OLLAMA_HOST=0.0.0.0 to share it exposes an unauthenticated API to the whole network. Put a reverse proxy in front that terminates TLS and enforces authentication.
• vLLM's --api-key rejects requests to the OpenAI-compatible endpoints without the key. Still terminate TLS at a proxy and restrict the server to an internal network.
• Log every request with user identity at the proxy layer. Our ISO 42001 audit-logging blueprint shows a write-once pattern for this.
• Neither tool is compliant on its own. HIPAA and SOC 2 obligations attach to how you deploy and govern the system; see the on-prem medical records blueprint for a reference architecture.

Benchmark methodology

Throughput comparisons are only meaningful with the conditions attached, so here is the method we use and the harness itself.

Setup. The same model at comparable precision on both servers (Ollama qwen3:32b, Q4_K_M GGUF; vLLM Qwen/Qwen3-32B-AWQ), on the same GPU, one server running at a time, after a warm-up request.

Metrics.

• Time to first token (TTFT): from sending the request to the first streamed token.
• Per-request decode speed: completion tokens divided by the time after the first token.
• Aggregate throughput: total completion tokens across all concurrent requests, divided by wall time.

Concurrency levels. 1, 4 and 16 simultaneous requests, with a fixed prompt and max_tokens.

Measured results on our reference hardware will be published here once the lab run is complete. Until then, run the harness yourself:

import asyncio
import statistics
import sys
import time

from openai import AsyncOpenAI

PROMPT = "List the main obligations of a data processor under GDPR Article 28 in five bullet points."


async def one_request(client: AsyncOpenAI, model: str) -> tuple[float, int, float]:
    start = time.perf_counter()
    first = None
    chunks = 0
    usage_tokens = None
    stream = await client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": PROMPT}],
        max_tokens=256,
        temperature=0,
        stream=True,
        stream_options={"include_usage": True},
    )
    async for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            first = first or time.perf_counter()
            chunks += 1
        if chunk.usage:
            usage_tokens = chunk.usage.completion_tokens
    end = time.perf_counter()
    tokens = usage_tokens or chunks  # fall back to chunk count if usage isn't reported
    return (first or end) - start, tokens, end - (first or end)


async def run(base_url: str, api_key: str, model: str) -> None:
    client = AsyncOpenAI(base_url=base_url, api_key=api_key)
    await one_request(client, model)  # warm-up, not counted
    for concurrency in (1, 4, 16):
        started = time.perf_counter()
        results = await asyncio.gather(*(one_request(client, model) for _ in range(concurrency)))
        wall = time.perf_counter() - started
        ttft = statistics.median(r[0] for r in results)
        decode = statistics.median(r[1] / r[2] for r in results if r[2] > 0)
        total = sum(r[1] for r in results)
        print(f"concurrency={concurrency:>2}  TTFT p50={ttft:.2f}s  "
              f"decode p50={decode:.1f} tok/s  aggregate={total / wall:.1f} tok/s")


if __name__ == "__main__":
    asyncio.run(run(sys.argv[1], sys.argv[2], sys.argv[3]))

Run it against each server in turn:

pip install openai
python bench.py http://localhost:11434/v1/ ollama qwen3:32b
python bench.py http://localhost:8000/v1/ "$VLLM_API_KEY" Qwen/Qwen3-32B-AWQ

For deeper load testing of vLLM itself, its vllm bench serve command reports TTFT, inter-token latency and throughput percentiles; check the flags for your installed version, as they change between releases. Run each test several times, and restart the server between runs so prefix caching doesn't flatter repeated prompts.

Which should you choose?

Choose Ollama when the priority is simplicity: one-command installs, GGUF models from a large catalogue, Apple silicon and Windows support, several models on one machine, and a small number of users. It is the right default for developer laptops, proofs of concept and departmental tools, as long as it stays on localhost or behind an authenticating proxy.

Choose vLLM when one model must serve many users or applications from shared NVIDIA or AMD GPUs. Its batching and memory management turn the same GPU into far more aggregate throughput, its API key and fixed memory reservation suit a dedicated inference server, and its tensor parallelism scales to multi-GPU nodes.

Because both speak the OpenAI API, the decision is reversible: build against the OpenAI client, prototype on Ollama, and move the shared deployment to vLLM when concurrency grows. Run the harness above on your own hardware before committing to either.

Size the GPU first

Whichever server you pick, the model has to fit. Use the calculator below to check weights and KV cache for your model, quantization and context length, and remember that Ollama multiplies the KV cache by its parallel slots while vLLM fills whatever memory is left.