Ollama, llama.cpp, or vLLM for This Workload?
A team is building an internal support-ticket triage feature that will serve roughly 200 concurrent employees during business hours, each sending occasional requests, on infrastructure the team fully controls. Someone proposes Ollama because "it's simple and we already use it on our laptops for testing."
- Is Ollama a reasonable choice for this production workload? Explain what specifically would or wouldn't scale.
- Which tool from this subject's stack is the better fit for this workload, and name the two specific techniques that make it a better fit.
- Would your answer change if this were instead a single internal tool used by one engineer at a time? Explain why or why not.
1. Is Ollama reasonable here?
Not for this workload as described. Ollama is built around the single-user local convenience use case — it wraps llama.cpp-family backends without the request-scheduling machinery needed to serve many concurrent requests efficiently. At 200 concurrent employees making occasional but real requests, a naive request-at-a-time (or simple static-batch) approach underutilizes the GPU badly during decode, since decode is memory-bandwidth-bound and batching multiple requests together is exactly what amortizes that bandwidth cost across more useful output — a benefit Ollama's design doesn't center the way a production serving engine does. It would likely work at low concurrency but degrade in latency and throughput as concurrent load actually materializes.
2. The better-fit tool and why
vLLM is the better fit, specifically because of continuous batching (which keeps the GPU's batch dimension full by immediately backfilling a freed slot from a finished request with a new waiting request, rather than waiting for an entire static batch to finish) and PagedAttention (which allocates KV cache memory in small, fixed-size, non-contiguous blocks on demand rather than pre-allocating a worst-case contiguous block per request) — both of which are exactly the mechanisms that let a serving stack sustain much higher throughput per GPU under real concurrent load than a single-user-oriented runner, which is precisely the workload shape described (many concurrent, if individually occasional, requests).
3. Would the answer change for a single engineer at a time?
Yes — for a single-user, one-request-at-a-time internal tool, Ollama (or LM Studio) is the better choice, not just an acceptable one: vLLM's continuous batching and PagedAttention have nothing to batch across or page-manage efficiently when there's only ever one request in flight, so its added operational complexity (multi-GPU awareness, production-serving configuration) buys nothing at that concurrency level. This is the general rule from the subject: the dividing line is concurrent request volume, not raw workload importance — reaching for vLLM at concurrency of one is over-engineering in the same way reaching for Ollama at 200 concurrent users is under-engineering.
Share this question