Serving vLLM Behind an OpenAI-Compatible API

Most junior engineers assume the hard part of building an AI product is "calling the model." In practice, the hard part is serving it: building an API clients can actually hit, handling streaming, batching requests efficiently, managing GPU memory, scheduling work, and staying compatible with the tools people already use — like the OpenAI SDK.

That's the problem vLLM solves. It lets you run Qwen, Llama, or any other supported open-source model behind an API that looks and behaves like OpenAI's, so your application can call it almost the same way it calls api.openai.com.

Here's what we're aiming for:

python
Copy
1client = OpenAI(
2    base_url="http://localhost:8000/v1",
3    api_key="dummy"
4)

And then:

python
Copy
1response = client.chat.completions.create(
2    model="Qwen/Qwen2.5-7B-Instruct",
3    messages=[
4        {"role": "user", "content": "Explain vLLM simply"}
5    ],
6    stream=True
7)

The client shouldn't care whether it's talking to OpenAI, Qwen, Llama, or a model running on our own GPUs. That's the whole point of an OpenAI-compatible serving layer — swap the base_url, keep everything else.

Copy
AI App / SDK / curl
        |
        | POST /v1/chat/completions
        | { model, messages, stream: true }
        v
vLLM OpenAI-Compatible Server
        |
        v
FastAPI / Uvicorn API Layer
        |
        v
AsyncLLMEngine
        |
        v
Scheduler + Continuous Batching
        |
        v
PagedAttention KV Cache
        |
        v
Model Weights: Qwen / Llama
        |
        v
GPU Inference
        |
        v
SSE Streaming Response
        |
        v
Client receives tokens

From the outside, it looks like the OpenAI API. Underneath, there's a lot more going on.

Most AI apps today are already built around OpenAI's API shape — /v1/chat/completions, /v1/completions, /v1/models. If your self-hosted server speaks the same contract, you don't need to rewrite your application to use it. You just change one line:

python
Copy
1from openai import OpenAI
2
3client = OpenAI(
4    base_url="http://localhost:8000/v1",
5    api_key="dummy"
6)

Now the same SDK your app already uses talks to your own infrastructure instead. In practice, that means you can switch between OpenAI and self-hosted models without touching the frontend or backend, run inference entirely inside your own infra for privacy or cost reasons, and try out Qwen or Llama without ripping out existing code.

Worth being clear about one thing, though: OpenAI-compatible doesn't mean it is OpenAI. It means the server follows the same API contract — the actual model, latency, and quality are entirely different, and that's on you to manage.

For local development, Docker Compose is the path of least resistance:

yaml
Copy
1services:
2  vllm:
3    image: vllm/vllm-openai:latest
4    container_name: vllm-openai
5    runtime: nvidia
6    ports:
7      - "8000:8000"
8    volumes:
9      - ~/.cache/huggingface:/root/.cache/huggingface
10    environment:
11      - HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}
12    command: >
13      --model Qwen/Qwen2.5-7B-Instruct
14      --host 0.0.0.0
15      --port 8000
16      --dtype auto

bash
Copy
1docker compose up

Once that's running, you've got an OpenAI-compatible endpoint at http://localhost:8000/v1 — something a backend, frontend, CLI tool, or test script can call exactly like a normal OpenAI endpoint.

The quickest sanity check is a curl request:

bash
Copy
1curl http://localhost:8000/v1/chat/completions \
2  -H "Content-Type: application/json" \
3  -d '{
4    "model": "Qwen/Qwen2.5-7B-Instruct",
5    "messages": [
6      {
7        "role": "user",
8        "content": "Explain vLLM in simple words"
9      }
10    ],
11    "stream": true
12  }'

With stream: true, the server doesn't wait for the full answer before responding — it streams tokens back as they're generated, as Server-Sent Events:

Copy
data: {"choices":[{"delta":{"content":"vLLM"}}]}
data: {"choices":[{"delta":{"content":" is"}}]}
data: {"choices":[{"delta":{"content":" an"}}]}
...
data: [DONE]

If you've built against OpenAI's streaming API before, this format will look familiar.

When a request hits POST /v1/chat/completions, vLLM doesn't just run it straight on the GPU — that would waste most of the hardware's capacity. Instead, the request passes through a few distinct layers.

This is the HTTP server that receives the request, validates it, and exposes the familiar endpoints (/v1/chat/completions, /v1/completions, /v1/models). It's what makes vLLM OpenAI-compatible in the first place — it takes a request like:

json
Copy
1{
2  "model": "Qwen/Qwen2.5-7B-Instruct",
3  "messages": [
4    {
5      "role": "user",
6      "content": "Hello"
7    }
8  ],
9  "stream": true
10}

and translates it into an internal generation request the engine can actually work with.

This is the core of the system — think of it as the service layer that manages inference asynchronously. It's responsible for the request lifecycle, generation state, streaming output, scheduling, model execution, and cancellation.

That complexity exists for a reason. A typical HTTP request hits a database and returns; an LLM request generates output token by token, over time. The server has to track many of these in-flight generations simultaneously, not just fire off a single call and wait.

This is arguably the biggest reason vLLM performs as well as it does. A naive inference server processes requests more or less like this:

Copy
Request A -> generate full answer
Request B -> wait
Request C -> wait

That's a lot of idle GPU time. vLLM instead uses continuous batching, which lets multiple active requests generate together, one decoding step at a time:

Copy
Step 1:
Request A generates token 1
Request B generates token 1
Request C generates token 1

Step 2:
Request A generates token 2
Request B generates token 2
Request C generates token 2

New requests can join the batch mid-flight, even while older ones are still generating — a meaningful difference from static batching, which groups requests up front, runs them together, and only then moves on. For LLM serving specifically, batching isn't a nice-to-have optimization; it's a large part of what determines throughput and cost per token.

During generation, the model needs to remember everything it's already produced — that context lives in the KV (key/value) cache. Without it, the model would recompute the entire prior context on every single token, which is far too slow to be usable.

The problem is that KV cache memory adds up fast, especially with many concurrent users, and naive allocation wastes a lot of it. vLLM's answer is PagedAttention: rather than reserving one large contiguous block of GPU memory per request, it splits the KV cache into smaller blocks — much like how an operating system manages memory pages.

The payoff is fairly direct: more concurrent requests fit on the same GPU, utilization goes up, less VRAM sits idle, and overall throughput improves. Since GPU memory is the expensive, limited resource in this whole system, that efficiency gain matters more than almost anything else in the stack.

Say a user sends:

json
Copy
1POST /v1/chat/completions
2{
3  "model": "Qwen/Qwen2.5-7B-Instruct",
4  "messages": [
5    {
6      "role": "user",
7      "content": "What is Kubernetes?"
8    }
9  ],
10  "stream": true
11}

Here's what happens, roughly in order:

From where the client sits, all of that collapses into a stream of tokens. Underneath, vLLM is juggling batching, memory, scheduling, and GPU execution the entire time.

For a chat product, streaming isn't a nice-to-have — it's close to mandatory. Without it, users stare at a blank screen until the entire response is ready, which feels slow even when total generation time is identical. Streaming gets output in front of them immediately, and that alone makes the product feel dramatically faster, even though the underlying compute cost hasn't changed at all.

Copy
data: {"choices":[{"delta":{"content":"Kubernetes"}}]}
data: {"choices":[{"delta":{"content":" is"}}]}
data: {"choices":[{"delta":{"content":" a"}}]}
data: {"choices":[{"delta":{"content":" container"}}]}
data: {"choices":[{"delta":{"content":" orchestration"}}]}
data: [DONE]

Frontend teams need to actually handle streamed chunks rather than waiting for the full payload, and backend teams need to make sure nothing upstream — a proxy, a gateway, middleware — buffers the response before it reaches the client.

python
Copy
1from openai import OpenAI
2
3client = OpenAI(
4    base_url="http://localhost:8000/v1",
5    api_key="dummy"
6)
7
8stream = client.chat.completions.create(
9    model="Qwen/Qwen2.5-7B-Instruct",
10    messages=[
11        {
12            "role": "system",
13            "content": "You are a helpful senior engineer."
14        },
15        {
16            "role": "user",
17            "content": "Explain Docker Compose in simple words."
18        }
19    ],
20    stream=True
21)
22
23for chunk in stream:
24    delta = chunk.choices[0].delta
25    if delta.content:
26        print(delta.content, end="", flush=True)

The line doing all the work is base_url="http://localhost:8000/v1" — that's what redirects the OpenAI SDK to your own vLLM server instead of OpenAI's.

Don't think of vLLM as "a model runner." It's an inference server, and the distinction matters. A model runner loads weights and generates text. An inference server has to survive real traffic — which means HTTP handling, streaming, request scheduling, batching, GPU execution, memory management, error handling, metrics, and a story for scaling. vLLM gives you a genuinely strong foundation for all of that, but it's still just the foundation.

Running vLLM locally is easy. Running it in production is a different exercise entirely, and it means thinking through: GPU type and VRAM size, model size, max context length, batch size, request and stream timeouts, authentication, rate limiting, logging, metrics, autoscaling, health checks, load balancing, and model warmup.

A more realistic production setup puts a gateway in front of vLLM rather than exposing it directly:

Copy
Client
  |
Cloudflare / API Gateway
  |
Backend Gateway
  |
vLLM Server
  |
GPU Node

The gateway handles auth, rate limiting, tenant checks, billing, logging, request validation, and model routing. vLLM's job is inference — not business logic — and it's worth keeping that boundary clean.

Treating vLLM like OpenAI. The API surface looks the same, but you're now responsible for the infrastructure OpenAI normally handles for you — scaling, uptime, GPU provisioning, billing, safety tooling, all of it.

Ignoring GPU memory. LLM serving is, more than anything else, a memory management problem. Push the context length or concurrent request count too high without headroom, and the server runs out of VRAM. Keep an eye on usage.

Skipping rate limiting. One user sending requests aggressively can monopolize the GPU for everyone else. Rate limits belong at the gateway layer, not as an afterthought.

Not testing streaming end to end. A non-streaming request working fine tells you very little — streaming can break silently through proxies, Cloudflare, Nginx, or frontend code that buffers responses. Test the actual streaming path, not just the happy-path response.

Defaulting to the biggest model available. Bigger isn't automatically better. A smaller model with noticeably better latency often wins on actual product experience. Before committing, compare quality, latency, cost per request, tokens per second, GPU memory footprint, and concurrency limits — together, not in isolation.

Once that mapping clicks, the rest of the architecture stops feeling mysterious.

vLLM matters because it lets you serve open-source models like Qwen and Llama behind an API that speaks OpenAI's language — same SDK, same /v1/chat/completions shape, same behavior your app already expects. Underneath that familiar surface, it's doing real engineering: continuous batching, PagedAttention, KV cache management, async request handling, and GPU-efficient execution.

Calling an LLM is easy. Serving one efficiently is engineering work. vLLM gives you the serving layer — the job that's left is wrapping it in the right gateway, auth, rate limits, and monitoring to make it production-ready.