Self-Hosted LLM for Sensitive Data: Hosting Is Not Security

On 9 November 2026 I tested vLLM v0.33.0's own auth middleware with an API key set. A POST to /invocations with no key got HTTP 200. The same request to /v1/chat/completions got 401. A self-hosted LLM for sensitive data keeps the data on hardware you own, but owning the hardware does not decide which routes answer without a key.

In the same test, an nginx allowlist in front turned the unkeyed /invocations and /metrics requests into 404s.

TL;DR: vLLM's --api-key guards only paths under /v1, /v2, /inference and /cohere. /invocations, /metrics and operational routes such as /pause and /update_weights answer without a key. Running on your own H100s or H200s does not change that. A reverse proxy that allowlists routes does. My test ran vLLM's middleware, not a full server on GPUs.

What platform is this self-hosted LLM post written from?

The platform was an on-prem AI system for pharma and medical work that ran evaluations over big data, and its data was too sensitive to send to a cloud model API. The team procured on-prem hardware: NVIDIA H100s first, then an upgrade to H200s. vLLM served the models. I was Lead Engineer, DevSecOps Architect and Team Lead, and I managed security and auditing.

The platform's stack was fine-tuned self-hosted GPT-OSS, Llama and Qwen, a custom chat and evaluation engine, Vue, FastAPI, self-hosted Supabase and scraping microservices, with RAG and complex math over huge datasets. The model trade-offs are in fine-tuning versus RAG for a self-hosted LLM.

The platform's own controls stay under NDA, and this post does not say whether it used --api-key or a proxy. Every statement about controls below comes from vLLM's documentation, vLLM's source, a published advisory, or the test I ran on 9 November 2026.

What happens when you call vLLM without its API key?

In my test, with a key set, an unkeyed request to /v1/chat/completions returned 401 and an unkeyed request to /invocations returned 200. I ran vLLM v0.33.0's authenticate.py unmodified (sha256 starting ceb576aa) in front of two stand-in routes that share one handler, the way /v1/chat/completions and /invocations share inference in vLLM.

The test covers the middleware, not a full vLLM server. My test host had no GPU and 8 GB of disk free, so I did not install vLLM. A stand-in handler returns a fixed string, so a 200 means the middleware let the request through. The environment was Python 3.12.3, starlette 0.48.0, httpx 0.28.1 and uvicorn 0.38.0, and s3cret-key is a placeholder.

async def infer(request):
    return JSONResponse({"output": "model ran"})

app = Starlette(routes=[
    Route("/v1/chat/completions", infer, methods=["POST"]),
    Route("/invocations", infer, methods=["POST"]),
    Route("/metrics", lambda r: JSONResponse({"metrics": "served"})),
])
app.add_middleware(AuthenticationMiddleware, tokens=["s3cret-key"])

POST /v1/chat/completions   key=no  -> 401 {"error": "Unauthorized"}
POST /v1/chat/completions   key=yes -> 200 {"output": "model ran"}
POST /invocations           key=no  -> 200 {"output": "model ran"}
GET  /metrics               key=no  -> 200 {"metrics": "served"}

The POST /invocations request is the result that matters: a route sharing a handler with the protected one answered a caller with no key. The /metrics route stands in for vLLM's /metrics, which the documentation lists as unauthenticated. None of this is a bug report. vLLM documents the behaviour, and the test shows what the documentation means in four requests.

Why does a self-hosted LLM need more than an API key?

vLLM's API key check matches a path prefix, not a list of routes, so a route outside its four prefixes answers without a key. In v0.33.0's authenticate.py, line 11 sets the guarded prefixes, and a request is challenged only if its path starts with one of them. The path /invocations starts with none of them.

GUARDED_PREFIX = ("/v1", "/v2", "/inference", "/cohere")

The module's docstring says authentication is skipped when "The request path doesn't start with GUARDED_PREFIX (e.g. /health)." Line 59 is the check itself:

if url_path.startswith(GUARDED_PREFIX) and not self.verify_token(headers):

A failing check returns 401 with {"error": "Unauthorized"}, the response my unkeyed /v1/chat/completions request got. In the same tag, sagemaker/api_router.py registers @router.post("/invocations", ...), validates the body against the inference request types and calls the matching endpoint.

vLLM's documentation says the same in prose: the flag works "only for endpoints under the /v1, /v2, /inference, and /cohere path prefixes. Many other sensitive endpoints are exposed on the same HTTP server without any authentication enforcement." It adds: "Do not rely exclusively on --api-key for securing access to vLLM." On /invocations specifically: "particularly concerning as it provides unauthenticated access to the same inference capabilities as the protected /v1 endpoints."

vLLM's documentation lists these routes as unauthenticated even with --api-key set:

• Inference-style routes: /invocations (a "SageMaker-compatible endpoint (routes to the same inference functions as /v1 endpoints)"), /generative_scoring, /pooling, /classify, /score and /rerank.
• Operational routes: /pause, /resume, /abort_requests, /scale_elastic_ep and /update_weights. The documentation says /update_weights "can alter model behavior".
• Information routes: /metrics ("Prometheus metrics, including model, throughput, queue, and KV-cache statistics"), /tokenize, /detokenize, /health, /ping, /version and /load.

The documentation says an attacker who can reach the server can "Cause denial of service by calling /pause, /scale_elastic_ep, or /abort_requests without a token".

The key check has also been bypassed outright on the guarded routes. CVE-2026-48746 (GHSA-94f4-hr76-p5j6, CVSS 9.1), found in an X41 D-Sec source code audit, affected vLLM from 0.3.0 until 0.22.0: the old middleware read the path from a URL that starlette rebuilds with the Host header, so a / or ? in that header could change the path the check saw. The advisory says "Instances behind an RFC-conforming web server (such as nginx) are not affected." In v0.33.0 the middleware reads scope["path"] instead.

How do you put an allowlist in front of vLLM?

A reverse proxy in front of vLLM should name the routes users may reach and return 404 for everything else. vLLM's documentation recommends nginx, Envoy or a Kubernetes Gateway for this. For the test I used nginx 1.29.8 with the config below, which lets through only POST /v1/chat/completions to the middleware on 127.0.0.1.

# Only the routes users need reach vLLM. Everything else stops here.
server {
    listen 127.0.0.1:18080;

    location = /v1/chat/completions {
        limit_except POST { deny all; }
        proxy_pass http://127.0.0.1:18000;
    }

    location / {
        return 404;
    }
}

The same four requests through the proxy (on the last two rows, 404 is the status and 404 Not Found is nginx's body):

POST /v1/chat/completions   key=no  -> 401 {"error":"Unauthorized"}
POST /v1/chat/completions   key=yes -> 200 {"output":"model ran"}
POST /invocations           key=no  -> 404 404 Not Found
GET  /metrics               key=no  -> 404 404 Not Found

For the test I chose an allowlist over a denylist, because a denylist means naming every route above and every route a later version adds. An allowlist names the one route users need, and anything I forgot is closed by default. The 401 on the unkeyed chat request still comes from vLLM's middleware: the proxy decides which routes exist, and the key decides who may call them.

How do you secure a self-hosted LLM for sensitive data?

vLLM's security documentation gives six checks for a deployment that serves sensitive data, and I tested only the first, in the 9 November test above. The other five are the documentation's advice, not my results.

1. Allowlist routes at a reverse proxy. The documentation asks for a proxy that "Explicitly allowlists only the endpoints you want to expose to end users" and blocks the rest, "including the unauthenticated inference and operational control endpoints".
2. Firewall everything except the API port: "Block all incoming connections except to the TCP port the API server is listening on." The torch.distributed TCPStore, the documentation notes, "by default, listens on all network interfaces".
3. Isolate multi-node traffic: "All communications between nodes in a multi-node vLLM deployment are insecure by default and must be protected by placing the nodes on an isolated network."
4. Never set VLLM_SERVER_DEV_MODE=1 in production. The documentation marks this "CRITICAL", because it exposes /collective_rpc, which can "Execute arbitrary RPC methods on the engine (extremely dangerous)".
5. Cap VLLM_MAX_N_SEQUENCES, the upper bound on the n parameter (how many output sequences one request asks for). The default is 16384, and the documentation suggests 64 or 128 for public-facing deployments.
6. Add authentication, rate limiting and logging at the proxy layer, which the documentation recommends alongside the allowlist.

My own addition is to repeat the four-request test after every vLLM upgrade, because the routes that answer without a key belong to the vLLM version.

Do H200s instead of H100s change the data boundary?

No. A GPU upgrade changes how much memory a card has, not which routes answer without a key. NVIDIA describes the H200 as "the first GPU to offer 141 gigabytes (GB) of HBM3e memory at 4.8 terabytes per second (TB/s)", which it puts at "nearly double the capacity of the NVIDIA H100 GPU with 1.4X more memory bandwidth." Those are NVIDIA's figures, not measurements from the platform.

The ownership argument is in local LLM versus cloud, and who idle RAM serves, and where regulated work should run is in hybrid cloud for regulated workloads.

What does this vLLM test not prove?

The test proves that vLLM v0.33.0's auth middleware lets unkeyed requests to /invocations and /metrics through when a key is set, and that an nginx allowlist stops them. It does not prove the behaviour of a full vLLM server.

• The stand-in handler ignores the request body. The real /invocations validates it, and I did not run that code.
• The docs page is /en/latest, read 9 November 2026. The source is tag v0.33.0, the latest stable on PyPI that day. They may not match route for route.
• I sent the same four requests twice, once direct and once through nginx. I did not test path normalisation differences between nginx and starlette, such as double slashes or percent-encoding.
• The nginx config is minimal: no /v1/models, no health route, no TLS, no rate limiting, no logging.
• The platform's own controls are under NDA, so this post says nothing about what it had in place.

Where the server sits tells you who owns the hardware. The proxy config tells you who can ask it questions. More of this work is in the Self-Hosted AI field notes.

If you run vLLM with --api-key, which routes answer on your server without one, and what checked that after your last upgrade?

References

• vLLM documentation, Security: the --api-key scope, the unauthenticated endpoint list, the reverse proxy, firewall, multi-node, VLLM_SERVER_DEV_MODE and VLLM_MAX_N_SEQUENCES guidance. Read 9 November 2026.
• vLLM v0.33.0, vllm/entrypoints/serve/middleware/authenticate.py: GUARDED_PREFIX on line 11 and the prefix check on line 59. Read 9 November 2026.
• vLLM v0.33.0, vllm/entrypoints/serve/sagemaker/api_router.py: the /invocations route registration. Read 9 November 2026.
• GHSA-94f4-hr76-p5j6, vLLM: OpenAI auth bypass (CVE-2026-48746): affected versions, the Host header mechanism and the nginx note. Read 9 November 2026.
• NVIDIA H200 Tensor Core GPU: 141 GB HBM3e at 4.8 TB/s, compared with the H100. Read 9 November 2026.
• The test itself, run 9 November 2026: the stand-in routes, the nginx config and both outputs are quoted in full above.