KV Cache Leaks: What Cloudflare's GLM-5.3 Bug Means for You

Cloudflare just published a post-mortem that every team sending business data to a hosted model should read. For 13 days, from August 27 to September 9, its GLM-5.3 model on Workers AI returned 155 responses that contained other customers' data. The model wasn't the problem. The leak came from the KV cache, the short-term memory an inference server keeps while it generates text.

That detail matters. Most AI risk talk focuses on prompts and models, but this was plain infrastructure: a sizing mismatch, an engine bug and a test suite that only checked status codes. Every request came back HTTP 200, and nobody noticed until a researcher reported it.

TL;DR#

• A KV cache sized smaller than the advertised context window, plus a bug in SGLang's disaggregated mode, let long prompts read stale memory from earlier requests.
• Cloudflare fixed it within a day of the report, and SGLang v0.5.22 now rejects oversized requests before any cache work starts.
• If you call hosted LLMs, test outputs (not just status codes), know your provider's serving stack, and keep sensitive fields out of prompts that don't need them.

What a KV cache is, in plain terms#

When a language model writes a reply, it produces one token at a time. Each new token needs to "look back" at everything before it. Recomputing that history at every step would be painfully slow, so inference engines store intermediate results, called keys and values, for every token already processed. That store is the KV cache.

Think of it as a whiteboard the model writes on while it works. Production servers carve GPU memory into pages, hand them to whichever request needs them, and reuse them once that request finishes. Cloudflare's write-up notes that these pages are reused across requests without being zeroed, which is normal for performance reasons.

Prefill and decode, split apart#

Large deployments often split work in two. Prefill nodes read the prompt and build the cache, then decode nodes take it over the network and generate the answer. This "disaggregated" setup helps throughput, but it adds a handoff, and handoffs are where assumptions break.

What actually went wrong at Cloudflare#

Two separate problems lined up.

The configuration mismatch. GLM-5.3 was deployed with a context window of 1,048,576 tokens, but the prefill KV pool only held 681,280 tokens. The engine sizes the pool from whatever GPU memory remains after loading the weights, and nothing forced it to match the context length. Cloudflare says it had noticed the gap before launch, but its tests checked HTTP codes and streaming, not whether the answers were correct.

The engine bug. In SGLang's disaggregated mode, a prompt larger than the pool was marked as rejected and swapped for a one-token placeholder. The request still went through. Here's the path, simplified from the post-mortem:

long prompt (> 681,280 tokens)
   -> prefill: "rejected", replaced with 1-token placeholder
   -> admission queue checks "finished?" not "rejected?"  -> admitted
   -> 1 KV page transferred to decode
   -> decode reserves 8 pages, writes 1, attends over all 8
   -> 7 stale pages from earlier requests leak into the answer
   -> HTTP 200 + a fluent reply with someone else's data

The researcher, Gadi Evron of Knostic, reported it through HackerOne on September 8. Cloudflare reproduced it with two accounts in 25 minutes, took the model private, and had a patched deployment back online that evening. According to AI Weekly, Cloudflare found no evidence of deliberate exploitation.

Why it matters even if you never touched GLM-5.3#

Only one model was affected, and only for very long prompts. The failure pattern is still general. Cloudflare's own note says the same bug had been reported upstream as SGLang issue #30233 in July. Any provider or in-house team running the same engine mode with a pool smaller than the context window could hit the same path. AI Weekly called it the third security story that week where the leak came from inference infrastructure rather than the model.

The deeper lesson is about monitoring. Your dashboards probably alert on error rates and latency. This bug produced neither. It produced confident, well-formed text that happened to belong to another tenant. As the DEV write-up on the incident puts it, zero errors is not zero leaks.

For businesses, the practical exposure sits in long-context jobs: pasting whole contracts, ticket histories, CRM exports or codebases into a single prompt. Those are exactly the workloads people push to the biggest context windows.

Technical trade-offs worth understanding#

Shared inference isn't unsafe by default. Isolation is a property of the serving stack, and stacks differ.

The checks Cloudflare added#

The fixes are instructive. The prefill pool was resized to about 2.04 million tokens, roughly twice the context length. SGLang now retires rejected requests at the admission queue before any allocation or transfer. Cloudflare also added deployment checks that compare KV pool size against context length from the running engine, and SGLang v0.5.22 adds per-transfer KV checksums. A prototype that tagged pages on reallocation cost under 1% in throughput, which shows integrity checks don't have to be expensive.

What to do this week: a checklist#

You don't need to become an inference engineer. You need better questions and a few cheap tests.

1. List every place you send data to a model. Include automations in CRMs and support tools, not just your own code.
2. Ask each provider about their serving stack. Which engine and version? Do they run disaggregated prefill and decode? Were they on an SGLang build older than v0.5.22?
3. Run a canary test. Plant a unique string in one request, send several unrelated requests from a separate session, and scan the replies for the string. Repeat after provider changes.
4. If you self-host on SGLang, upgrade to v0.5.22 or later and enable KV transfer checksums and deferred page release where supported.
5. Assert pool size against context length in your deployment pipeline. Fail the deploy if the pool is smaller.
6. Validate output, not only status. Add schema checks and sanity checks on model responses, especially in automated pipelines where no human reads the text.
7. Shrink what you send. Use retrieval or chunking so a single call rarely carries a whole customer file. Strip fields like card numbers or health data that the task doesn't need.

How MagicMakers Lab approaches this#

When we embed AI into an existing stack, we treat the model as an untrusted component. Outputs get validated before they touch a CRM, billing system or customer message, and prompts carry only the fields the task needs. It's the same habit behind Framico, where 200+ orders ship a day with zero manual steps: you only take people out of the loop once checks exist that do their job.

Key takeaways#

• The Cloudflare leak was an infrastructure bug in KV cache handling, not a model failure.
• A KV pool smaller than the context window plus a missed rejection check let stale memory leak across tenants.
• Error-rate monitoring can't catch this class of bug, so test the content of responses.
• Ask providers which inference engine and version they run, and run your own canary tests.
• Sending less data per prompt is the cheapest protection you control.

FAQ#

What is KV cache in an LLM?#

The KV cache stores the keys and values a transformer computes for each token it has already processed, so it doesn't recompute them for every new token. It makes LLM inference much faster, especially for long prompts. In production it lives in GPU memory pages that are shared and reused across requests, which is why correct isolation between requests matters.

Did the Cloudflare KV cache leak affect other models?#

Cloudflare says no. Only the GLM-5.3 model on Workers AI was affected, and only responses to prompts longer than 681,280 tokens between August 27 and September 9, 2026. Its other disaggregated deployments had KV pools larger than their context windows, so they couldn't reach the faulty code path. Cloudflare reported 155 affected responses.

How can I test if my LLM provider leaks data between requests?#

Use a canary test. Put a unique random string in one request, then send several unrelated requests from a different session and check whether the string appears in any reply. A clean result isn't proof of safety, but repeating the test after provider updates and in your CI pipeline catches regressions that error monitoring would miss.

Is self-hosting an LLM safer than using an API?#

Not automatically. Self-hosting removes other tenants from your GPUs, but you inherit the same engine bugs and become responsible for patching, sizing and testing. For many businesses, a reputable hosted API plus output validation, smaller prompts and canary tests is the better balance. Self-hosting makes sense when data rules or volume justify the operating cost.

If you're piping customer data into AI tools and aren't sure what your providers run or how your outputs get checked, a short review usually surfaces the gaps quickly. Book a free audit.

Sources#

• Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang (Cloudflare)
• Cloudflare Says KV Cache Bug Leaked 155 GLM-5.3 Responses (AI Weekly)
• 155 responses leaked between AI customers: the GLM-5.3 KV cache bug and the canary test for your provider (DEV Community)
• Everything That Happened in AI Today, October 28 (The Neuron)