But LLM traffic has structure that fixed algorithms can’t see. Conversations continue. Conversations branch. The same system prompt fans out into a thousand different user queries, and every one of them shares thousands of tokens of prefill with requests that came before. A load balancer that remembers where it sent those earlier requests can route the later ones to the GPU that already has the work cached.

This post is about what “remembering” actually looks like: how a routing table learns from traffic, and why one route per request isn’t enough. The second half is about the trade-offs that show up once your load balancer’s memory becomes a data structure you have to manage.

The Shape of Conversational Traffic

Consider a typical multi-turn chat request. By turn three, the request body contains the entire conversation so far:

[system: 256 tokens][user1: 50][assistant1: 100][user2: 40][assistant2: 80][user3: 30]

Two properties make this traffic special:

Conversations grow by appending. Turn N+1 contains turn N as an exact token prefix. The KV cache computed for turn N is reusable for turn N+1—if the request lands on the same GPU.

Conversations branch from shared prefixes. A hundred users hitting the same RAG application send a hundred different questions behind the same 256-token system prompt. Those requests share a prefix with each other even though no two are identical.

A learning router exploits both properties. After it forwards a request and sees a successful response, it records: this token sequence lives on that backend. The next request that starts with the same tokens routes to the same place, and the GPU skips the prefill it already did.

The question is: which token sequence do you record?

The Naive Answer Fails Quietly

The obvious approach is to record one route per request: hash or store the first N tokens (say, 128) and map them to the backend that served them.

This works for exact repetition and fails for everything else. Watch what happens with two requests that share a system prompt:

Request 1: [system: 256 tokens][user: "Summarize this contract"]
Request 2: [system: 256 tokens][user: "What's the termination clause?"]

If your route key is “the first 128 tokens,” both requests match. Great. But if your route key is the full request prefix, or a fixed length that straddles the user message, the different user queries produce different keys. Request 2 misses the route that Request 1 learned, falls back to hash routing, and the hash, computed over tokens that include the unique user query, sends it to a different backend. The system prompt’s KV cache sits warm on Backend 1 while Backend 2 recomputes it from scratch.

A fixed-length route key is always wrong for someone. Too short, and you can’t distinguish conversations that diverge after the system prompt. Too long, and you can’t match conversations that diverge before your key ends. The right boundary isn’t a length: it’s wherever the messages end, and that’s different for every request.

Learning at Every Boundary

The fix is to stop learning one route per request and start learning one route per message boundary. When a request succeeds, the router stores a route at every point where a message ends:

[system: 256][user1: 50][assistant1: 100][user2: 40]
            ↑          ↑                ↑          ↑
       route @256  route @306      route @406  route @446

                    (all → Backend 1)

One successful request produces four routes instead of one. Each route is a prefix of the next. Now look at what the router can match:

A branching conversation. Different user query, same system prompt:

[system: 256][user1': "different question"]
  → longest prefix match: route @256 → Backend 1 ✓

A continuing conversation. The next turn of the original thread:

[system: 256][user1: 50][assistant1: 100][user2: 40][assistant2: 80][user3: 30]
  → longest prefix match: route @446 → Backend 1 ✓

A conversation resumed from the middle. A client that truncated history:

[system: 256][user1: 50][assistant1: 100][new user turn]
  → longest prefix match: route @406 → Backend 1 ✓

Every one of these routes to the backend with the warm cache, and every one would have been a miss under single-route learning. A ten-turn conversation creates ten opportunities to match instead of one.

The data structure that makes this practical is a radix tree with longest-prefix-match semantics. The lookup doesn’t ask “is this exact sequence in the tree?” It asks “what is the deepest stored route that prefixes this request?” Storing routes at multiple depths costs nothing extra at lookup time: the traversal is O(L) in the prefix length either way, and it naturally passes through every stored boundary on its way down.

Where the Boundaries Come From

Learning at message boundaries requires knowing where the messages are, in token space rather than character space. The request JSON tells you where each message ends as a character offset; the routing tree needs token positions. Two strategies, tried in order:

Marker scan. Chat templates like llama3 and chatml inject a distinct token at the start of every message (<|start_header_id|>, <|im_start|>). Scan the token array for those markers and you have exact boundaries. O(n) in token count, precise, but only works when the template uses single-token markers.

Proportional estimation. For templates without scannable markers, map each message’s character offset to a token position using the overall token-per-character ratio. Accuracy is ±2-5 tokens. That sounds sloppy for a routing key, but it isn’t. The next section is why.

If both strategies fail, the router falls back to a fixed prefix length. Everything still works; you just get single-depth behavior instead of multi-depth.

Block Alignment Forgives Small Errors

vLLM’s PagedAttention manages the KV cache in fixed-size blocks, 16 tokens by default. Cache reuse happens at block granularity: two requests that agree on the first 240 tokens and differ at token 250 share exactly 15 blocks of cache, because tokens 240-255 form a partial block that can’t be reused.

So the router aligns every learned route down to a block boundary before storing it. A boundary detected at token 306 is stored as a route at token 304. This has two useful consequences:

Estimation error stops mattering. A boundary that’s off by ±5 tokens usually lands in the same 16-token block as the true boundary, and aligns to the same route. The routing key matches the granularity of the thing it’s actually predicting (cache blocks) rather than pretending to a precision the cache can’t use.

Nearby boundaries collapse. Boundaries at tokens 256, 260, and 262 all align to 256 and deduplicate into a single route. Short messages (a “yes,” an emoji, a one-line tool result) don’t bloat the tree with routes that could never produce distinct cache hits anyway.

The Cost: Your Routing Table Is Now a Cache Too

Multi-depth learning multiplies route count by the average messages per conversation. A tree sized for 100,000 routes holds 100,000 conversations under single-depth learning, but only ~10,000 ten-message conversations under multi-depth. The router’s memory is now itself a cache with an eviction policy, sitting in front of the GPU cache it’s trying to model.

Three mechanisms keep it bounded:

LRU eviction. Every route lives on an intrusive LRU list. When the tree hits its cap, the least-recently-matched route is evicted in O(1) via a tail pointer. Active conversations stay hot; abandoned ones age out. Note what this means for multi-depth: an idle conversation’s deepest routes expire first, while its system-prompt route survives because other traffic is still matching it. The tree automatically keeps the prefixes that are earning their memory.

Trust-ranked eviction. In a cluster, routes learned from your own traffic are tagged LOCAL; routes learned from peer gossip are tagged REMOTE. Under capacity pressure, REMOTE routes evict first. Your own observations are ground truth; a peer’s observations are secondhand and possibly stale.

TTL expiry. Routes older than an hour (configurable) expire regardless of position. A route’s claim—”this prefix is cached on that GPU”—decays with time, because the GPU’s own cache evicted it long ago.

The practical guidance: when you turn on multi-depth learning, scale your route capacity by your expected messages per conversation, or accept that eviction starts sooner. Watch the route-count metric. The failure mode is gentle (evicted routes mean hash fallback rather than errors), but it silently caps your cache hit rate.

What Learning Doesn’t Fix

A learned route is a prediction. The router believes the prefix is cached on that backend; the backend’s own memory pressure decides whether it still is. Three honest limitations:

The router can over-commit a backend. If 80% of traffic shares one system prompt, pure prefix affinity sends 80% of traffic to one GPU. A learning router needs a disobedience mechanism: when a backend’s in-flight count exceeds twice the median, divert to the least-loaded backend and eat the cache miss. In our benchmarks, that trade costs about 5 points of cache hit rate and buys a 45% improvement in P99 latency. Affinity is a preference the router has to be willing to break.

Learned state must propagate. In a cluster, a route learned on node A is useless to node B until gossip delivers it. Until then, B’s consistent-hash fallback sends same-prefix requests to a deterministic backend, so even unlearned traffic converges, just less cleverly.

The tree learns where traffic went, not where it should have gone. If the first request for a prefix lands on a bad backend (by hash), every subsequent request follows it there. Learning amplifies the initial placement, good or bad. Health checks and the load-pressure override are the correctives.

The Takeaway

The interesting shift isn’t the radix tree or the boundary detection. It’s that the load balancer has state that improves with traffic. Request one routes by hash. Request one thousand routes by accumulated knowledge of where every shared prefix in your workload already lives.

That only works because the router never fully trusts its own memory. The tree is capped, routes learned from gossip evict first, everything expires after an hour, and load pressure can override any match. Otherwise the tree fills with routes that outlive the caches they point to, and all that learning is just a memory leak.

We built this as part of Ranvier, a Layer 7 traffic controller for LLM inference. Earlier posts in this series covered why load balancers waste GPUs, what KV cache misses cost, and the tokenization bottleneck that feeds the router its tokens.

Ranvier is a project of Minds Aspire, LLC.

You’re probably not measuring it. Most LLM proxies treat tokenization as instantaneous—call the function, get the tokens, move on. But on an event-loop architecture, 5-13ms isn’t a rounding error. It’s an eternity. Every millisecond your event loop spends inside a tokenizer FFI call is a millisecond where no other request is read, no response is forwarded, no health check is answered, no connection is accepted.

This post is about a bottleneck hiding in the gap between “fast enough” and “actually non-blocking.”

Why Tokenization Blocks

If you’re doing prefix-aware routing, request rewriting, cost estimation, or priority classification, your proxy needs to tokenize the input before forwarding it. That means calling a tokenizer, usually HuggingFace’s tokenizers library, the same BPE implementation used by most serving engines.

The problem is that tokenization is CPU-bound work executed through an FFI boundary. The Rust tokenizers crate does the actual BPE encoding. Your proxy calls it through a C binding. The call takes 5-13ms depending on input length. During that call, your thread is gone.

In a thread-per-request architecture (Go, Java, threaded Python), this is fine. One thread blocks; the others keep working. In an event-loop architecture—Node.js, Seastar, anything built on epoll/io_uring with cooperative scheduling—it’s a disaster. The event loop processes everything sequentially. While it’s inside the tokenizer, it processes nothing else.

Let’s make this concrete. You have an event loop handling 1,000 requests per second. Each tokenization call takes 10ms. If you tokenize synchronously on the event loop, you can process at most 100 tokenizations per second on that core. Your other 900 requests are queued, their latency inflating by 10ms for each request ahead of them in line.

At 20 concurrent users, we measured tokenization accounting for 10.6ms of total routing overhead, while the actual routing decision (a radix tree lookup) took 0.01ms. The tokenizer was 1,000x slower than the thing it was feeding.

The Caching Layer That Actually Works

The first optimization is the most obvious: don’t tokenize the same text twice.

LLM traffic has a property that makes caching extraordinarily effective: repetition. Every request to a RAG application includes the same system prompt. Every multi-turn conversation starts with the same instruction prefix. Every API call from the same client sends the same role tags (<|system|>\n, <|user|>\n).

We added an LRU cache in front of the tokenizer. Hit rates depend entirely on content type, and the spread is dramatic. Here’s what we expect:

That 80-90% hit rate on system messages means that for most requests, the expensive part—tokenizing the 2,000-4,000 token system prompt—is a hash table lookup returning in microseconds instead of a 10ms FFI call.

The implementation is straightforward: a hash map keyed on the input text, with an LRU eviction list capped at a configured maximum (we use 1,000 entries). On hit, move the entry to the front. On miss, tokenize, insert at the front, evict from the tail if full. No locks needed: in a sharded architecture, each core has its own cache.

Two details matter:

Cap cached text—but don’t cap it too low. Our first instinct was to cap at 8KB: surely a long RAG document won’t repeat verbatim often enough to earn its memory. That was a mistake. The long, stable system prefixes we most wanted to cache routinely exceed 8KB, and refusing them reintroduced a 5-7ms P50 regression at 20+ concurrent users, exactly the cost we were trying to delete. We raised the cap to 64KB. Worst case is 1,000 entries × 64KB ≈ 64MB per shard, which is cheap insurance. The cache key is the full input text (the tokens themselves are a small vector of int32s), so the cap is really about bounding key memory. And the texts most worth caching are precisely the long ones. (A separate, tighter 32KB limit applies to the cross-core dispatch path, because that one copies the string across a core boundary, where large copies aren’t free.)

Don’t cache unique content. User queries have a 10-30% hit rate; most are unique. The cache handles this naturally through LRU eviction: unique queries enter the cache, never get hit, and fall off the tail. The system prompt stays hot at the front.

When Caching Isn’t Enough

A 90% cache hit rate sounds great until you think about what happens on the other 10%. At 1,000 requests per second, 10% misses means 100 tokenizer calls per second, each blocking for 10ms. Your event loop can handle exactly 100 of those per second. You’re at capacity with zero headroom. And that’s assuming uniform arrival, which real traffic never is.

A burst of 20 cache misses in a row blocks your event loop for 200ms. Every request that arrives during those 200ms—including the ones that would have been cache hits—waits.

You need a way to tokenize without blocking the event loop.

Option 1: Thread Pool Offload

The most direct solution: move the FFI call to a dedicated worker thread. The event loop submits a job, gets a future back immediately, and continues processing other requests. When the worker thread finishes tokenizing, it signals the event loop to resume the request.

The implementation needs care:

One thread per core. The HuggingFace tokenizer isn’t thread-safe for concurrent calls on the same instance, so you need one tokenizer instance per worker thread. In a sharded architecture, that means one worker thread per shard: no contention, no locks on the tokenizer itself.

Lock-free job queue. The event loop (producer) and worker thread (consumer) communicate through a bounded SPSC (single-producer, single-consumer) queue. No mutexes on the hot path. When the queue is full, the event loop falls back to other strategies rather than blocking.

Memory isolation across thread boundaries. This is the subtle one. If your event loop uses a per-core memory allocator (Seastar does, and so does anything using jemalloc with thread-local arenas), you can’t pass heap-allocated objects from the event loop thread to the worker thread and back without corrupting allocator metadata. The input string must be reallocated on the worker thread before calling the tokenizer. The output tokens must be reallocated on the event loop thread when the result returns. Two copies that feel wasteful but prevent silent memory corruption.

The overhead is ~50-200μs per call—negligible compared to the 5-13ms it keeps off the event loop.

Option 2: Cross-Core Dispatch

If you’re running a multi-core architecture with per-core sharding, you can hand the tokenization to a different core. Instead of tokenizing locally (blocking this core’s event loop), dispatch it elsewhere.

This doesn’t eliminate the blocking: the target core’s event loop still blocks for 5-13ms. But it moves the blocking away from the core that’s serving the request, keeping the request-handling core responsive.

The selection algorithm is a knob, and how far you turn it depends on how often this path fires. Our shipping implementation keeps it as simple as possible: rotate to the next core (round-robin). It costs nothing to compute, and because cross-core dispatch is only a fallback (it fires when the thread pool is saturated, which is rare), even distribution is good enough and load skew rarely has time to matter. Round-robin spreads cost evenly; it does not look at which cores are actually busy.

If your dispatch path fires often enough that even spreading isn’t good enough, the next step up is Power-of-Two-Choices (P2C): sample two cores at random, pick the less loaded one. It’s O(1), avoids thundering herd, and produces near-optimal distribution. We run a P2C balancer elsewhere in the system, and the cross-core tokenization path is wired to adopt it, but today the selector still ignores load and just rotates. The rule of thumb: match the selector’s sophistication to the frequency of the path. Don’t pay for P2C on a path that fires once in a thousand requests.

The cross-core dispatch has its own memory safety requirement: the input text must be copied into an owned string before crossing the core boundary, and the output tokens must be copied again when returning. Same principle as the thread pool—allocator domains don’t mix.

Option 3: Both

The strategies compose. On a cache miss:

1. Try the thread pool — if the SPSC queue has space, submit and continue. Event loop never blocks. Best case.
2. Try cross-core dispatch — if the thread pool is full, hand the work to another core (we rotate round-robin). Calling core stays unblocked. Target core blocks briefly.
3. Local fallback — if everything else fails, tokenize on the local event loop, gated by a semaphore that limits concurrent blocking tokenizations to one per core. This caps the worst case: at most one 5-13ms stall at a time, rather than unbounded stacking.

In practice, the cache handles 80-90% of requests. The thread pool handles most of the rest. Cross-core dispatch and local fallback are rarely needed but prevent pathological behavior under burst traffic.

The Semaphore Matters More Than You Think

That local fallback semaphore deserves a closer look. Without it, a burst of cache misses can compound: five concurrent misses on the same core means five sequential tokenizations, 50-65ms of total blocking. Every other request on that core is frozen for the duration.

A semaphore with one permit ensures at most one blocking tokenization at a time. The second, third, and fourth concurrent misses either wait for the semaphore (adding latency to those specific requests) or bail out and route without tokens (falling back to hash-based routing instead of prefix-aware routing).

The choice between “wait for the semaphore” and “bail out” depends on your architecture. If tokenization is required for correctness (e.g., token counting for billing), wait. If it’s an optimization (e.g., prefix routing), bail out. A request routed by hash instead of prefix is slightly less optimal but doesn’t block the event loop.

Measuring the Problem in Your Stack

Most LLM proxies don’t instrument tokenization latency. Here’s what to look for:

Histogram the tokenization call. Wrap your tokenizer call with a timer. Bucket at 100μs, 500μs, 1ms, 5ms, 10ms, 50ms, 100ms. If you see significant mass above 5ms, you have a blocking problem.

Track cache hit rate. If you have a cache, measure it. Anything above 70% means caching is working. Below 50% means your traffic patterns don’t repeat enough for caching to help, and you need the thread pool or dispatch strategies.

Correlate with tail latency. If your P99 latency spikes correlate with low cache hit periods, tokenization blocking is likely the cause. The correlation is indirect—the blocking doesn’t slow the tokenized request much, but it freezes every other request on that core.

Monitor event loop stalls. If your framework reports reactor stalls or event loop delays (Seastar does, Node.js has monitorEventLoopDelay), check whether they correlate with tokenization activity. A 10ms stall that appears under load and disappears when you reduce traffic is a classic sign of a synchronous call that shouldn’t be synchronous.

When You Don’t Need Any of This

If your proxy doesn’t tokenize, none of this applies. Many LLM proxies are pure HTTP forwarders—they pass the request body through without parsing it. No tokenization, no bottleneck.

You need to tokenize if you’re doing any of:

• Prefix-aware routing (matching token sequences for KV cache locality)
• Token counting (enforcing context window limits before hitting the backend)
• Cost estimation (pricing based on input token count)
• Request priority classification (longer inputs get different priority)
• Request rewriting (injecting or modifying tokens before forwarding)

If you’re doing these in a thread-per-request architecture (Go, Java), the blocking is absorbed by the thread model. The problem is specific to event-loop architectures, and it scales with the number of requests that miss the cache.

The Takeaway

Tokenization is a 5-13ms synchronous operation hiding inside systems that assume everything is asynchronous. At low traffic, it’s invisible. At high traffic, it’s the bottleneck. The tokenizer itself isn’t especially slow; the damage is that it freezes every other request on the core while it runs.

The fix is layered defense: cache the common cases (80-90% of requests), offload the rest to worker threads (event loop never blocks), and fall back to cross-core dispatch when the thread pool is full. Each layer handles a different failure mode. Together, they keep a 5-13ms FFI call from becoming a 200ms tail latency event.

If you’re building an LLM proxy that does anything with tokens before forwarding, wrap your tokenizer call in a histogram before you reach for any of these fixes. You can’t budget for the mass above 5ms until you can see it.

This is the fourth post in a series on LLM infrastructure performance. The first covered why your load balancer is wasting your GPUs. The second covered 24 hard rules for writing correct async C++. The third covered KV cache locality, the hidden variable in your LLM serving cost. The next will cover how routing decisions improve when the load balancer learns from every request it forwards.

Ranvier is a project of Minds Aspire, LLC.

That recomputation takes real time and real money. On a Llama 3.1 70B at half precision, a 4,000-token prefill takes over a second. If eight GPUs each recompute the same system prompt independently because round-robin sent one request to each, you just paid for the same work eight times. Multiply by every request, every hour, every day.

This post is about the cost of that mistake, how to measure it, and what changes when your load balancer understands token locality.

What the KV Cache Actually Saves You

A transformer processes input tokens in two phases. Prefill computes the key-value pairs for every input token: the system prompt, the conversation history, the RAG context. This is the expensive part. It scales with token count and model size, and it’s compute-bound on the GPU. Decode generates output tokens one at a time, each one reusing the key-value pairs from prefill. This is the cheap part.

vLLM and other serving engines cache the key-value pairs from prefill in GPU memory. When a new request arrives with the same token prefix, the engine skips prefill entirely and jumps straight to decode. This is the KV cache hit.

On our benchmarks, a cache hit on CodeLlama 13B returns in 18ms at P50. A cache miss takes around 500ms. That’s a 28x gap in time-to-first-token, decided entirely by whether the tokens were already on that GPU.

But here’s the thing: the KV cache is per-GPU. GPU 0’s cache doesn’t help GPU 3. If your load balancer sends Request A to GPU 0 and the identical Request B to GPU 3, Request B pays full prefill cost even though the work was already done. The cache exists. It’s just on the wrong card.

The Math on Wasted Prefill

Let’s make this concrete. You’re running a RAG application with a 4,000-token system prompt. You have 8 GPUs serving CodeLlama 13B. You’re handling 30 concurrent users with a stress workload (heavy on large and extra-large prefixes). Here’s what we measured on 8x A100s:

Round-robin routing:

• Cache hit rate: 12.5%
• P99 TTFT: 6,800ms
• Throughput: 36.3 req/s

With 8 backends and random routing, you’d expect ~12.5% cache hits by chance. One in eight requests happens to land on the GPU that already has its prefix cached. The other 87.5% recompute from scratch.

Prefix-aware routing:

• Cache hit rate: 97.5%
• P99 TTFT: 1,000ms
• Throughput: 44.4 req/s

Same GPUs. Same model. Same workload. The only change is which GPU receives which request.

That throughput difference, 36.3 vs 44.4 requests per second, is a 22.3% improvement. On hardware costing ~$10/hour, that’s either 22% more throughput for free or the same throughput on fewer GPUs. Over a month of continuous operation, on a single 8-GPU node, the wasted prefill in round-robin comes to roughly $1,200–$1,800 in GPU-hours (22% of ~$7,300/month at $10/hr) that produce no useful work. Multiply by the number of nodes in your cluster.

Where the Savings Compound

The benefit scales with three variables: model size, prefix length, and prefix sharing ratio.

Model size

Larger models have more expensive prefill, so cache misses cost more.

The 8B numbers are the warning case. When prefill is already fast (~420ms total inference), the 7-10ms routing overhead eats into the savings. If prefill isn’t your bottleneck, prefix-aware routing doesn’t help.

The 70B numbers tell a different story. Aggregate throughput doesn’t change because the GPUs are already compute-saturated. But individual requests are 44% faster on cache hit (P50: 1,498ms hit vs 2,665ms miss). Your users feel the difference even if your throughput dashboard doesn’t.

The sweet spot is 13B-70B models where prefill is expensive enough to matter but the GPUs aren’t so saturated that they can’t benefit from skipping it.

Prefix length

Longer shared prefixes mean more wasted compute per cache miss.

At 16K tokens, a cache miss wastes nearly 400ms of GPU compute that a hit avoids entirely. As context windows keep growing, this gap widens.

Prefix sharing ratio

This is the percentage of tokens shared across requests. A RAG application where every request includes the same 4,000-token knowledge base has a high sharing ratio. A chat application where every conversation is unique has a low one.

Even at 50% sharing, where half the tokens are unique, prefix-aware routing still achieves 91% cache hits. A consistent hash fallback (deterministic routing based on prefix when no learned route exists yet) ensures that requests with the same prefix land on the same GPU even before the system has observed them.

The P99 Story

Cost isn’t just GPU-hours. It’s also the cost of slow responses.

At 30 concurrent users on CodeLlama 13B over 30 minutes of sustained load, round-robin routing produced a P99 TTFT of 6,800ms. That’s 6.8 seconds before the first token appears. For an interactive application like code completion or chat, that’s a broken experience. Users don’t wait 6.8 seconds.

Prefix-aware routing brought that same P99 down to 1,000ms. Same hardware, same model, same concurrency. An 85.3% improvement on tail latency.

Why does the tail improve so much? Because tail latency in LLM serving is driven by cache misses under load. When the GPU is busy generating tokens for other requests, a new request that requires full prefill gets queued behind them. With round-robin, 87.5% of requests need full prefill, so the queue is always full of expensive work. With prefix-aware routing, 97.5% of requests skip prefill entirely, so the queue drains faster and the few remaining misses get processed sooner.

This is the strongest argument for KV cache locality. Throughput improvements look good on a dashboard. Tail latency is what users actually experience.

What Doesn’t Work

Prefix-aware routing isn’t free, and it doesn’t help everywhere.

Small models (≤8B): Inference is already fast enough that the routing overhead (~10ms for tokenization + tree lookup) approaches the prefill savings. The net effect is roughly zero.

Short prefixes (<500 tokens): The prefill cost for short sequences is small enough that cache misses don’t meaningfully hurt. The routing overhead (~3ms minimum) can exceed the savings.

Unique conversations: If every request has a completely different prefix (no shared system prompt, no shared context), there’s nothing to cache. The routing tree learns routes that are never reused.

Load imbalance: Strict prefix affinity can create hot spots. If 80% of your traffic shares the same system prompt, prefix-aware routing sends 80% of traffic to one GPU. We handle this with a load-aware fallback that diverts requests when a backend’s in-flight count exceeds twice the median. This trades a cache miss for a balanced GPU, reducing P95 by 36% and P99 by 45% compared to strict affinity. The cache hit rate drops about 5 points, which is the right trade.

Measuring Your Own Cache Locality

Before you change anything, measure your current cache hit rate. Most vLLM deployments expose this via Prometheus:

• vllm:gpu_prefix_cache_hit_rate (or vllm:gpu_prefix_cache_queries_total and _hits_total on older versions; check your /metrics endpoint)
• Compare TTFT distributions between requests with shared vs unique prefixes
• Look at your P99/P50 ratio. A ratio above 5x suggests cache thrashing

If your cache hit rate is already above 80%, you’re either lucky or your traffic naturally clusters. If it’s below 30%, you’re leaving performance on the table.

The variables that matter most:

1. How many GPUs are you routing across? More GPUs = lower chance of random cache hits. With 8 GPUs, random routing gives ~12.5% hits.
2. How long are your shared prefixes? Longer = more wasted compute per miss.
3. What’s your prefix sharing ratio? Higher = more opportunity for reuse.
4. What model size are you serving? Larger = more expensive prefill per miss.

If you have many GPUs, long shared prefixes, high sharing ratios, and large models, you’re likely wasting 20-40% of your GPU compute on redundant prefill.

The Takeaway

KV cache locality is not a tuning knob. It’s a multiplier on your existing hardware. The same GPUs, serving the same model, handling the same traffic, produce measurably different throughput and latency depending on one decision: which GPU gets which request.

Round-robin doesn’t make that decision. Least-connections doesn’t make that decision. They balance load without understanding what the load is. When every request carries thousands of tokens that might already be cached somewhere in your cluster, “balanced” and “efficient” are not the same thing.

We built Ranvier to make that decision. It routes requests to the GPU that already has their token prefix cached, using an adaptive radix tree that learns routes in real time. The first post in this series covered why your load balancer is wasting your GPUs. This post covered what that waste costs. The next one will cover how we tokenize 50,000 requests per second without blocking the event loop.

All benchmarks run on 8x A100 GPUs (Lambda Labs), February 2026. Workloads use the stress distribution (10% small, 20% medium, 30% large, 40% xlarge prefixes) with 90% prefix sharing ratio unless noted. Full methodology and raw data available in the benchmark guide.

Ranvier is a project of Minds Aspire, LLC.

I maintain a ~50K LOC C++20 service built on Seastar. I catalogued every class of bug that burned me and turned them into 24 rules I enforce on every commit. Each one cost at least a day to diagnose. Here they are.

Memory That Isn’t Yours Anymore

Most of the worst async C++ bugs are lifetime bugs. In synchronous code, if the object exists, the scope that created it still exists. In async code, that’s not true. The object is alive but the scope that created it finished long ago.

Rule 16 — Lambda coroutines in .then() are use-after-free. This is the scariest bug on the list because it looks completely correct. You write a lambda that contains co_await, pass it to .then(), and everything compiles. Here’s what actually happens: .then() moves the lambda into internal storage. The lambda’s operator() is called, which creates a coroutine frame on the heap. The coroutine suspends at co_await. .then() is done with the lambda and destroys it. The coroutine resumes into freed memory.

The broken version:

// BROKEN — use-after-free when the coroutine suspends
future<> handle(request req) {
    return async_lookup(req.key()).then([req = std::move(req)](auto val) -> future<> {
        co_await async_log(req, val);   // .then() has already freed this lambda
        co_return;
    });
}

The fix:

// FIXED — seastar::coroutine::lambda() keeps the frame alive
future<> handle(request req) {
    return async_lookup(req.key()).then(seastar::coroutine::lambda([req = std::move(req)](auto val) -> future<> {
        co_await async_log(req, val);
        co_return;
    }));
}

The compiler will never warn you about this. I found it after three days of chasing a heap corruption that only appeared under sustained load.

Rule 5 — Timer callbacks need gate guards. A repeating timer fires after stop() has already begun destroying this. The callback dereferences member variables that no longer exist. The fix is seastar::gate, but the gate holder must outlive the entire async operation, not just the try block.

// BROKEN — gate guard scoped to try body; catch runs outside the gate
void on_timer() {
    try {
        auto holder = _gate.hold();
        co_await do_work();
    } catch (...) {
        _logger.warn("failed");  // _logger is destroyed during shutdown
    }
}

// FIXED — gate guard covers the entire operation including error handling
void on_timer() {
    auto holder = _gate.hold();
    try {
        co_await do_work();
    } catch (...) {
        _logger.warn("failed");
    }
}

During shutdown, _gate.close() waits for outstanding holders. If the holder is scoped inside the try, the catch path runs unguarded, touching members that stop() has already destroyed.

Rule 21 — Coroutine reference parameters dangle. Just take coroutine parameters by value. Always. A coroutine that takes const std::string& looks correct, compiles fine, passes every unit test, and breaks under load. The caller’s string goes out of scope, the coroutine suspends, and when it resumes the reference points to freed memory.

// BROKEN — reference dangles when caller's scope ends before coroutine resumes
future<> process(const std::string& key) {
    co_await db.lookup(key);  // key may be freed by now
}

// FIXED — take by value, always
future<> process(std::string key) {
    co_await db.lookup(key);
}

The cost of a copy is nothing compared to debugging a dangling reference that only shows up at the 99.9th percentile.

Rule 20 — Missing & in do_with lambdas. seastar::do_with allocates objects on the heap and passes them by reference to your lambda. Forget a single & and you get a copy instead.

// BROKEN — buf is captured by value; the copy dies when the lambda returns
return seastar::do_with(std::move(buf), [](auto buf) {
    return async_write(buf);  // dangling reference to destroyed copy
});

// FIXED — capture by reference; do_with owns the object for the future's lifetime
return seastar::do_with(std::move(buf), [](auto& buf) {
    return async_write(buf);
});

One missing character. The copy is destroyed when the lambda returns, but the future it spawned is still running, still holding a reference to the now-dead copy. Heap corruption that shows up in completely unrelated code, sometimes minutes later.

Two ways to keep this from biting you in the first place. If you control the type, make it move-only — the broken version won’t compile. And in C++20 coroutine style you rarely need do_with at all; a local variable in a coroutine lives across co_await suspensions, which is the whole point.

Rule 23 — share() on temporary_buffer pins the whole allocation. You call .share() to grab a 32-byte header from a 64KB network buffer. Both shared views now pin the same underlying allocation. You cache the header, but the “temporary” buffer lives forever. The result is unexplained memory growth that doesn’t correlate with logical data sizes. The fix: copy the bytes you need into a new buffer, then release the shared view.

The Reactor Is Not a Thread

Seastar is cooperative. There is no kernel to preempt you. Every microsecond you block is a microsecond where that core serves zero requests.

Rule 2 — No co_await in unbounded loops over external resources. The pattern for (auto& item : items) { co_await process(item); } is O(n) latency for that request. 100 items at 10ms each means a full second before the caller hears back. The shard keeps serving other connections in the meantime — it isn’t idle — but this one request is needlessly serialized. The point of async is that you get to pick the execution shape: serial, fully concurrent, or bounded concurrency. Pick deliberately. When you want parallelism, use seastar::parallel_for_each or seastar::max_concurrent_for_each with a cap so you don’t exhaust memory or downstream connections.

Rule 12 — No std::ifstream in coroutines. It compiles. It works in testing with SSDs. In production, one 10ms disk stall freezes the entire shard. Every connection on that core drops packets for 10ms. Use Seastar’s file I/O, which goes through the reactor and yields properly. And don’t reach for seastar::thread as an escape hatch — it’s a stackful coroutine that runs on the reactor, so blocking inside it stalls the shard exactly the same way. If you genuinely have no async-friendly alternative, run the blocking call on a dedicated OS thread outside the reactor and ferry the result back through the alien-thread API.

Rule 17 — Preemption points in hot loops. A tight loop that runs for 500μs without yielding starves everything else on that core. Insert co_await seastar::coroutine::maybe_yield() every ~100 iterations. The cost is a branch that’s almost never taken. The cost of not doing it is a reactor stall warning in your logs and a mystery latency spike that disappears when you reduce load.

Cross-Shard Is Cross-Universe

Each core in Seastar has its own memory allocator. This isn’t an implementation detail you can ignore. It’s a load-bearing invariant, and violating it corrupts allocator state silently.

Rule 0 — std::shared_ptr destructs on the wrong shard. The refcount is atomic, so the decrement is “safe” from any core. But the destructor runs on whichever core decrements last. That destructor frees memory through the wrong core’s allocator.

// BROKEN — destructor runs on whichever shard releases last
std::shared_ptr<session> s = std::make_shared<session>();
// ... shared across shards via submit_to() ...
// shard 3 drops the last reference; ~session() frees memory
// allocated by shard 0's allocator. Silent corruption.

// FIXED — foreign_ptr ensures destruction on the owning shard
seastar::foreign_ptr<seastar::lw_shared_ptr<session>> s;

Use seastar::lw_shared_ptr (non-atomic refcount, shard-local only) for objects that stay on one shard. Wrap cross-shard pointers in seastar::foreign_ptr, which ensures the destructor runs on the owning shard. This was the first bug that burned me and the last one I expected, which is why it’s Rule 0.

Rule 14 — Cross-shard heap data must be reallocated locally. You submit_to() another shard with a std::string. The target shard reads memory allocated by the source shard’s allocator. Maybe it works today. Maybe the allocator metadata is adjacent and you corrupt it on the next allocation. Copy on receive. Always. It feels wasteful but it prevents silent corruption.

Rule 15 — FFI across shard boundaries needs reallocation in both directions. Passing Seastar-allocated memory to an FFI boundary (Rust, C libraries) means the foreign code may free or reallocate through a different allocator. Reallocate into standard malloc memory before calling FFI. Reallocate the result back into Seastar’s allocator before returning to the reactor.

Futures Are Not Exceptions

C++ effectively has two error propagation systems now: exceptions and future chains. Code that mixes them has gaps where errors fall through.

Rule 18 — Discarded futures silently swallow errors. Calling an async function without co_await means the returned future is destroyed immediately. If that future eventually resolves with an exception, nobody sees it. Seastar logs a warning at runtime, but by then the damage is done: a write that didn’t complete, a cleanup that never ran. Every future must be co_awaited, returned, or explicitly discarded with a comment explaining why.

Rule 22 — Throwing before returning a future bypasses .finally(). If an exception is thrown synchronously before the function returns a future, it propagates as a regular C++ exception. Any .finally() attached to the expected return value never executes. Cleanup is skipped. Resources leak. Use seastar::futurize_invoke() to wrap the call, which catches synchronous exceptions and converts them into failed futures. Or just use coroutines, which handle this naturally.

Rule 19 — Raw semaphore::wait()/signal() leaks units on throw. You call wait(), do work, call signal() in a .finally(). But if the work throws synchronously before you attach .finally(), the units are never returned. The semaphore’s available count decreases monotonically until everything deadlocks. Use seastar::with_semaphore(), which handles the lifecycle correctly regardless of how the operation fails.

The Rules I Didn’t Expect

Some rules aren’t about the language at all.

Rule 4 — Every growing container needs MAX_SIZE. No unbounded buffers, ever. A single malicious peer sending oversized messages will OOM your process if nothing caps the queue. Every std::vector and std::deque, every ring buffer gets a configured maximum.

Rule 9 — Every catch block logs at warn level. A silent catch(...) is the number one cause of “it works but something is wrong” in production. If you’re catching an exception, something unexpected happened. Log it. If it’s too noisy, fix the root cause instead of silencing the symptom.

Rule 7 — Persistence only stores, never validates. This is a design rule, not a language rule. When the persistence layer also validates, you can’t test business logic without spinning up storage. When it only stores, you can test validation in isolation and reason about correctness without thinking about I/O.

The Remaining Rules

For completeness, here are the rules not covered in full above:

• Rule 1 — Metrics accessors must be lock-free, no std::mutex in query methods.
• Rule 3 — Null-guard all C string returns. sqlite3_column_text() returns NULL on empty columns; dereferencing it is undefined behavior.
• Rule 6 — Deregister metrics first in stop(). Prometheus scrape lambdas capture this; if this is destroyed first, the next scrape is a use-after-free.
• Rule 8 — Single ShardLocalState struct per service, no scattered thread_local variables.
• Rule 10 — Validating helpers for string-to-number conversions. std::stoi() throws on bad input; raw calls in request parsing are a crash waiting to happen.
• Rule 11 — std::call_once or std::atomic for one-time global initialization, never a bare static with lazy init.
• Rule 13 — Thread-local new needs an explicit destroy function registered with the allocator, or the memory leaks on shard shutdown. (Switching to std::make_unique doesn’t fix this if the smart pointer itself has thread_local storage — the hazard is the destruction order at shard teardown, not the allocation syntax.)

How I Enforce Them

These rules live in a reference document I consult on every commit. They’re enforced by discipline, not tooling. No linter can tell you that a lambda coroutine in .then() is a use-after-free.

Numbering them matters. “Rule 16” is a faster shorthand than re-deriving the coroutine frame lifetime problem each time you encounter it.

The list started at Rule 0 and grew to 24. I add a rule only when a bug burns me. Never speculatively. If you’re building something similar, start your own list. The specific rules matter less than the habit of writing them down.

The Takeaway

Async C++ gives you performance that no garbage-collected language can match, but it takes away the safety nets. You have to build your own. Write your rules down.

I’m building Ranvier, a Layer 7 load balancer for LLM inference on Seastar. If this kind of systems work interests you, check out the source.

Updated 2026-05-04 with corrections from reader feedback on r/cpp: Rule 12 no longer recommends seastar::thread for blocking I/O (it’s a stackful coroutine on the reactor, not an escape hatch), Rule 2 was reworded to clarify that the cost is per-request latency rather than shard-wide idleness, and Rule 20 notes that move-only types and C++20 coroutines largely sidestep the do_with hazard. Thanks to the commenters who flagged these.

Ranvier is a project of Minds Aspire, LLC.

The Problem: KV Cache Thrashing

Modern LLM inference engines like vLLM and TensorRT-LLM use KV caching to avoid recomputing attention for tokens they’ve already seen. When a user sends a follow-up question about a document, the engine can skip directly to generating the answer—if that document’s context is still in memory.

Here’s what actually happens with standard load balancing:

Request A loads a 4,000-token PDF into GPU-1. Request B, asking a question about that same PDF, arrives moments later. Your load balancer—following Least Connections or Round Robin—routes it to GPU-2. GPU-2 has no idea this document exists. It re-computes the entire 4,000-token prefill from scratch.

At scale, this pattern can waste significant inference capacity. You’re paying for GPUs to redo work that’s already been done.

The Insight: Content-Aware Routing

What if your load balancer understood tokens, not just packets?

The fix is conceptually simple: route requests to the GPU that already has the relevant KV cache. Skip the prefill entirely. But implementing this requires solving a hard problem—you need sub-millisecond lookups across potentially millions of cached prefixes, updating in real-time as caches fill and evict.

This is the problem Ranvier solves.

Introducing Ranvier

Ranvier is a Layer 7+ load balancer purpose-built for LLM inference. Instead of treating requests as opaque HTTP packets, it inspects the token sequence and routes to the backend most likely to have that prefix cached.

How it works: Ranvier maintains an Adaptive Radix Tree (ART) that maps token prefixes to backend GPUs. When a request arrives, Ranvier tokenizes the prompt, performs an O(L) prefix lookup (where L is prefix length, not total keys), and routes to the matching backend. When backends report cache evictions via gossip protocol, Ranvier updates its routing table.

Why it’s fast: Built on C++20 and the Seastar framework—the same shared-nothing, thread-per-core architecture that powers ScyllaDB. No locks, no atomics, no cross-core coordination in the hot path. Total routing overhead is 6-8ms with server-side tokenization. For production deployments where clients send pre-tokenized requests, overhead drops to <1ms.

The name comes from the Nodes of Ranvier in neuroscience—the gaps in myelin sheath that allow nerve signals to “jump” between points (saltatory conduction), dramatically increasing signal speed. Ranvier does the same for inference: jumping past redundant computation to reach cached state.

The Results

We tested Ranvier on 8x A100 clusters running vLLM across multiple instances, comparing prefix-aware routing against standard round-robin:

CodeLlama-13b, 10-30 concurrent users, 10-30 minutes, stress-distribution prompts (70% large/xlarge prefixes). Results validated across four Lambda Labs instances (Feb-March 2026). Per-request, XLarge prompts (4K-8K tokens) saw 33-39% faster time-to-first-token on cache hits. Full methodology and raw data in the benchmark guide.

Larger models benefit more. 70B models show the highest per-request improvement (44-49% faster XLarge TTFT on cache hit) because the KV cache saves more compute per token. 13B models show the strongest aggregate gains—P99 tail latency drops 60-85% and throughput increases 12-22%—because backend queuing under load amplifies the benefit of cache-aware routing. 8B models are routing-neutral: inference is fast enough (~420ms) that cache savings don’t affect aggregate latency, though per-request XLarge improvement remains strong at 28-40%.

Routing overhead is 6-8ms with server-side tokenization, <1ms with pre-tokenized requests. You’re trading milliseconds of routing for seconds of tail latency improvement.

Where Ranvier shines: RAG pipelines, multi-turn chat with system prompts, few-shot learning—any workload where requests share common prefixes of 2K+ tokens.

Landscape and Positioning

The inference ecosystem is moving fast. vLLM shipped their own router in December 2025. Red Hat’s llm-d takes a different approach—introspecting vLLM’s KV cache directly via live events. SGLang builds prefix caching into the engine itself via RadixAttention.

Ranvier takes a third path: engine-agnostic, external routing based on token-level prefix matching. It works with any OpenAI-compatible backend—vLLM, SGLang, TensorRT-LLM, Ollama—without requiring engine modifications or instrumentation hooks. The tradeoff is that Ranvier infers cache state from routing history rather than observing it directly, which means slightly lower cache hit precision in exchange for portability.

We think there’s value in a routing layer that isn’t coupled to a specific inference engine, especially as the ecosystem fragments.

Full roadmap: VISION.md

Code and Docs

Ranvier is open source under Apache 2.0: github.com/Ranvier-Systems/ranvier-core

If you’re interested in the internals—why an Adaptive Radix Tree instead of a hash map, how the gossip protocol tracks distributed cache state, what we learned building on Seastar’s shared-nothing architecture—start with the system design doc.

• Benchmark methodology
• Deployment guide

Issues and feedback welcome—especially if you run it against a workload we haven’t tested.

Ranvier is a project of Minds Aspire, LLC.