Engineer's view. This is a required dropdown with no "Not applicable" option. Validation always passes, so the data looks clean. But users who had no right answer picked the first item, and your reports now count them. A JSON schema enforced token by token works the same way.

TL;DR. Forced JSON mode removes the tokens your schema forbids and rescales the rest, so the output always parses. The model still chooses among legal tokens. When it wanted something illegal, it guesses, and the guess looks as sure as a real answer.

Explain Like I'm 5

A teacher hands out a quiz where every question has three boxes. You must tick one. There is no box for "I don't know." So when you don't know, you tick a box anyway. The teacher sees a neat page with every box ticked. The teacher cannot tell which ticks you knew and which you guessed. The fix is easy: add a fourth box that says "I don't know."

The Problem

You have shipped this bug before, in a system with no AI in it.

A signup form asks for a country. The field is required, and the dropdown has no "Other" option. Users from a missing country still have to submit. So they pick the first item in the list. Months later, an analyst asks why so many users live in Afghanistan. The validation never failed once. It only made sure every answer was a legal one.

Structured output has the same bug. You give the model a JSON schema, and the engine guarantees the response matches it. OpenAI, Gemini, vLLM, SGLang and llama.cpp all offer this. Teams switch it on to stop parse errors, and the parse errors stop.

Then quieter failures start. FriendliAI ran Gemma 4 31B against a complex schema. The output filled with newlines until it hit the token limit, and the JSON never closed. In other pipelines, a field limited to three values gets one of the three, even for a document the model cannot read. The value looks as sure as any other.

All of these come from one rule. The engine decides which tokens are legal. The model still chooses among them, and it must choose even when it wanted to say something else.

The fix: give the model a legal way to say what it means, bound what it can repeat, and measure how much it was forced.

The Fix: Give the Model a Legal Way Out

How does the engine force valid JSON?

A model writes one token at a time, a word or a piece of one. At each step it gives every token in its vocabulary a probability. Constrained decoding adds one step. The engine checks the grammar built from your schema, deletes every token that would break it, and rescales the rest to sum to one. Then it samples as usual.

So what happens when the model wanted an illegal token?

Its wish is deleted, and the leftovers are scaled up. Take a field that must be high, medium or low. On a clear invoice, those three hold 95% of the model's probability. On a blurry scan, they hold 15%, because the model wanted to write "unknown." After rescaling, the blurry scan still gets high 40% of the time. Both outputs look the same.

Why does output sometimes fill with whitespace?

JSON allows spaces and newlines between fields, so the grammar never deletes them. A little leftover whitespace cannot do this alone. Even at a 99% share each step, filling 2,048 tokens has odds of about one in a billion. The loop needs the model to be nearly certain at every step. That happens when repetition feeds itself: each newline makes the next more likely.

Does the order of fields matter?

The engine writes fields in the order the schema lists them, left to right. A confidence value written before a reasoning field cannot depend on that reasoning.

What This Means for You

When this matters. You use JSON mode, a response schema or strict tool calls, and your code acts on a field's value. Agents do this constantly. A tool call is a schema the model must fill in. A routing step reads one enum. An extraction job writes rows to a database.

How it affects you. Parse-error rates drop to zero, so dashboards built on them look healthy. The failures move into the values. A forced pick is a wrong row nobody flags. A whitespace loop is a timeout, or a bill for 2,048 tokens of nothing.

This is the same bug as the country dropdown. Validation guarantees that every answer is a legal one. It says nothing about whether the model had an answer.

What to do about it. Today, open the schemas your code acts on. For every enum, ask what the model should say when it does not know. If there is no legal answer for that, add one. Next, put any reasoning field before the fields that depend on it. Whether that improves accuracy depends on the task and the model. It is cheap to test, so measure it on yours rather than assume. Then cap output length and treat a truncated response as a failure. If you host the model, switch off free whitespace and start logging how much probability the grammar removed.

Implementing It

The change.

Four roles touch this: whoever writes the schema, whoever writes the prompt, whoever runs the server, and whoever reads the response.

Role 1, the schema. Give every field your code acts on a legal "I don't know." Put reasoning first. Bound free strings where your provider's schema subset allows maxLength:

from typing import Literal
from pydantic import BaseModel, Field

class Invoice(BaseModel):
    reasoning: str = Field(max_length=600)        # written first, read by the fields below
    vendor: str | None = Field(max_length=120)    # None is the legal "not on the page"
    total: float | None
    confidence: Literal["high", "medium", "low", "unknown"]

Lint every schema in CI with check_schema() from code_example.py. It flags an enum with no way out, an unbounded string, and an answer written before its reasoning.

Role 2, the prompt. Include one filled-in example of the output. FriendliAI's whitespace loop went away when the prompt carried a concrete example. The mask only removes options; the example puts probability on the right ones. Use a real record, not placeholders. An example full of "string" teaches the model to write "string". Show an unknown case too, so the model sees that the way out is allowed.

Role 3, the server, if you host the model. vLLM can forbid whitespace between JSON fields, which removes the loop's open door:

vllm serve <model> \
  --structured-outputs-config '{"backend":"xgrammar","disable_any_whitespace":true}'

It works only with the xgrammar and guidance backends. With transformers, wrap your grammar processor to record how much probability it removed:

import torch
from transformers import LogitsProcessor

class KeptMass(LogitsProcessor):
    def __init__(self, grammar):                  # any processor that sets illegal tokens to -inf
        self.grammar, self.kept = grammar, []
    def __call__(self, input_ids, scores):
        before = torch.softmax(scores.float(), dim=-1)
        out = self.grammar(input_ids, scores.clone())
        self.kept.append((before * torch.isfinite(out)).sum(-1))
        return out

Role 4, the caller. A cut-off response is a failure, not something to repair:

resp = client.chat.completions.create(model=MODEL, messages=msgs,
    response_format={"type": "json_schema", "json_schema": SCHEMA}, max_tokens=800)
choice = resp.choices[0]
if choice.finish_reason == "length":
    raise TruncatedOutput(resp.id)                # retry or route to review; never parse
row = Invoice.model_validate_json(choice.message.content)

Gemini reports the same case as MAX_TOKENS in finishReason. Count truncations per schema, because one bad schema hides inside a healthy average.

How you know it worked. Three numbers per field, on a week of traffic. The truncation rate should be near zero. The unknown rate should be above zero, and roughly match how often a human finds no answer in a sample. If it is exactly zero, the model is still being forced somewhere. If you host, look at kept mass on value tokens only. Structure tokens like { lose mass on every request for harmless reasons. A value position that kept under half the model's probability is a guess worth reviewing.

When Forcing JSON Is the Wrong Tool

When the model needs room to think. A schema that starts on token one leaves no space for free reasoning. For hard tasks, let the model answer in plain text first. Then make a second, cheap call that formats the answer into JSON.

When your schema is huge or changes per request. The engine compiles each new grammar before the first token. A cached schema costs little, but a new one per request adds delay to every first token. Grammar engines cache compiled schemas for this reason.

When you use a small model on a hard schema. Forcing makes the output parse. It cannot make a weak model know the answer. Nested optional fields are where small models fill values with whatever is left.

When the provider's schema subset cannot express your rules. Some keywords are silently unsupported or rejected. Check the response against the full schema in your own code anyway.

Before you adopt it, ask:

• What should each field say when the model does not know?
• Which fields does my code act on, and is anything written before its reasoning?
• How would I notice a value the model was forced into?

Glossary

• token — a word or piece of a word; the model writes one at a time.
• Structured output — an API mode that guarantees the response matches a JSON schema you supply.
• Constrained decoding — deleting the tokens a grammar forbids at each step, then sampling from what is left.
• enum — a schema field limited to a fixed list of allowed values.
• kept mass — the share of the model's probability that survives the grammar's deletion at one position.

Further reading

• Structured Output Requires More Than Guided Generation — FriendliAI Tech & Research
• vLLM: StructuredOutputsConfig — vLLM documentation
• Achieving Efficient, Flexible, and Portable Structured Generation with XGrammar — MLC team (CMU and collaborators)

Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

Engineer's view. This is a Node.js server doing a synchronous parse on every upload. Each parse is quick, but it blocks the one loop every connection shares, so every response slows. With a fixed connection pool, slower responses then mean waiting for a free connection. The decode server's hand-off work behaves the same way.

TL;DR. When prompt processing and token generation run on separate GPUs, each request's cache must cross between them. The bytes are cheap. Setup work on the generation server's loop slows every token, and that can run the server out of slots.

Explain Like I'm 5

A bakery has one person mixing dough and another baking it. The mixer hands each bowl to the baker. Passing a bowl takes a second. But this baker stops all the ovens to write a label for every new bowl. Each label is quick. When bowls arrive fast, the ovens are stopped so often that every loaf bakes slower. Soon the oven shelves are full, and new bowls wait in line.

The Problem

You have shipped this bug before, in a system with no AI in it.

An API reads from a database through a pool of 20 connections. A release adds a small synchronous step to each query. Every query now holds its connection a little longer. At normal traffic nobody notices. At peak, all 20 connections are busy, and requests wait in line for one. The step costs milliseconds. The wait costs seconds. Nobody finds it by looking at the database, because the database is fine.

Disaggregated serving has the same bug. It splits an LLM server in two. Prefill GPUs read the whole prompt. Decode GPUs then write the answer one token at a time. Splitting them stops long prompts from stalling other users' answers. In between, each request's KV cache, the model's notes on the prompt, moves from one pool to the other.

Teams blame the network when the first token is slow. The numbers say otherwise. vLLM measured Qwen3-32B on two eight-GPU H200 servers. At 8 requests per second with 16,000-token prompts, the first token took about 1,744 ms with the default hand-off. A different hand-off cut it to about 540 ms. Moving that prompt's cache takes about 10 ms. How to Work Out How Much GPU Memory One Request Really Needs sized that cache.

The lost second is setup work. Part of it runs on the decode server's main loop, where it pauses every request in progress.

The fix: keep hand-off work off the decode loop, and size decode slots with headroom. Then the split pays for itself.

The Fix: Keep Hand-Off Work Off the Decode Loop

Why split prefill and decode at all?

Prefill is limited by arithmetic. Decode is limited by memory reads. On one server, a long prefill delays every answer being written. Splitting them lets you tune first-token time and per-token time separately. vLLM's own guide is blunt about the cost: disaggregation does not improve throughput. It buys predictable latency.

How much do the bytes actually cost?

Very little. Qwen3-32B stores 256 KiB per token. A 16,000-token prompt is 4.2 GB. With tensor parallelism across 8 GPUs, each GPU sends its own slice over its own links. At 50 GB/s per GPU, that takes about 10 ms.

Then where does the second go?

Into two places. First, pull runs its steps in series after prefill. The router relays where the blocks are. Decode allocates space. Then decode reads the blocks. Second, decode does that setup on its main loop. That loop also produces every token for every request. While it posts transfer work, nobody gets a token.

So the loop loses arrivals per second times setup time per request. A vLLM pull request measured 14.7 ms of posting per request at 2,000-token prompts. At 8 requests per second, the loop loses 12%. Moving that work to a thread took time per token from 6.14 ms to 5.52 ms.

Why does a slower token slow the first token?

Because a request holds a decode slot until its answer ends. Slots held equals arrivals times output tokens times time per token. That is Little's law. Past --max-num-seqs, new requests wait for a slot before their cache can land.

What does push change?

Decode registers its free blocks as soon as the request arrives. Prefill writes straight into them from a background thread. Nothing waits after prefill, and nothing runs on the decode loop.

What This Means for You

When this matters. You run prefill and decode on separate GPUs with vLLM, SGLang, Dynamo or llm-d. Your prompts are long, and your traffic is high.

How it affects you. Look at vLLM's 16,000-token run at 8 requests per second. Time per token rose from 7.0 ms with push to 10.4 ms with pull. With 128-token answers, that means 7.2 slots held with push and 10.6 with pull. The benchmark started its servers with --max-num-seqs 8. So pull asked for more slots than existed. That turns a few milliseconds per token into a second of waiting.

This is the same arithmetic as the connection pool. A small per-request cost decides nothing until the pool is nearly full. Then it decides everything. It also explains why the two modes match at low load. At 1 request per second, even pull holds barely one slot. The setup still happens, but nobody is waiting behind it.

What to do about it. Today, compute slots held for your own traffic at peak. Use your measured time per token, not the benchmark's. Keep it under about 80% of --max-num-seqs. Then move hand-off work off the decode loop, and re-measure. If you cannot change the hand-off this quarter, raise --max-num-seqs only if decode memory allows it. More slots need more KV cache space on every decode GPU.

Implementing It

The change.

Three roles touch this. Whoever plans capacity sizes the decode pool. Whoever owns the serving config picks the hand-off. Whoever runs the load test proves it.

Role 1, capacity. Compute the three numbers for your model and traffic. The first is usually small. The third is the one that bites:

def kv_bytes(tokens, layers, kv_heads, head_dim, dtype_bytes=2):
    return tokens * layers * 2 * kv_heads * head_dim * dtype_bytes

def transfer_ms(nbytes, link_gb_s, tp=1):
    return nbytes / tp / (link_gb_s * 1e9) * 1000   # each GPU sends its slice

def slots_held(qps, out_tokens, tpot_ms):
    return qps * out_tokens * tpot_ms / 1000         # Little's law

transfer_ms(kv_bytes(16_000, 64, 8, 128), 50, tp=8)  # 10.5 ms
slots_held(8, 128, 10.4)                             # 10.6, with 8 slots

Take tpot_ms from production at peak, not from a quiet benchmark. Use your real output length, too. Agents that write long answers hold slots far longer than chat does. A 2,000-token agent answer holds a slot sixteen times longer than a 128-token chat reply. So the same request rate needs sixteen times the slots.

Role 2, the serving config. The connector is a setting in --kv-transfer-config on both servers. The vLLM post ran push with this, on the prefill side:

vllm serve Qwen/Qwen3-32B --tensor-parallel-size 8 --max-num-seqs 8 --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlPushConnector","kv_buffer_device":"cuda",
    "kv_role":"kv_producer","engine_id":"prefill-engine-001","kv_rank":0,
    "kv_parallel_size":2,"kv_connector_extra_config":{"backends":["LIBFABRIC"]}}'

The decode server uses the same connector with "kv_role":"kv_consumer", as in vLLM's NIXL guide. Mixing modes crashes the server after the first request. Push also needs a proxy that knows the prefill server's address up front. The post uses disagg_proxy_pushconnector_demo.py. Pull stays the right choice if you route each request to the least-loaded decode server after prefill. Push picks decode early, so it gives that up.

Role 3, measuring. Sweep request rate and prompt length together, because the gap only opens at the top of both:

for qps in 1 2 4 8; do for len in 1024 16384; do
  vllm bench serve --model Qwen/Qwen3-32B --request-rate $qps \
    --random-input-len $len --random-output-len 128 --num-prompts 400
done; done

How you know it worked. Push and pull should match at low rate and short prompts. At the top of the sweep, time per token should stay near its light-load value. Watch vllm:num_requests_waiting on the decode server. It should stay near zero at your peak rate. If it climbs, you are out of slots, whatever the hand-off. Then split first-token time into its parts. Compare the prefill server's own first-token time with the proxy's. The difference is the hand-off plus any wait for a slot. After the change, that difference should shrink most at long prompts and high rates.

When Splitting Prefill and Decode Is the Wrong Tool

When you want more throughput. Splitting does not add any. It trades GPUs for steadier latency.

When your prompts are short. Prefill then barely disturbs decode, so there is little interference to remove. Chunked prefill on one server may be enough.

When you have one GPU node. The hand-off then crosses no network, but you still pay for two pools and a proxy.

When you need load-aware decode routing. Push chooses the decode server before prefill finishes. If your decode servers run unevenly, pull's late choice may be worth its setup cost.

When the decode pool is already full. No hand-off fixes too few slots. Add decode capacity first, then tune the hand-off.

Before you adopt it, ask:

• How many decode slots does my peak traffic hold, by Little's law?
• Is my first-token time spent in prefill, in the hand-off, or in a queue?
• Do I need to route decode after prefill, or can I choose early?

Glossary

• Disaggregated serving — running prompt reading and answer writing on separate GPU pools.
• Prefill — the step that reads the whole prompt and builds its cache.
• Decode — the step that writes the answer, one token at a time.
• KV cache — the model's stored notes on every token so far, kept on the GPU.
• Little's law — items in a system equal arrival rate times time each one stays.
• slot — one of the requests a decode server can generate for at once.

Further reading

• Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM — vLLM Blog (Sunita Nadampalli, AWS, and Nicolò Lucchesi, Mistral)
• vLLM pull request 45211: offload NIXL READ submission to a dedicated reader thread — vLLM project, GitHub
• Throughput is Not All You Need: Maximizing Goodput in LLM Serving using Prefill-Decode Disaggregation — Hao AI Lab, UC San Diego (the DistServe authors)
• vLLM: Disaggregated Prefilling — vLLM documentation

Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

The shape of the system

• An SDK on the developer's backend reports { userId, model, token counts, feature? } after each AI call. It never sees prompts or answers.
• A webhook (RevenueCat, Stripe beta) or a revenue API reports payments.
• The server joins both on user ID and computes margin per user, plan and free trial.

Stack: TypeScript, Next.js, Hono, Postgres (Supabase), Drizzle.

1. Idempotent events

Networks retry. If a batch is sent twice, cost doubles and every margin number is wrong. Each event carries a client-generated UUID and the server dedupes on it, so retries are always safe. The SDK times out, retries once, then drops the batch rather than block the app.

2. Fail open

A metering tool must never take down the product it measures. Tracking is batched in the background, and the optional spending cap works the same way: meter.check(userId) runs before an AI call and fails open, so if FiscaLLM is unreachable the call goes through. The trade-off is that caps use cost already recorded, so a user can go slightly over before being refused.

const meter = createMeter({ apiKey: process.env.FISCALLM_KEY, waitUntil });
const res = await openai.chat.completions.create({ model, messages });
meter.trackOpenAI(res, { userId, feature: "chat" });

3. Price cached tokens separately

Providers charge cached input tokens at a different rate. Treating all input tokens the same overstates cost for apps with long, stable prompts. Cached tokens carry their own price.

4. Never count an unknown model as zero

A new model ID with no price entry would silently show $0 and make a user look profitable. Unknown models are flagged as unpriced instead, so the gap is visible. Built-in prices are checked against provider pages in ACCURACY.md.

5. Identity: RevenueCat aliases

A user is often anonymous first, then logs in. RevenueCat reports aliases, and if you don't merge them you split one person's revenue and cost across two IDs. FiscaLLM resolves aliases so both sides land on one user.

What's not solved

• Needs a backend between the app and the AI provider.
• Stripe is beta, tested on sample payloads only, USD only. No direct App Store or Play connection.
• Prices haven't been compared with a real invoice over weeks.
• No customers yet; the demo runs on sample data.

Try or self-host

• Demo (sample data, no sign-up): https://fiscallm.com/demo?utm_source=hashnode&utm_medium=article&utm_campaign=launch
• Repo: https://github.com/magna56/fiscallm. Self-host with SELF-HOSTING.md so your data never touches a third party's servers, or use the free hosted version.
• Roadmap: https://github.com/users/magna56/projects/1

I'd like feedback on the design, especially from anyone running AI costs at scale.

Engineer's view. This is a database that rewrote the whole table file on every transaction so it could roll back. Real databases append each change to a log, checkpoint now and then, and roll back by truncating the log. A Mamba layer can store its state the same way.

TL;DR. A Mamba layer overwrites its state every token, so undoing a rejected guess needs a saved copy. At large batch sizes those copies cost more than speculation saves. Logging each token's inputs and checkpointing now and then makes undo a pointer move.

Explain Like I'm 5

You are writing a story on a whiteboard, and a friend guesses your next four words. You write them in. If a guess is wrong, you must erase back to where you were. On paper you just cross out the last words. On a whiteboard that gets wiped and redrawn every word, you need a photo of the board before each guess. Taking all those photos is slow. A faster trick: photograph the board once in a while, and keep a list of the words since then.

The Problem

You have shipped this bug before, in a system with no AI in it.

A collaborative editor needs undo. The first version saves a full copy of the document before every edit. It works in testing. In production, a hundred people type into large documents at once. Each keystroke now writes megabytes, and the disk spends its time copying documents nobody will ever restore. The edits are cheap. The copies are not. The fix everyone lands on is an operation log: store each edit, which is small, and save a full snapshot now and then. Undo drops the newest edits from the log.

Hybrid models have the same bug inside speculative decoding. Speculative decoding lets a cheap drafter guess the next few tokens, and the big model checks them all in one pass. Wrong guesses are thrown away. How a Model Writes Several Words per Step covered why that pays.

Hybrid models mix two kinds of layer. Attention layers keep a cache that only grows. A Mamba layer keeps one fixed-size state per request and overwrites it on every token. Nemotron-3-Super has 40 Mamba layers and 8 attention layers. Each Mamba state is 4 MiB per request. That is as much as one attention layer stores for about 8,000 tokens. How to Work Out How Much GPU Memory One Request Really Needs sized that fixed part. Here it is the traffic that hurts.

To throw away a guess, an attention layer moves a pointer back. A Mamba layer cannot, because the old state is gone. So the server saves a full copy of the state after every draft token. With four drafts, that is four copies per layer, per request, per step.

The fix: store each token's small inputs plus a periodic checkpoint, the way the editor stores an operation log. Then throwing away a guess is a pointer move again.

The Fix: Log the Inputs, Checkpoint the State

New to what a model stores between tokens? Start at AI basics → Tokens and sampling.

Why are the copies so expensive?

Because producing a token is limited by how fast the GPU reads and writes memory. The weights are read once per step and shared by the whole batch. The state is per request. So as the batch grows, state traffic grows with it and soon outweighs the weights. Every copy adds a full state's worth of writes.

How does a log replace the state?

A Mamba-2 layer updates its state with one rule: decay the old state, then add the new token's contribution. That rule can be unrolled. Today's output is the checkpoint, decayed, plus each logged token's contribution, decayed by how long ago it came. So the layer computes outputs straight from the checkpoint and the log. It never writes the state in between.

When does the state get written?

When the log fills. NVIDIA and Tri Dao call this ReplaySSM. With a window of 16 tokens, the full state is written once every 16 steps instead of every step. Rejected drafts are never added to the log.

Does it change the answers?

No. The math is the same, rearranged. code_example.py checks this on random drafts and rejections. The two methods differed by under a millionth of a millionth.

What This Means for You

When this matters. You serve a hybrid model yourself. Nemotron 3, Qwen3.5 and Qwen3-Next mix Mamba-style layers with attention. You run speculative decoding, often through the model's own built-in drafter. And your batches are large.

How it affects you. Speculation can quietly make you slower. NVIDIA measured vLLM's speculation on these models. It stopped gaining beyond batch 128. On Qwen3.5-122B-A10B it fell below plain decoding at batch 512. Checking 8 draft tokens took nearly five times a normal step. With replay, the same check took about one normal step.

This is a second wall, on top of the one in the earlier article. That one was about idle compute: speculation only uses arithmetic the GPU was not using. This one is about memory traffic, and it arrives sooner, because every request in the batch pays for its own copies.

The rule underneath is simple to compute. Once state traffic dominates, a step with copies moves about (1 + T) / 2 times the state of a plain step, where T is the number of drafts. Speculation wins only if each step keeps more tokens than that.

What to do about it. Today, find the batch size your hybrid model really runs at. Then turn speculation off above it, or turn on replay. The next section shows both.

Implementing It

The change.

Three roles touch this. Whoever sizes the deployment computes the break-even. Whoever owns the serving config turns replay on. Whoever runs the benchmark proves it.

Role 1, sizing. Read your model's acceptance from the server metrics, then compute both sides of the rule:

def tokens_kept(p, T):
    """The real token, plus drafts until the first miss."""
    return sum(p**i for i in range(T + 1))

def copy_rule(p, T):
    """Above 1, speculation with state copies still wins once state traffic dominates."""
    return tokens_kept(p, T) / ((1 + T) / 2)

copy_rule(0.6, 4)   # 0.92: loses at large batch
copy_rule(0.8, 4)   # 1.34: still wins

Read acceptance from your own traffic, not a benchmark. Code completion keeps more drafts than open-ended chat, so the same model can sit on both sides of the line.

The state size per request is Mamba layers × heads × head size × state size × bytes. For Nemotron-3-Super that is 40 layers of 4 MiB, about 160 MiB per request, read every step.

Role 2, the serving config. Upstream vLLM has two flags for Mamba-2 models, both off by default:

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
  --use-replayssm --replayssm-buffer-len 16 \
  --mamba-ssm-cache-dtype float32 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 4}' \
  --max-num-seqs 512

The window of 16 is the default and was best in vLLM's own sweep. Tri Dao's write-up found 8 best for Nemotron 3, so test 8, 16 and 32. Keep the state in FP32. A low-precision copy is safe for reads only. When one shared NVFP4 state fed the recurrence, Nemotron-3-Nano-4B lost 31 points on GPQA-Diamond. Qwen3.5 uses Gated DeltaNet layers. Those kernels were in the authors' vLLM fork when this was written, not upstream, so check your release notes.

Role 3, measuring. Sweep batch sizes with each setup, and hold everything else fixed. Use the same prompts, the same output length and the same GPU. Run each point twice, because the first run warms caches and compiles kernels:

for batch in (16, 64, 128, 256, 512):
    for setup in ("plain", "spec", "replay", "replay+spec"):
        tok_s = bench(setup, max_num_seqs=batch)   # your harness, same prompts each time
        print(batch, setup, round(tok_s))

How you know it worked. Replay alone should match plain decoding at batch 1 and beat it at large batches. NVIDIA measured 1.20 to 1.47 times per decode step at batch 256. With replay, speculation should keep gaining past batch 128 instead of flattening. Then run your eval set on both. Scores should match within noise, because the math is exact.

When Replaying the Inputs Is the Wrong Tool

When your batches are small. At batch 1, the weights dominate every step. State copies are a rounding error there, and NVIDIA saw no speed-up from replay.

When the model is pure attention. Its cache already undoes a token with a pointer. There is no state to copy.

When your drafts are rarely accepted. Replay makes undo cheap, but it cannot make bad guesses useful. If each step keeps barely more than the real token, fix the drafter or turn speculation off.

When the layer is a different kind of recurrence. The trick needs the update to unroll as decay plus a sum. Mamba-2 and Gated DeltaNet do. Check before assuming a new architecture does.

When you are tempted to store the state in 4 bits. That is a separate decision. A low-precision copy for reading is fine. A low-precision state that feeds the next update builds up error.

Before you adopt it, ask:

• What batch size does my hybrid model actually run at?
• How many draft tokens does a step keep, from the server's own metrics?
• Did my eval scores hold with replay on?

Glossary

• speculative decoding — a cheap drafter guesses several tokens and the big model checks them in one pass.
• Mamba layer — a layer that keeps one fixed-size state per request and updates it every token.
• state — the fixed-size matrix a Mamba layer carries from one token to the next.
• checkpoint — a full copy of the state, written once per window.
• ReplaySSM — computing outputs from a checkpoint plus a log of recent inputs, so the state is written rarely.
• acceptance — the share of draft tokens the big model keeps.

Further reading

• ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding — NVIDIA Technical Blog (NVIDIA with Tri Dao)
• ReplaySSM: Cache SSM Inputs, Not State — Tri Dao's blog
• vLLM pull request 48018: ReplaySSM for Mamba2 standard decode — vLLM project, GitHub
• Johnny-Liou/ReplaySSM — GitHub, the authors' code (Apache-2.0)

Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

EngAI is a free iPhone app that does that translation for you. It's built for software engineers, and it's built on top of The AI Commit, a daily, hands-on AI engineering lab.

One story per card

Each card is one AI story in about 60 words. Swipe up for the next one. Categories are narrow on purpose: Models, Tools, Research and Platforms. If a story doesn't matter to someone who writes software, it doesn't get a card.

Every card links to the original source, so you can read the full piece when a story is worth it. A card that is vague, unsourced, or linked to a homepage counts as a failed card.

"For a software engineer"

This is the part nobody else writes. Under each summary, a short note says what the story means for the system you already run.

A new coding-agent release isn't just "Company X launched Y." The note tells you what changes for your job worker, your CI, or your API bill. It's the step between "interesting" and "do I need to do something about this?"

Deprecations and breaking changes, flagged with the date

Some stories aren't news, they're deadlines. Deprecations and breaking changes get a visible label and the date they take effect, so a model retirement or an SDK change doesn't surprise you in production.

A quick quiz

A short quiz checks what you just read. It takes under a minute, and it's the difference between scrolling past a story and remembering it in a design review.

Built on The AI Commit

The AI Commit publishes one hands-on lesson every weekday: a plain-English write-up, runnable Python, an interactive visualizer, and the sources behind it. Each lesson shows up in the EngAI feed as a card, so the 60-second summary links straight to the 30-minute deep dive when you want to go further.

Recent lessons include how to stop a coding agent and an AI reviewer from looping on one pull request, and how sorting your prompts speeds up a mixture-of-experts batch job.

Made for phone moments

EngAI doesn't replace your newsletters. Email is for the desk. EngAI is for the commute, the coffee line, and the five minutes between meetings.

• Free

• No account needed. Open it and start swiping.

• iPhone only for now

When you've read everything, the app tells you you're caught up. You can turn on notifications for the next batch, at most three a day.

Get it

Search "EngAI" on the App Store, or install it here: EngAI for Engineers on the App Store.

For the daily deep dives, visit theaicommit.com.

I'd love feedback from engineers using it. What would make a card more useful to you? Tell me in the comments.

Engineer's view. This is a batch job that touches records in arrival order. Each record needs a different index page, so the cache keeps reloading pages. Sort by key and each page loads once per batch. A mixture-of-experts model loads expert weights the same way, so a mixed batch pays for reads nobody shares.

TL;DR. A mixture-of-experts model reads only the experts the batch asked for, and similar prompts ask for the same ones. Sort an offline job by topic, and each step reads less, so the job finishes sooner.

Explain Like I'm 5

A library has a hundred shelves. A helper runs to fetch books for everyone at one table. If the table holds a cook, a pilot and a poet, the helper runs to many shelves each round. If everyone at the table is a cook, they mostly want the same cooking shelf. The helper makes fewer trips, and the table gets its books sooner. Nobody reads faster. The seating just changed.

The Problem

You have shipped this bug before, in a system with no AI in it.

A nightly job processes a million records in the order they arrived. Each record looks up a row in a big table. Records from the same customer sit on the same index pages, but arrival order scatters them. So the database keeps loading a page, evicting it, and loading it again an hour later. The job is slow, and the code is fine. Someone sorts the input by customer ID, and the job runs twice as fast. Nothing about the work changed. Only the order did.

A mixture-of-experts model has the same bug inside every decode step. Each layer holds many small blocks of weights, called experts. Each token uses only a few of them. Qwen3-30B-A3B has 128 experts per layer, and each token picks 8. But a server runs many requests together as one batch. On each step it must read every expert that any request in the batch picked.

That read is the expensive part. Producing one token per request is limited by how fast the GPU reads weights from memory, not by arithmetic. A random batch of 32 Qwen requests touches about 94 of the 128 experts in a layer. Code prompts favor one set of experts, French prompts another. Mix them, and the step reads both sets.

The fix: before an offline batch job, sort the prompts so similar ones run together. Then each step reads fewer experts. Hugging Face measured this on unchanged vLLM and got 5 to 17% more throughput.

The Fix: Group Similar Prompts Before the Batch Job

New to how a model produces text one token at a time? Start at AI basics → Tokens and sampling.

Why does the order of a list change what the GPU reads?

Because the server fills its batch from the front of the list. vLLM admits waiting requests first come, first served. So neighbors in your list end up decoding side by side. Sort the list, and each batch is mostly one topic.

Topics really do prefer their own experts. In one layer of Qwen3-30B-A3B, a topic's 32 favorite experts took 68 to 80% of its routing decisions. Even routing would give them 25%.

How much does one expert cost?

About 2.9 microseconds per layer on an AMD MI300X. That is the time to read about 9.4 MB of expert weights. The MoE layer's time rose in a straight line with the number of distinct experts the batch touched.

Why was the measured gain smaller than the reads saved?

Sorted Qwen batches read 19 to 25% fewer experts, which predicts a 20 to 27% speedup. The measured gain was about half that. Only the expert layers got cheaper. Attention, sampling and scheduling cost the same as before.

Do I need topic labels?

No. Embed each prompt with a small embedding model, cluster the vectors with k-means, and sort by cluster. On unlabeled chat data, this still gave Qwen about 5% more throughput.

What This Means for You

When this matters. You run offline jobs on a mixture-of-experts model that you host yourself. Think evals, synthetic data, labeling, summarizing a backlog. Qwen3 MoE, gpt-oss, DeepSeek and Mixtral are all this kind of model. The job generates many tokens per prompt, and nobody waits on any single answer.

How it affects you. The win is GPU hours. A ten-hour job on a rented node can finish in about nine. Your outputs will also shift a little. In the Hugging Face run, 92% of greedy outputs matched between sorted and shuffled order. Two shuffled runs matched each other 90% of the time. That drift is ordinary batch nondeterminism, the effect from How the Same Model Gives Two Different Answers. Sorting does not add new drift.

The gain also depends on where your batch size sits. The simulation in code_example.py shows the shape: zero at batch 1, largest at small and middle batches, fading once a one-topic batch reads nearly every expert.

What to do about it. Today, look at your next offline job's input file. If it already has a dataset, task or language column, sort by it before you submit. That costs one line. If it has no labels, add the clustering step below. Then run the A/B test in Implementing It before you trust any percentage, including the one in this article.

Implementing It

The change.

Three roles touch this. Whoever builds the input file groups the prompts. Whoever runs the job submits them in that order and restores the original order. Whoever owns the cluster checks the gain is real. The serving side changes nothing for an offline job.

Role 1, grouping. Use labels when you have them. Otherwise, cluster embeddings. The source used Qwen3-Embedding-0.6B and 16 clusters:

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans

emb = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B").encode(
    prompts, normalize_embeddings=True, batch_size=64)
labels = KMeans(n_clusters=16, n_init="auto", random_state=0).fit_predict(emb)
order = np.argsort(labels, kind="stable")  # stable keeps file order inside a cluster

Keep the embedding model small. It runs once over the file, before the expensive model loads.

You may not be able to sort the whole file at once, for example when you stream it in pieces. Then sort inside each piece. At batch 32, sorting within chunks of 128 prompts gave +11.7%, and chunks of 256 gave +14.8%, against +17.3% for a full sort. Make each piece several batches long, so every batch can fill from one cluster.

Role 2, the batch job. vLLM returns outputs in the order you submitted. So submit in sorted order, then put each output back where its prompt came from:

# before
outs = llm.generate(prompts, params)

# after
outs = llm.generate([prompts[i] for i in order], params)
results = [None] * len(prompts)
for pos, i in enumerate(order):
    results[i] = outs[pos]

Skip the restore step and every downstream join is wrong. You will see it as answers attached to the wrong question, not as an error.

Your batch size is max_num_seqs on the LLM constructor. It decides where you sit on the curve above, so record it with every result.

Role 3, measuring. Run the same prompts in both orders and compare tokens per second. Force equal output lengths so only the order differs:

import time
params = SamplingParams(temperature=0, max_tokens=256, ignore_eos=True)

def tok_per_s(batch):
    t = time.perf_counter()
    outs = llm.generate(batch, params)
    return sum(len(o.outputs[0].token_ids) for o in outs) / (time.perf_counter() - t)

Alternate the runs: shuffled, sorted, shuffled, sorted. A single pair can be skewed by a warm cache. The author's full harness is vllm_bench.py in the sort-your-prompts repository on GitHub. It had no license file when this was written, so read it for the method.

How you know it worked.

The sorted runs should beat the shuffled runs on every repeat, not just on average. On a mixture-of-experts model with labeled topics, expect something near 10%. Then run the same test on a dense model of similar size as a control. It should show no gain, because a dense model reads every weight on every step. If the dense model "improves" too, your measurement is noise. Last, compare outputs. The share that differs between sorted and shuffled should match the share that differs between two shuffled runs. If it is much higher, check the restore step.

When Sorting Prompts Is the Wrong Tool

When someone is waiting. Grouping means holding a request until similar ones arrive. In online serving, that queueing delay costs more than the step time it saves.

When the model is dense. Every weight is read on every step, whatever the order. A dense model like Llama 3 gains nothing.

When the work is mostly prefill. A long prompt with a short answer spends its time reading the prompt. Thousands of prompt tokens touch nearly every expert anyway. A classifier that emits one label per document is this case.

When batches are very large. Once even a one-topic batch touches almost every expert, there is nothing left to save. On unlabeled chat data, gpt-oss-120b at batch 128 showed no measurable gain.

When the model has few experts. A coarse model with 32 experts per layer saved 8 to 11% of reads, against 19 to 25% for 128 experts.

Before you adopt it, ask:

• Is my model a mixture of experts, and is this job decode-heavy?

• What batch size does my job really run at?

• Did a dense control show zero gain on the same A/B test?

Glossary

• experts — small blocks of weights in a layer; each token is routed to a few of them.

• mixture-of-experts — a model whose layers hold many experts and use only a few per token.

• decode step — one round in which every request in the batch produces its next token.

• batch — the set of requests the server runs together on each step.

• k-means — a clustering method that groups vectors around a chosen number of centers.

• batch nondeterminism — small output changes caused only by which requests shared a batch.

Further reading

• Sort your prompts: a free speed-up for MoE batch inference — Hugging Face blog (Abdennacer Badaoui)

• Abdennacer-Badaoui/sort-your-prompts — GitHub, the author's code

• Qwen3-30B-A3B model card — Qwen team, Hugging Face

• Mixture of Experts Explained — Hugging Face blog

Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

#AI, #LLM, #Machine Learning, #Python, #Performance