Shadow traffic: how we test inference changes on real requests without storing prompts and impacting users

By: Angelo

We're hiring engineers to work on problems like this. Open roles →

TL;DR

• Shadow traffic sends a copy of live requests to a test system. The live system still answers every customer, and the test system's answers are only measured.
• It's how we test on real traffic. We don't keep customer prompts, so there's nothing stored to replay.
• The test system usually sees only part of the traffic. A result is only meaningful if we also know which requests it skipped.
• Replay of recorded traffic shape (timing and length, never text) narrows the options, and shadow traffic tests the finalists.

Why shadow traffic?

Every change to an inference stack is a bet on how it behaves with real customers, whether it's a routing rule, a caching approach, a server setup or a new model. Benchmarks can't recreate the real mix of short questions, long conversations and long answers.

Shadow traffic lets us test on that mix safely. Each incoming request is copied to a candidate, the version we're testing. The primary, our live system, answers the customer. The candidate's answer is measured and thrown away.

The live system answers the customer. A copy goes to the test system, which is measured and never answers anyone. Only results, never customer text, go back for review.

The catch is capacity. The candidate is usually much smaller than the primary, so it can't run every copy. Most of this post is about reading results from a partial slice correctly.

What can we learn from it?

Is a new server setup faster?

For each type of request, on the same compute budget, we measure time to first token (how long before the answer starts), the gaps between words as it streams, and total throughput. Splitting by type shows the trade-offs: speeding up short questions may be worth slowing long conversations slightly, within limits.

A fair comparison needs a baseline, a second test system running the current setup under the same conditions. Comparing against the full-size primary changes too many things at once.

Would better caching or routing help?

When a model reads a prompt, it saves some of that work in a KV cache, so repeated text such as earlier conversation turns doesn't have to be read again. Cache-aware routing (as in llm-d) sends requests to the server holding the right cache, but that server may be busy. Cache offloading moves the cache to regular memory or storage instead of deleting it, but fetching it back takes time. Shadow traffic shows whether either trade pays off.

Would a different model answer better?

Here we also check correctness, valid structured output and well-formed tool calls. The primary's answer isn't assumed to be right, so we use task-specific checks or an evaluator. We record decoding settings and test enough requests to see natural variation.

What it can't tell us

Shadow traffic judges single answers. In a conversation, each message responds to the primary's previous answer, so the candidate joins a conversation it didn't shape. Had it been answering, the user might have asked something else. Measuring that takes an interactive evaluation or a controlled rollout.

Tool calls from the candidate are scored but never executed against production systems. If a test needs them to run, it uses an isolated environment.

Where do we copy requests?

Choosing the capture point

The copy has to pass through whatever we're testing. A request goes through a gateway (which checks who sent it and standardizes it), then a router (which picks a server), then the engine (which runs the model). The earlier we copy, the more of that path the test covers.

Our default: copy right after the gateway, unless the gateway itself is what we're testing.

Never slowing down the customer

Copies go either straight to the candidate or through a separate receiver that limits and times them. Either way, the rule is fixed: customers never wait for the test. We cap how many copies are in flight, skip copies when a limit is hit, and decide whether to copy before duplicating a request.

Not storing prompts doesn't mean the test never sees them. Copies are processed in flight, so every part of the test (receivers, candidates, evaluators, logs) follows the same retention rules, and saved cache data has an explicit expiry. Speed comparisons need only timings and token counts, never content.

Which requests do we copy?

The sample shapes the result. If we copy the fifth message of a conversation without the first four, the candidate has to reprocess history the primary already cached, and looks slower than it is.

Our default: whole conversations for caching tests, random requests for broad checks, and sampling by type for rare cases. Two cautions apply. Conversation IDs should come from the customer's app, because an API key or shared opening text doesn't identify a conversation. And capacity tests must include peak hours, because a quiet candidate says little about real load.

Which copies does the candidate run?

Even a good sample can overwhelm the candidate during a spike. Dropping extra copies keeps timing realistic but loses part of the sample. Queueing keeps more of the sample but changes timing and caching.

Our default: a clear cap on work in flight, plus a short, time-limited queue only when keeping the sample matters more than realistic timing.

How do we know the results are trustworthy?

Say the candidate is sent three requests of 1,000, 1,000 and 18,000 tokens, and only the two short ones finish. It completed two thirds of the requests, but only 10% of the text we meant to test. Fast answers mean nothing if the hard requests were skipped.

So every experiment reports:

• The funnel: requests seen, selected, admitted, completed, and completed within our speed targets
• Requests and tokens, since each shows coverage differently
• Where time went, with forwarding and queueing delay separated from time in the engine

Worked example: comparing two routers

Question: does cache-aware routing make long conversations faster?

Setup: the current and new routers run side by side on identical hardware, model settings, warm-up and admission limits, and both receive copies of the same conversations.

Verdict: the new router passes if long conversations start faster and every other request type stays within agreed limits. Looking faster because it dropped hard requests is a fail.

Where replay fits

Shadow testing costs extra capacity, and runs at different times see different traffic. Replay avoids both by rerunning the same traffic pattern while only the setup changes.

We can replay without storing prompts. Dynamo's trace-replay records only arrival times, lengths and shared opening text, then generates stand-in requests for the AIPerf benchmark. Stand-ins can't judge answer quality and may behave differently from real text, so we use replay to narrow the options and shadow traffic to test the finalists.

Outlook

A shadow test is only useful evidence when we can explain both what we measured and what was left out. Questions we're still working on:

• How do we sample whole conversations when apps don't send a conversation ID?
• How do we measure a model's effect on later messages without a live rollout?
• When should we queue to keep the sample, and when should we drop to keep timing honest?

If you want to work on these, we're hiring →