Testing tomorrow’s inference stack on today’s traffic

Author: Angelo Capossele

How do we test a new routing policy, cache strategy or serving configuration on real traffic without making customers depend on an unproven deployment? Shadow traffic gives us a way: we copy live requests to a candidate deployment while the primary continues answering customers.

This lets us explore changes using the mix of short questions, long conversations and generation patterns our service actually receives. For us, it also addresses a practical constraint: we do not retain customer prompts, so there is no stored collection of conversations to replay.

The candidate may have much less capacity than the primary, so copying every request is not always practical. A mirrored slice runs under different conditions from production. Interpreting its performance requires understanding the workload and tracking how much selected traffic completed and met our performance targets.

What we can learn from shadow traffic

The evidence we need depends on what we want to learn. Here are three examples of the decisions shadow traffic can help us make.

Can another serving configuration handle this mixture better?

When a short question arrives alongside a long conversation, we care about how quickly the new response starts and how smoothly the ongoing response continues. We can measure time to first token (TTFT), inter-token latency (the gaps between generated tokens), and throughput for each request group under the same compute budget. If short requests dominate, improving their latency may justify some slowdown for long conversations, provided those still meet acceptable limits. Looking at each group makes that trade-off visible.

Attributing a performance difference to the configuration requires a baseline under comparable conditions. Comparing the live primary with a smaller shadow deployment also changes the workload and capacity.

Would different cache management or routing help?

Routing and cache policies can save repeated computation, but introduce other costs. For example, llm-d’s cache-aware routing balances prefix reuse against server load: sending a request to a server with the right cache can save computation, but may mean waiting behind other requests. Cache offloading keeps KV state in CPU memory or storage after it leaves GPU memory. Retrieving it adds transfer overhead, so the benefit depends on how often it is reused and how much computation it saves. Shadow traffic lets us test whether these changes reduce response times or serve more traffic within our budget.

Would another model serve these requests better?

We can compare task correctness, structured-output validity or tool-call behaviour alongside latency and cost. The primary answer is not automatically correct, so quality needs task-specific checks, known outcomes or a suitable evaluator. Record decoding settings and compare scores across enough requests to estimate variability; repeat a subset when small differences could change the decision.

These checks assess responses to individual requests. Subsequent turns still follow the primary’s answers, so the candidate is evaluated within a conversation it did not shape. Its own answer might lead a user to ask another question or an agent to call a different tool. Evaluating those consequences requires an interactive evaluation or a controlled rollout. Tool calls emitted by the shadow model should be scored without executing them against production systems; if execution is part of the test, use an isolated environment.

Where the experiment runs

For any of these comparisons, the copied requests need to exercise the components we want to evaluate. Testing a routing policy requires copies to pass through the candidate router, while testing an engine setting may only require sending them directly to an inference server.

Once we have chosen that path, the control plane defines the experiment: candidate configuration, selection policies and evaluation criteria. The workload plane applies those decisions to each request, copying selected traffic, enforcing admission limits and running inference. This separates experiment setup from the work performed on the request path. The diagram shows how these responsibilities connect, even when they span several services. When using a separate shadow baseline, that deployment follows the same pattern as the candidate path shown here.

The capture boundary in the diagram is the point at which we copy a request. Its position determines which processing has already happened before the primary and candidate paths separate:

For experiments on components downstream of the gateway, capture after authentication and normalization is a useful starting point. Move capture earlier when that earlier processing is itself under test.

At the chosen boundary, we may be able to use an existing copying mechanism. For example, llm-d’s Gateway Mode supports mirroring through its gateway integration. From there, copies can go directly to a candidate endpoint or through an independent receiver that enforces admission limits, consumes streams and measures TTFT and gaps between streamed chunks. The receiver adds a service to operate, but forwarding an HTTP request alone does not provide those measurements.

Whichever delivery path we choose, the primary should finish without waiting for the candidate’s result. Copying still consumes shared CPU, memory and network capacity, so bound the size and number of in-flight copies and their forwarding time. When a limit is reached, skip shadow work rather than block the primary. If selection can use headers or other available metadata, decide before allocating an additional body copy.

Even without retaining prompts, the shadow system processes customer content in flight. The receiver and candidate process prompts transiently, and a quality evaluator may also need the primary and candidate outputs. This explains the diagram’s primary-to-evaluation connection; performance comparisons can instead use timings, token counts and outcomes. The same retention rules must cover receivers, candidates, evaluators, logs and traces, with explicit lifetimes for reusable cache state, including offloaded state.

Choosing what to copy and what to run

With capture and delivery in place, the next decision is how much work reaches the candidate. To limit testing cost, Baseten’s testing guidance suggests starting with a random sample of production traffic. Applying that approach requires two decisions: sampling chooses which requests are eligible; admission decides which can run, and when.

Sampling

The earlier examples call for different samples. For a cache study, start with whole conversations when usable IDs exist. Random requests are a simple starting point for broad request-level checks; selection by group can give rare tasks enough coverage. The cache case shows why the choice matters: if the first request we copy is turn five of a conversation, its prompt may include all four earlier turns, but the candidate may need to recompute a history the primary already has cached. Copying from the first turn gives it a chance to build that cache, although admission can still drop turns and cache entries can be evicted. The main options are:

Whole-conversation sampling works best with a stable, tenant-scoped ID supplied by the client. An API key can cover many conversations, and different conversations may share a prompt prefix. Neither an API key nor a shared prefix identifies a conversation on its own. If grouping is inferred from request content, report it as approximate.

Selecting by request group can change each group’s share of the sample. Report groups separately or weight their results by their usual traffic share, but weighting cannot undo changes to queueing, batching or cache behaviour during the test. Choose test windows that cover the demand relevant to the decision, including peak periods when evaluating capacity. A lightly loaded candidate may help compare answer quality but tell us little about queueing under production load.

Admission

Selecting the traffic does not guarantee capacity to run it: even a well-chosen sample may exceed capacity during a burst. Dropping requests loses part of that sample; queueing retains more of it, but delays execution and can change concurrency and cache reuse. Admission controls determine how much work can run and how long selected requests may wait:

Start with an explicit limit on active work. Add a short bounded queue when retaining selected requests matters more than preserving their arrival timing; use immediate admission when delay would undermine the experiment. If queueing is used, bound both its size and waiting time so overload does not become an ever-growing backlog.

Admission also affects the evidence we collect. Consider three selected requests with 1,000, 1,000 and 18,000 input tokens. If only the two small requests complete, we have served two thirds of the requests but just 10% of the selected input tokens. Their responses might be fast, yet most of the selected input volume never completed. Request share is not compute share either: generation length and cache reuse change the work required.

Track observed, selected, admitted and completed requests, including how many complete within the test’s performance targets. Request and token counts offer complementary views of coverage. Count input and output tokens separately, using a common tokenizer for cross-deployment coverage comparisons or explicitly labelling model-specific counts. Latency measured from the capture boundary includes forwarding and admission delays; label receiver and engine timings separately.

An example: comparing routing policies

Suppose we want to test whether cache-aware routing improves latency for long conversations. We run the current router and the proposed router on separate pools with the same GPU hardware and capacity, identical model and engine settings, and the same warm-up procedure. Both receive copies of the same selected conversations as requests arrive, with the same admission limits.

We compare long-context time to first token alongside the share of selected requests completing within our performance targets. We also check the other request groups against their agreed regression limits. The candidate is promising if it improves the target workload within those limits; lower latency caused by dropping more difficult requests would not establish that.

How replay complements shadow testing

Running both shadow deployments requires additional capacity. Testing configurations at different times introduces changes in live demand. Replay lets us repeat a chosen request pattern while changing the serving configuration, which is useful for investigating regressions or comparing routing, caching and admission policies.

Not retaining customer prompts does not rule this out. For example, Dynamo’s trace-replay workflow records arrival times, token lengths and shared-prefix relationships without storing the original text, then uses those records to construct synthetic requests for AIPerf. These reproduce selected workload properties, but cannot assess answers to the original questions. Synthetic content can also produce different execution behaviour, even when token lengths match.

We can use replay to narrow down configurations under controlled conditions, then shadow traffic to assess promising candidates on incoming requests. If the two short requests finish quickly while the long request never runs, their latency alone cannot tell us whether the configuration is better. Shadow testing becomes useful evidence when we can explain both the performance we measured and the work left out.

If you have any questions, or simply want to chat. Please reach out: angelo@boundless.network.