Reducing Latency in LLM Applications

Spinning up an LLM application is relatively easy nowadays. Keeping it useful and responsive is the harder part. An answer can be correct and still leave users frustrated if they spend too long waiting for it.

Some experiments at work prompted me to think more carefully about where that waiting comes from. This post covers practical ways to reduce latency: streaming, model selection, fast mode, reasoning effort, prompt caching, and specialized decision models. Each comes with trade-offs to evaluate in your own application.

The goal is to reduce the time until users get something useful, while checking that the application still meets its quality requirements.

Measure where the waiting happens

To make the discussion concrete, consider a hypothetical Q&A bot that first identifies which document collections to search, retrieves relevant excerpts, and then generates an answer.

Chatbot Request Flow

The user has to wait for routing and retrieval before answer generation can start. Switching the answering model might help, but it leaves those earlier steps untouched.

I would track three things:

For this sequential workflow, the wait for the first answer text consists of routing time, retrieval time, and the answering model's time to first visible text, plus application and network overhead.

Be explicit about where your timer starts. Time to first token for a model call and time to first answer text for the whole application are different measurements. For reasoning models, also distinguish visible answer text from reasoning or other stream events.

Measure completion time too. A response that starts quickly but takes too long to finish can still be frustrating. For a router, completion is particularly important: the application needs a complete, valid selection before it can retrieve documents.

Stream answers as they arrive

If you wait for the entire answer before displaying it, users experience the full generation time as an empty wait. Streaming lets them start reading while the rest is generated. OpenAI's streaming guide explains how to receive incremental output.

Streaming changes when you deliver the answer. It does not inherently make the model compute its first answer token faster, and it cannot remove the routing or retrieval work that precedes generation.

Streamed Answers vs. Non-Streamed Answers

For our Q&A bot example, I would stream the final answer and expose actual progress during the earlier stages: “Finding relevant documents,” then “Preparing your answer.” Progress feedback is useful, but a loading indicator should not be counted as an improvement in time to first answer text.

For a decision that must finish before the application acts, the complete result remains the important milestone. Structured data can arrive incrementally, but receiving part of an object does not necessarily mean it is ready to use.

Enabling Streaming on the Chatbot

Try a faster model

Smaller models generally require less computation per token and often run faster. OpenAI identifies model size as an important factor in its latency optimization guide. Parameter count alone, however, is not a reliable prediction of the latency your application will experience.

A practical starting point is to test a faster model on a narrow task. Our router selects document collections; it does not need to write the final answer. It is worth evaluating whether a faster model can make that selection reliably.

Build a test set of representative queries and the collections each query needs. Compare the baseline and candidate models on both decision latency and routing quality.

For this example, I would prioritize recall of required collections. Missing a collection could leave the answering model without necessary evidence. Selecting unnecessary collections matters too: it can introduce irrelevant context and additional work.

Neither mistake automatically tells us whether the final answer will be wrong. Evaluate final answers separately, including whether the bot abstains when the available evidence is insufficient.

The decision is whether the faster model meets the task's requirements. A speedup is only useful if the resulting behavior is acceptable.

Fast mode is another option

If you need the capabilities of your current model, a faster serving tier may be worth testing. Both OpenAI and Anthropic document fast modes for supported models at premium pricing.

This adds cost to the decision. Instead of changing models or reducing reasoning effort, you may be able to pay more for faster processing on the selected model.

Check what the provider's speed claim measures. Higher output tokens per second does not necessarily mean an equivalent improvement in time to first visible text, and neither translates directly into the same speedup for the whole application.

Lower reasoning effort when the task allows it

Reasoning models can spend tokens working through a task before, or between, visible outputs. That work can contribute to latency.

The reasoning-effort setting guides how much reasoning the model applies. OpenAI's reasoning documentation describes lower effort as favoring speed and lower token usage. It is not a hard token budget, and the model can still adapt its reasoning to the task.

This differs from max_output_tokens, which places a ceiling on generated tokens, including reasoning and visible output. Hitting that ceiling can leave an incomplete response, potentially before any answer text appears.

For the router, the hypothesis is straightforward: lower effort may produce a valid selection sooner while maintaining acceptable recall. Test that hypothesis by holding the model and prompts fixed, changing the effort setting, and recording latency, reasoning-token usage, and routing quality.

Lower effort does not guarantee a faster response on every request or necessarily reduce quality on every task. Whether the trade-off is worthwhile needs to be measured.

Reuse prompt computation with caching

Before generating an answer, the model processes the input. Prompt caching lets it reuse intermediate computation for an unchanged prefix instead of processing that prefix again.

OpenAI's prompt caching documentation describes the saved state as key-value, or KV, tensors. These are not cached answers. The model still processes new input and generates a new response.

For our router, a useful prompt order is:

1. Fixed routing instructions.
2. Output format and document-category descriptions.
3. The current user's question.

For answer generation, put stable instructions before retrieved excerpts and the question. Retrieved excerpts can extend a reusable prefix when their content and order match across requests. Being frequently retrieved is not enough by itself.

Utilize Prompt Caching

Eligibility also matters. Current OpenAI documentation specifies a minimum cacheable prefix of 1,024 visible input tokens for GPT-5.6 and later; requirements and controls vary by model and provider.

Verify reported cached-token usage instead of treating a faster second request as proof of a cache hit. Then compare time to first visible answer across repeated requests. A short router prompt may have little or no caching opportunity, while an answering prompt with substantial reusable context may offer more.

Consider specialized models for decisions

If you use a generative model only to return a category, probability, or score, a decision-oriented model is another candidate.

Jev returns typed decisions and probabilities rather than generated prose. The OpenAI Decisions API supports predicates, choices, and scores, including request-routing use cases.

For our router, which may select multiple collections, one possible formulation is to evaluate whether each collection is needed. Compare that with the generative router on the same queries and labels.

OpenAI advertises approximately 10× faster responses for Decisions compared with Responses. That is a vendor claim, not a result established for this workflow. Measure complete-decision latency and quality in your own application.

These endpoints do not replace free-form answer generation. If you need a written explanation, use a component that can generate it and include that extra work in your measurements. You can still record inputs, decisions, probabilities, and configuration without a generated rationale.

Check the trade-off before keeping the change

Start with a baseline for your application and change one variable at a time. Test the router independently, test answer generation against fixed retrieved context, and then check the full workflow. This helps distinguish a model-level improvement from one that actually shortens the user's wait.

Repeat requests and interleave configurations so one configuration is not measured only during a quieter period. Start with controlled concurrency, record cache status and token usage, and keep errors and incomplete responses in the results. Compare typical latency and variability; a single fast request is weak evidence.

Define quality requirements before selecting the winner. For routing, that means coverage of required collections and unnecessary selections. For answers, check correctness, support from the documents, and appropriate abstention. Track cost alongside both.

A faster model, lower reasoning effort, or a cache hit is a means to an end. Keep the change when it improves the user's wait and the application still does the job it is supposed to do.