Optimizing inference speed and costs: Lessons learned from large-scale deployments

Optimizing inference speed and costs: Lessons learned from large-scale deployments

How can teams reduce inference latency without massive costs?
Achieving faster inference doesn't always mean paying more for a bigger cluster. At Together AI, we’ve seen teams that consistently deliver both low latency and low cost share these key habits:

By excelling in these areas, your cluster can provide faster responses while simultaneously reducing the cost per token.

Why inference cost efficiency matters

AI products are getting more competitive by the week — and user expectations are rising just as fast.

For leading AI-native companies — like Cursor, who needs massive throughput without compromising speed, and Decagon who needs real-time responses despite unpredictable traffic patterns — the pressure is the same everywhere:

Across customers, we consistently see the same imperative: deliver sub-second responses, without doubling the GPU bill. The good news? You don’t  need exotic architectures or hundreds of extra GPUs to maintain inference cost efficiency.

Most teams get meaningful wins by optimizing how their inference runs, not purely how much hardware they buy.

How inference optimization works

Here are the levers that reliably move both speed and cost in the right direction.

1. Start at the model level: quantization and distillation

Quantization
Dropping precision (FP16 → FP8 → FP4) makes the model lighter on memory and faster to run — with virtually no quality loss when done well, like how we do it here at Together.

This unlocks:

We’ve seen in many production deployments that FP8 or FP4 quantization delivers 20–40% throughput improvement, without harming output quality.

Distillation
Not every workload needs the full weight of a frontier model. Distillation trains a smaller model to mimic a larger one, preserving reasoning patterns while cutting compute cost dramatically.

DeepSeek-R1 is a great example. Its distilled variants are fast, lightweight, and still excellent at reasoning — making them perfect for:

You can see how teams deploy R1 and its distilled variants securely on Together AI in this post.

Distilled R1 variants deliver a quality-to-latency ratio that’s extremely compelling for production workloads — often enabling 2–5× lower cost at similar quality bands for many tasks.

Together, quantization and distillation offer some of the largest cost reductions available before touching hardware or cluster architecture.

2. Reduce network latency at the edge (regional inference proxies)

Sometimes the biggest latency win isn’t compute, but geography. Even with extremely fast models, network distance is often the slowest part of the request path.

Dropping a lightweight proxy in the same region as your inference cluster cuts out long round-trip paths before generation even starts.

This alone can shave 50–100 ms off TTFT, and make tail latency far more predictable.

3. Reduce unnecessary compute (memory stalls, KV inefficiencies, fragmentation)

Most models aren’t slow— the pipelines around them are. So your GPU spends a lot of time doing nothing and just… waiting. The biggest culprits tend to be:

At Together AI, we’ve run benchmarks across Llama, Qwen, Mistral, and DeepSeek families (highlighted in our fastest inference for the top open-source models blog) which show that kernel fusion, smarter MoE execution, streamlined tokenization, and better scheduling can eliminate wasted time, unlocking faster responses and higher throughput.

4. Use the right decoding optimization (MTP, speculative decoding, draft models)

Decoding is where a lot of time gets lost — and also where some of the easiest wins live.

When tuned properly, these techniques often deliver 20–50% faster decoding and significantly higher throughput per GPU.

5. Pick the right hardware for your workload (and use the right parallelism)

With a new hardware type that comes out every year or so, hardware choice increasingly shapes both cost and latency.

But to fully benefit from this hardware, large models need to be split and scheduled intelligently across devices. That’s where parallelism strategies come in:

Teams running billions of tokens per day typically see a clear drop in cost-per-token when moving heavy workloads to NVIDIA Blackwell-class hardware with the right parallelism strategy.

6. Dynamically shift GPU capacity across endpoints

Traffic is rarely evenly distributed across all services. For example:

This is especially valuable for customers with mixed workloads: coding, chat, RAG, batch, and long-form generation. Together AI allows customers to update capacity on their endpoints via a simple API call.

What you can do with better inference optimization

Teams that implement these optimizations unlock:

Getting started

Here’s a practical, low-friction way to begin:

  1. Measure your baseline (TTFT, decode TPS, TPM/GPU, network RTT)
  2. Deploy a regional proxy if requests originate far from your inference cluster
  3. Enable adaptive continuous batching and monitor GPU utilization
  4. Turn on MTP or speculative decoding depending on workload
  5. Rebalance endpoints by dynamically shifting GPU capacity as traffic changes

FAQ

Do throughput optimizations increase latency?
No. Continuous batching + fused kernels let you raise throughput while lowering latency.

Is NVIDIA Blackwell only for huge workloads?
No. Any workload with meaningful concurrency or long contexts benefits from its bandwidth and memory improvements.

How can I tell if my GPUs are under-utilized?
Look for low decode TPS, small active batch sizes, or long gaps between token generations.