Speech-to-Speech vs STT-LLM-TTS: Clear Choice for AI Voice Agents

You've had this moment. You watch a speech-to-speech demo: the agent answers almost instantly, and it sounds like a person, not a script reading itself aloud. And you start wondering why you're still building an AI voice agent the old way, stitching speech-to-text to a language model to text-to-speech, like some assembly line nobody asked for.

That demo sells you one belief: latency is the whole game, and speech-to-speech wins it. Fewer hops, less lag, more feeling. Case closed, right?

But not quite. The speech-to-speech vs STT-LLM-TTS decision isn't about which one feels smoother in a five-minute demo. It's about what happens on call 4,000, 6 months in, when your compliance team asks what the agent said, and you can't produce a clean answer.

Speech-to-speech vs STT-LLM-TTS is one of those voice agent architecture calls you don't get to take back cheaply. Pick wrong, and you're not tweaking a setting. You're re-architecting.

I've built with both. And I've learned the hard way that the demo is not the deployment. That gap, between what looks great on stage and what survives production, is where most voice agent projects die. It's the gap we spend most of our time closing when we build voice agents for clients.

Two Ways to Turn a Voice Into a Voice (And a Third Nobody Talks About)

Every voice agent boils down to one of three shapes. Get this sorted first, because the whole speech-to-speech vs. STT-LLM-TTS debate rests on it.

The cascaded setup, STT-LLM-TTS, runs three stages back-to-back. Speech-to-text turns the caller's voice into words. The language model reads those words and decides what to say or do. Text-to-speech turns the reply back into audio. Turn-taking and barge-in wrap around the outside like guardrails, deciding when the caller's done talking and what happens if they cut the agent off mid-sentence. 

Text is the currency this whole system trades in. Every backend integration, every compliance filter, every log line, it all happens at that text layer.

Speech-to-speech skips the middleman entirely. One neural network takes audio in and puts audio out, no text pit stop along the way. That matters because tone, pacing, hesitation- the stuff a transcript throws away- survive the whole loop. The agent doesn't just hear the words you said. It hears how you said them.

The hybrid nobody planned for

Then there's the hybrid model, which most serious platforms have quietly landed on. The LLM runs the dialog, but STT and TTS remain swappable components you can upgrade without touching anything else, and the backend logic lives in its own controlled layer. Less elegant than pure speech-to-speech. More useful in practice.

So the speech-to-speech vs STT-LLM-TTS choice isn't really a choice between two things. It's three, once you count the compromise. And I'd argue the hybrid is where most production builds quietly end up, even when nobody planned it that way.

Cascaded vs speech-to-speech vs hybrid, side by side

If you want the full picture of how these pieces stack together before you commit to one, our guide to the AI voice stack covers it end-to-end. And it's worth knowing early that not every voice bot is actually a voice agent, because that gap shapes which of these three architectures even makes sense for you.

Here's the thing though. All three of these live or die on one number. And it's smaller than most people guess.

The Latency Budget Nobody Escapes

Humans don't wait around when they talk. The natural gap between one person finishing a sentence and the other jumping in sits around 200 milliseconds. That's the rhythm your brain expects. After 500 milliseconds, a conversation starts to feel sluggish, like the other person's distracted. Cross 700 milliseconds and something worse happens. 

Callers stop treating the agent like a conversation partner and start treating it like a phone menu. They repeat themselves. They get short. Task completion drops, and people just hang up.

This is where the old-school cascade fell apart. Picture the caller speaking for three seconds. Then speech-to-text takes 280 milliseconds to transcribe it. The language model chews on that for two to four seconds. 

Text-to-speech needs an additional 500 milliseconds to convert the reply into speech. Add it up, and you're at three seconds plus before a single word comes back. Nobody's sticking around for that.

The fix wasn't a faster chip. It was rethinking the whole flow so stages overlap instead of waiting in line. Once every stage streams into the next, your perceived delay stops being the sum of all the parts. It becomes whichever path is slowest.

Perceived latency ≈ max(concurrent stage paths), not their sum.

That one shift is what makes real-time voice AI possible at all.

Good teams aim for a P95 under 500 milliseconds. In practice, most production stacks land closer to 600. Still fast enough to feel human, if every stage does its job.

But here's what surprised me when I actually measured this stuff. The latency that kills conversations rarely hides where teams go looking for it. It's not the STT engine. It's not the TTS voice. If your voice agent latency feels off, the real culprit is somewhere else entirely, and we're getting there next.

Inside the Cascaded Pipeline: What Actually Happens in One Turn

Let's walk through an actual turn, because this is where speech-to-speech vs. STT-LLM-TTS stops being an abstract debate and becomes an engineering problem.

Caller says something. A voice activity detector flags when speech starts. Streaming speech-to-text starts spitting out partial transcripts almost immediately, roughly every 50 milliseconds, and those partials keep getting revised as more audio comes in. Once the system decides the caller's actually done talking, not just paused to breathe, the language model kicks in and starts generating a reply. 

Text-to-speech doesn't wait for that reply to finish either. It starts speaking the first few words while the model's still working out the rest. And the whole time, barge-in sits ready in case the caller cuts in mid-sentence.

Here's the part people miss. Every stage in this chain must emit output before the previous stage completes. If you don't build it that way, you're waiting on each step one after another, and you're back to three seconds plus of dead air before anything comes back.

I hear this a lot: "it's just three APIs glued together." It's not. Anyone can bolt an ASR endpoint to an LLM to a TTS voice in an afternoon. 

The actual work is the orchestration around it. Tuning when VAD fires, deciding when a turn's really over, keeping the streams synced, handling interruptions, routing function calls at the right moment. You can swap your STT vendor in a day. Try swapping your orchestration layer, and you're rewriting half the system.

Three techniques keep this whole cascaded voice agent architecture under 500 milliseconds without falling apart.

Expert Tip: Run these three together, not in isolation. They cover each other's blind spots.
- Semantic completeness double-gating. Don't act on a partial transcript unless intent confidence clears 0.85 and the sentence actually looks finished. Otherwise you're firing on "can I get my..." before the caller says "balance."
- LLM cancel-and-restart. If the final transcript differs from the partial that triggered the model, kill that in-flight call and start over. Costs you 100-200 milliseconds of wasted compute time. Worth it. Just keep that cancellation rate under 3%, or you're burning latency for nothing.
- High-confidence tool prefetching. Once confidence crosses the 0.85 mark, fire the database or API call in parallel with the LLM's response, and then inject the result straight into the context. Saves you 200 to 400 milliseconds you'd otherwise lose waiting in line.

None of this works if your underlying components are weak to begin with, which is why choosing the right ASR model, LLM, and TTS voice actually matters more than most teams admit. Same goes for what the agent knows going in. A solid voice AI knowledge base makes half these gating decisions easier because the model isn't guessing.

But this whole chain has a weak spot baked into its design. And speech-to-speech exists specifically because someone got tired of it.

Inside Speech-to-Speech: What's Actually Happening Under the Hood

Here's where speech-to-speech gets genuinely interesting, and also where most explanations get lazy.

A text language model reads discrete tokens, chunks of words it's seen a billion times in training. Raw audio isn't like that at all. It's a continuous signal, tens of thousands of samples firing every second, no natural chunks to grab onto. 

So before any speech-to-speech model can "think" about what it just heard, something has to compress that wall of samples into tokens a language model can actually chew on. That something is the neural audio codec, and it's the real engine behind every S2S system you've demoed.

The flow looks like this. A waveform, usually sampled at 24kHz, gets pushed through convolutional downsampling. That produces continuous latent vectors. Those latents pass through a quantizer, and out comes a sequence of discrete audio tokens. Decoding just runs the whole thing backward.

The quantizer part is where it gets clever. Most codecs use residual vector quantization, RVQ for short, which sounds intimidating but isn't, once you see it laid out.

How RVQ actually works: the first codebook attempts to approximate your latent vector. It's not exact, so there's leftover error, the residual. The second codebook doesn't touch the original vector at all. It only quantizes that leftover residual. Then the third codebook quantizes whatever residual remains. Stack enough of these layers, and the reconstructed signal is just the sum of each codebook's contribution. Each layer cleans up what the last one missed.

Why bother with all this instead of just cramming in more codebooks? Frame rate. A lower frame rate means fewer tokens per second of audio, which means shorter sequences for the downstream transformer to process. 

Skip this, and you hit quadratic memory scaling the moment a conversation runs long. Nobody wants their voice agent's memory usage exploding because a caller kept talking for six minutes.

Different codecs make different trade-offs here, and honestly, this table tells you more than most vendor pitch decks will.

Mimi is worth sitting with. At 12.5Hz, its first codebook gets trained against a WavLM model, which forces it to carry pure semantic meaning. The remaining seven codebooks handle everything else: speaker identity, prosody, background noise. 

And here's the part I still find counterintuitive: when researchers trained Mimi using only adversarial loss, dropping the usual reconstruction math, the objective quality scores actually went down. But humans rated the audio as sounding more natural. Sometimes the metric and the ear disagree, and the ear wins.

Moshi takes this further. It's a 7-billion-parameter model built for real full-duplex conversation, meaning it doesn't politely wait for its turn. It processes the caller's audio and its own audio as overlapping streams, the way real conversation actually works.

It also runs an inner monologue, a text stream generated just ahead of the audio tokens, which lets it borrow the reasoning strength of a text LLM while cutting down on the kind of hallucinations you get from a model that's only ever "thought" in sound. A depth transformer handles the codebooks themselves, predicting the semantic one first and then all seven acoustic layers at once, in parallel, in the same timestep. 

That parallel prediction is exactly why S2S systems can hit time-to-first-audio under 200 milliseconds. It's genuinely fast.

Expert tip: if you're running any of this on-device, T-Mimi swaps out the convolutional decoder for a transformer-based one, cutting decoding latency from 42.1ms to 4.4ms. But the final two transformer layers and the last linear layers need to stay at full FP32 precision. Quantize those and your audio quality falls apart, no matter how fast the rest runs.

None of this is theoretical plumbing either. It's the reason a well-tuned S2S voice can sound genuinely human, which ties straight into what actually makes AI voice sound human in the first place.

So on raw feel, speech-to-speech wins the demo, hands down. Then why does production keep signing off on the pipeline instead? That's the part nobody puts in the sales deck.

The Four Questions Production Actually Asks

Demos answer one question: does this feel good for five minutes? Production asks four different ones, and none of them care about vibes.

What does it cost at scale?

Speech-to-speech bundles everything- transcription, reasoning, and voice synthesis- into one API call. That sounds simple until you look at the bill. Cost scales with both how long the caller talks and how long the agent responds, and it climbs even faster once your prompts get longer or the conversation runs for multiple turns.

A cascaded pipeline works differently. Each piece is billed separately, so you can actually see where the money goes. For a simple query, use a lighter LLM. Volume's high, switch to a cheaper ASR engine. Want a different voice, swap the TTS provider without touching anything else in the stack.

This isn't a pricing quirk that fixes itself next year. Even as speech-to-speech gets cheaper, and it will, the ability to tune each component on its own keeps the cascaded approach ahead for anyone running real volume. I'd bet on that gap holding for a while.

Here's what a five-minute call actually costs, based on Murf's own breakdown:

That's roughly ten times the cost, and this is a conservative estimate. Real contact centers aren't doing 100 calls a day. They're doing millions. Run those numbers, and the gap stops being a rounding error and starts being a line item someone gets fired over.

If you're trying to figure out what your own setup might run you, our breakdown on how much voice agents actually cost walks through it in more detail. And once volume becomes the goal, scaling AI voice agents the right way matters more than picking the flashier architecture.

Cost is the problem you see coming. Control is the one that shows up at 2 am when nobody's watching.

Can you actually control and debug it?

A cascaded pipeline hands you every stage on a plate. Transcripts, prompts, tool calls, outputs, all sitting there, observable. Something's off, you go find exactly which layer broke it and fix that one piece.

Speech-to-speech doesn't give you that. Understanding, reasoning, and generation are fused into a single loop. When it goes wrong, and it will eventually, you're left guessing whether the model misheard the caller, reasoned badly, or just synthesized a weird response. The control surface here is improving every quarter, but it's still narrower than what a cascaded voice agent architecture provides by default.

Voice control tells the same story. Speech-to-speech locks you into whatever voices the model ships with. OpenAI's Realtime API gives you around 10. Gemini Live offers 30 plus. A dedicated TTS engine? Over 150, with fine control of tone via SSML tags and pronunciation via custom lexicons.

This isn't a small detail if you're building for a brand voice or a regulated workflow. Getting your voice AI prompting right matters either way. Still, it matters more when you're also trying to run a multilingual voice agent across markets, where pronunciation control isn't optional. And none of this helps if you can't see what's happening in production, which is why a real monitoring playbook matters more than people think until they're the ones debugging a live outage.

In a regulated industry, control isn't a nice-to-have. It's the thing an auditor asks you to prove, and "the model just knew" isn't an answer they'll accept.

Can you audit and comply with what your agent said?

Here's where I get opinionated. If you're in healthcare, finance, or insurance, a cascaded pipeline should win this argument outright.

Text gives you a clean checkpoint before anything gets spoken. You can filter it, rewrite it, block it entirely, all before the caller hears a word. That's direct. That's auditable. You can show a regulator exactly what happened and why.

Speech-to-speech has to work harder to get to the same place. Parallel transcription running alongside the model, real-time audio scanning, guardrails baked into the prompt itself. All of it works, technically. None of it is deterministic. It's probabilistic control bolted onto a system that wasn't built with a checkpoint in mind, adding latency and complexity you didn't have before.

The field's catching up, though, and two approaches are worth knowing about. ALMGuard finds shortcut patterns in the audio itself and restricts changes to the specific frequency bands that jailbreaks exploit, which brought voice jailbreak success down to 4.6%. OmniGuard skips the external check entirely and reads the model's internal signals, scoring what it calls a U-Score across layers, achieving 93.09% accuracy on audio safety flags without adding any latency.

That's a good rule of thumb until you remember one thing. All of this assumes the transcript is accurate in the first place. Bad audio breaks that assumption before compliance even gets a say. Which brings up a question most teams don't ask until it's too late.

What happens when the audio is actually bad?

This is the cascaded pipeline's real weak spot, and it's not a minor one.

The LLM only ever sees what the STT engine hands it. Garbage transcript in, garbage reasoning out. And this hits hardest exactly where it matters most: entity extraction and tool calls, the moments where getting a name or number wrong actually breaks something.

A few specific ways this goes sideways:

• STT streams come out unpunctuated and in lowercase, stripping the exact formatting cues that named entity recognition depends on. Drop the casing alone, and you can lose over 40 F1 points on a standard NER model. That's not a small dip.
• Numbers and names are often spoken as words rather than digits, so "800-555" becomes "eight hundred five five fiv,e" and now your system has to reverse-engineer what should've been simple.
• Errors don't spread evenly either. They cluster on rare words, brand names, technical terms, exactly the spans carrying the most intent.
• Drop the signal-to-noise ratio to -10dB, and the character error rate climbs from 0.03 to 0.50. Exact-match answer recovery falls to 67%. That's a coin flip away from getting your caller's request completely wrong.
• Feed a corrupted transcript to an LLM, and it starts fabricating entities that were never said, somewhere between 3% and 12% of the time.

Speech-to-speech has a real answer here, and it's the strongest argument in its favor. Because it works directly with raw acoustic information rather than a lossy text transcript, it resists this kind of collapse. CORTIS-tuned spoken language models parse speech embeddings directly and beat cascaded systems under noisy conditions by up to 8.78 points of intent accuracy.

So neither architecture gets a clean win here. Two failure points remain, and both of them hide in the exact moments a human listener wouldn't even notice.

Turn-Taking and Barge-In: The Dark Art Nobody Talks About

Knowing when someone's actually done talking sounds simple. It isn't. This is one of the hardest problems in any voice agent architecture, and most teams underestimate it until their agent starts cutting people off.

Acoustic voice activity detection, the kind that WebRTC or Silero runs, works by reading 20-millisecond frames and checking for energy, spectral shape, and harmonics. It's fast. It runs locally. And it's completely deaf to meaning. Say "change my booking to..." and pause for 400 milliseconds while you think of the day, and acoustic VAD hears silence and fires the turn-end trigger right then. 

Cut off, mid-thought. Try to fix it by pushing the silence threshold up to 800 milliseconds, and now every completed sentence feels like the agent's falling asleep before it responds.

Semantic VAD is the smarter fix. Tools like Deepgram Flux or Gradium blend the acoustic signal with a running language model, so the system isn't just listening for silence; it's tracking whether the sentence sounds finished. Instead of one flat threshold, it outputs a probability array across multiple time horizons, forecasting the odds of silence at 0.5, 1, 2, and 3 seconds out. That's a genuinely different way of thinking about turn detection, and it's part of why semantic VAD has become the default for anyone serious about this.

There's a knob here called delay_in_frames, and it's the real trade-off between accuracy and speed. Each frame buys about 80 milliseconds of look-ahead. More frames, better accuracy, slower response. Fewer frames, faster response, more mistakes. 

And because the model's buffering audio to make that call, the transcript always trails what was actually said. When the turn ends, the last few words are still sitting in the buffer somewhere. That's what send_flush() is for. It forces the server to spit out whatever's buffered instead of waiting the full window.

Barge-in is the other half of this, and it has to happen in under 150 milliseconds or the whole thing feels broken. Here's the actual sequence:

Handle any tool call still running. If it's a read-only lookup, let it finish quietly in the background. If the caller just contradicted it, kill it immediately.

Get all this right and barge-in handling becomes invisible, which is the whole point. Get it wrong, and every interruption feels like talking over a bad phone line from 2003.

All of this, every millisecond of it, runs over a network connection. And that network layer quietly puts a ceiling on everything we just talked about, whether you've noticed it yet or not.

The Transport and Monitoring Layer Most Guides Skip

Nobody talks about this part, and that's a mistake, because it caps everything else you've built.

WebRTC over UDP beats HTTP or SSE over TCP for anything real-time. It's full-duplex, so audio flows both directions at once. Jitter stays lower. Barge-in works natively over a single multiplexed channel instead of you bolting it on. Opus handles packet loss gracefully rather than dropping chunks. And the connection setup occurs once per session via ICE, not on every request.

Telephony adds its own tax. G.711 and G.722 cap audio at 8kHz, and transcoding Opus down to that adds another 150 to 700 milliseconds. Worse, the compression artifacts can fool your VAD into thinking there's speech when there isn't, so PSTN deployments need their own custom thresholds, not whatever worked in your WebRTC tests.

One alignment detail that actually matters: process speech-to-speech audio in 80 millisecond chunks, matching four WebRTC packets, so it lines up with the codec's own frame boundaries. And watch your tunnels. A stray Gradio tunnel in your setup can quietly add 500 milliseconds you'll spend days trying to find elsewhere. Keep your SFU nodes sitting next to your GPU inference, not routed through some convenience layer three hops away.

None of this shows up in a demo. It shows up in production, at 2 am, when someone's asking why call quality dropped on Tuesday, and nobody thought to check the transport layer first.

Formula worth pinning to your dashboard:
Semantic accuracy = (correct intent extractions) / (total transcribed turns)
ROI = (cost of human agent time saved) / (total voice agent operating cost)

Track both per stage, not just at the end of the call. If you're weighing WebRTC against SIP for your setup, this is the layer that decision actually lives in, not a preference call.

With the full picture sitting on the table no-, latency, cost, control, compliance, noise handling, and transport- the choice stops being about which architecture feels newer.

So Which One Should You Actually Build?

Here's my honest take after all of this: the right choice was never about which architecture sounds more natural. It's about which one fits the job you're actually doing.

Operational fit beats naturalness every time it counts.

That said, I don't want to oversell the naturalness gap either. A well-tuned cascaded system, with good TTS prosody, tight streaming, and disciplined turn-taking, closes most of that gap anyway. The "robotic pipeline vs human-sounding S2S" framing is mostly outdated at this point.

So here's how I'd actually walk through the decision, in order:

Cascaded pipelines win outright when you need complex function calling, when cost discipline matters at scale, or when your industry demands deterministic, auditable safety. Speech-to-speech wins when the interaction is emotionally driven, when the environment is loud and unpredictable, or when the conversation needs true full-duplex back-and-forth, like two people talking over each other naturally.

And for a lot of real businesses, the honest answer is neither extreme. A hybrid setup, LLM handling dialog, swappable STT and TTS, backend logic kept in its own controlled layer, ends up being the pragmatic middle ground for structured processes that still want to sound human.

If you're comparing actual platforms against this framework, our voice agent platform comparison guide breaks down where each one leans. And this same logic- build for the job, not the demo- is exactly why the custom AI versus off-the-shelf question keeps coming up in every industry we work in, whether that's healthcare, insurance, ecommerce, retail, logistics, restaurants, lead qualification, or straight customer service.

None of these industries need the same architecture. That's the whole point.

Where This Leaves You

Most teams don't need to build this orchestration layer themselves. Honestly, most shouldn't try. What you do need is to understand it well enough to know exactly what you're buying, and, more importantly, where a vendor's promises will quietly fall apart once you're past call ten thousand.

That's the real value of walking through all of this. Not so you build a voice agent from scratch this weekend, but so nobody can hand you a pitch deck and call it production-ready without you asking the right question back.

If you're at that stage, deciding how to build an AI voice agent that actually holds up past the demo, that's exactly the conversation we have with teams every week. Happy to walk through your specific case and tell you honestly which architecture fits here.