Realtime STT-to-LLM-to-TTS Pipeline

Decide whether modularity is worth the weight

A realtime STT-to-LLM-to-TTS pipeline is valuable because every layer can be measured, replaced, priced, and replayed independently. It is the right architecture for phone agents, browser voice assistants, call analysis, AI tutors, and products that need provider control. It is not automatically better than speech-to-speech. It is better when you can use the control.

Use it when each layer has a reason to exist

Choose the modular path when you need domain vocabulary in STT, a specific LLM, a particular TTS voice, regional deployment, private media handling, provider fallback, or detailed call replay. If the goal is to validate whether users like talking to the product at all, start with a managed platform or a native realtime model first.

Avoid it without per-stage observability

The main failure mode is not writing too much code. It is being unable to explain the silence. A two-second pause might be WebRTC jitter, VAD waiting for final silence, STT finalization, LLM first-token delay, a slow tool call, TTS first-audio latency, or your playback buffer. Without traces, modularity makes debugging slower.

Design the reference path

Start with one boring chain: low-latency transport, VAD, streaming STT, LLM, streaming TTS, playback control, and one event timeline. Do not add three model routes on day one. Run 100 realistic calls through the first path, then optimize the failing layer.

Budget latency by stage

LiveKit’s current architecture guide breaks perceived latency into transport, STT first partial, LLM time-to-first-token, and TTS time-to-first-audio. A useful starting budget is under 50ms for WebRTC transport, 100-200ms for STT partials, 200-400ms for LLM first token, and 100-300ms for TTS first audio. Treat those as debugging targets, not guarantees.

Optimize geography before model choice

The non-obvious win is often placement. If media transport, STT, LLM, TTS, and your business API sit in different regions, round trips will erase the gains from a faster model. Co-locate the worker, inference endpoints, and tool backend where possible before paying for premium speech quality.

Make tool calls part of the pipeline

Voice agents do not only chat. They book, search, update records, and transfer calls. Log tool-call start, finish, failure, timeout, and cancellation policy separately. Audio playback can usually be cancelled on barge-in; a database write or payment action may need to finish silently or ask for confirmation.

Add fallback at each boundary

Every streaming boundary needs a failure behavior. If STT confidence is low, ask a short clarification rather than sending garbage to the LLM. If the LLM stalls, play a brief holding phrase or transfer. If TTS fails, fall back to a simpler voice or text channel. If transport reconnects, resume with a summary instead of pretending nothing happened. These fallbacks are unglamorous, but they are what make a modular chain survive real users.

Choose the stack by ownership

LiveKit is the strongest default when you want media transport, agent sessions, plugins, and observability in one stack. Deepgram is a practical STT layer when partial transcripts, endpointing, and noisy input matter. Cartesia or ElevenLabs are strong TTS choices depending on whether low-latency voice-agent work or polished brand voice is the priority.

Compare managed and self-hosted honestly

Deepgram lists Nova and Flux STT from $0.0077 per minute. Cartesia’s current pricing includes a free plan, Pro at $4/month, Startup at $39/month, Sonic TTS credits, and Line voice-agent rates. Those numbers only cover pieces of the system. Add LLM tokens, telephony, retries, storage, monitoring, and failed calls before calling a stack cheap.

Use local builds to learn the boundaries

The embedded LiveKit local-agent tutorial is useful because it makes the component boundaries visible: LiveKit for realtime media, Whisper for STT, a local LLM, and a local TTS service. Do not copy it directly into production unless the latency and hardware fit your use case; use it to understand what each layer owns.

Document the contract between layers

Write a small contract for each interface: audio frame format, transcript event shape, partial versus final behavior, tool-call schema, TTS chunk format, cancellation event, and error payload. The contract prevents provider swaps from becoming a rewrite. It also gives QA something concrete to test when a new model version changes punctuation, endpointing, or streaming cadence.

Launch with replay, not hope

Before launch, keep a replay set of successful calls and failures. You should be able to run the same audio through a new STT model, replay the same transcript through a new LLM prompt, and compare TTS first-audio timing across voices. This is the practical advantage of modular architecture.

Promote failures into tests

Track P95 time-to-agent-speech, STT empty or low-confidence segments, LLM first-token delay, TTS first-audio delay, tool-call timeout rate, missed barge-in, and user repeat requests. Every model change should run against the same sample set. The stack is production-ready only when you can change one layer without losing the ability to explain the result.

Security and retention notes

Decide early whether raw audio, transcripts, tool results, and derived summaries are retained, redacted, or deleted. Phone and support use cases often contain names, addresses, account details, or health and payment hints even when the product is not formally in a regulated category. Keep a short retention policy in the repo, and make sure debugging convenience does not quietly turn into permanent storage of sensitive calls.

For cross-border teams, include region in the trace metadata. A failure that appears to be model latency may be a routing problem between the caller, media worker, STT region, LLM region, and business API. Region-aware traces make scaling decisions less superstitious.