Realtime Speech-to-Speech Voice Agent

Decide whether S2S is leverage or opacity

Speech-to-speech agents send user audio into a realtime model and stream speech back without forcing every turn through a separate STT, LLM, and TTS chain. The value is not novelty. The value is that the model can preserve prosody, hesitation, interruption, and spoken context while reducing integration work.

Use it when conversation quality is the product

S2S is strongest for tutoring, coaching, companion, interview practice, live consultation, and high-touch support where users interrupt, hesitate, laugh, and change their mind. If the user experience depends on natural turn-taking more than perfect component control, a realtime model is worth testing early.

Avoid it when auditability dominates

Use a chained pipeline when you need an authoritative transcript, strict per-layer cost control, independent STT/TTS replacement, custom vocabulary handling, or regulated review. The tradeoff is simple: S2S can feel better in a demo, but failures are harder to attribute. Serious deployments need event logs, recordings, tool-call records, and eval calls.

Build the reference path

Start with one narrow use case: appointment changes, language practice, customer FAQ, lead qualification, or internal helpdesk triage. Define allowed actions, actions that require confirmation, and actions that are never allowed. Then wire WebRTC or SIP, create the realtime session, set the voice, configure tools, and log every session event.

Test interruption as a first-class feature

Do not test only polite three-turn conversations. Interrupt the agent during a long explanation. Change your mind after a tool call starts. Speak from a noisy room. Pause for two seconds, then continue. The practical promise of S2S is better turn-taking; if that does not survive messy calls, the architecture is not buying you enough.

Treat tool calls as state transitions

Realtime models can call tools, but voice mistakes make actions riskier. Reads can be fast. Writes should be confirmed. Cancelling audio is not the same as cancelling a database update, transfer, or booking. Track whether the agent is speaking, reasoning, waiting on a tool, or performing an irreversible action.

Keep a transcript even if the model does not need one

Native speech-to-speech may not need a transcript to respond, but your product team still does. Store an approximate transcript, session event log, tool arguments, and final user-visible action for every eval call. This gives support, safety, and engineering the same evidence when a user says the agent misunderstood them. If the transcript is generated by a secondary model, label it as a post-call artifact rather than the source of truth used by the realtime model.

Choose tools by product stage

OpenAI Realtime is the direct S2S route for testing native speech interaction, tool use, and WebRTC/SIP patterns. Retell AI and Vapi are better when phone numbers, analytics, simulations, and call operations matter immediately. LiveKit and Pipecat become more attractive when you need media ownership or a migration path back to a modular stack.

Price with real conversations

Retell currently lists AI voice agents at $0.07-$0.31 per minute. Other stacks combine platform fees, telephony, model usage, STT, and TTS. Price 100 representative conversations, including silence, retries, tool failures, hangups, and eval calls. A smooth demo call is not a cost model.

Keep an exit path

Even if S2S wins the first UX test, preserve transcripts, prompt versions, tool schemas, consent records, and eval audio. If cost, compliance, or provider lock-in becomes the problem, those artifacts let you move toward LiveKit or Pipecat with Deepgram, Cartesia, or ElevenLabs without redesigning the product behavior.

Separate prototype speed from production ownership

The fastest S2S prototype usually hides a lot of production work: user authentication, rate limiting, prompt versioning, content filters, call recording policy, human handoff, and incident review. Make those explicit in the roadmap. A good milestone is “50 successful supervised calls” before “self-serve beta,” and “one week of monitored production traffic” before routing important business calls through the agent.

Launch with eval calls

Before launch, write at least 30 test calls that cover normal tasks, long interruptions, accents, noise, user corrections, tool failures, forbidden requests, sensitive data, human handoff, and timeout. The embedded OpenAI Realtime tutorial is useful for the build path; your production gate is whether the agent behaves correctly under those scripted failures.

Do not let natural speech hide risk

The more natural the voice sounds, the more users may assume the agent understood, promised, or approved something. That matters in support, recruiting, finance, healthcare, and education. S2S can make the conversation smoother, but it does not remove the need for consent, privacy controls, escalation, and conservative action policy.

Pre-launch checklist

Before sending real users into the flow, confirm that every session has a timeout, a reconnect path, and a clear way to reach a human or text fallback. Run eval calls for interruptions, multilingual drift, tool hallucination, unsafe requests, private-data requests, and long silences. Review the first 100 calls manually enough to know which failure mode is actually common; teams often over-optimize model voice and under-invest in recovery prompts.

One final check: decide what happens when the agent sounds confident but the backend is uncertain. If inventory, calendar, account status, or policy data is missing, the agent should say that plainly and offer a follow-up path. A natural voice should never be used to smooth over missing state.

Ownership rule

Assign one owner for the workflow. Without an owner, generated assets accumulate, QA decisions drift, and no one knows which version is safe to reuse. The owner does not need to do every task, but they should maintain the checklist, approve final exports, and decide when a tool result is good enough or when the team should redo the work manually.