Restarting a live call session without going deaf

In-place recovery is a state problem: a restarted session must come back hearing, remembering, and speaking, in that order, and each one can fail silently.

Where the field is

The field has stopped arguing about whether voice agents work in demos and started publishing about why they fail in production. The recurring lists are consistent: conversation state management breaks in multi-turn calls, telephony infrastructure fails at scale, and teams lack the observability to diagnose which pipeline stage (audio, STT, LLM, tool use, TTS) actually failed. One analysis puts it bluntly: most agents look perfect in demos and fail in production, and the demo-to-production gap kills a large share of deployments.

What's striking is what the published guidance covers for mid-call crashes: almost all of it is about failing OVER, not recovering IN PLACE. The standard prescription is a threshold (5 to 10 seconds of degraded service), then transfer to a human or a callback queue with the conversation context attached. Session persistence gets discussed as "recognize the caller when they dial back." Those are good patterns, but they concede the call. For an agent whose whole job is answering calls a business would otherwise miss, hanging up and hoping the caller returns is the failure, not the mitigation. Recovering the same conversation on the same live call gets very little coverage, and having now shipped it, we think we know why: the failure modes in that path are silent, model-specific, and only reproduce when something else has already gone wrong.

What we shipped against it

Our voice worker can restart a crashed realtime session in place: same room, same conversation stage, no re-greeting. The caller hears a beat of silence and a brief apology, then the conversation continues where it left off. Getting there surfaced two failure modes, both observed live on our demo line, both invisible in any demo, because the restart path only executes after something has already failed.

Failure one: the deaf restart. The restarted session must not process caller audio until the conversation history has been replayed into the fresh model stream (the model requires history before audio, an ordering constraint of the realtime API we run on). Our first implementation expressed that as "start the session with audio input disabled." Reasonable, and wrong: starting with audio disabled creates no input stream at all, so the later enable call is a no-op that raises "Cannot enable audio input when it's not set." The agent came back speaking normally and permanently deaf. Nothing errored at restart time. The fix is one move earlier in the lifecycle: start with the input stream attached (the default), then immediately mute it until history lands, then unmute. Same intent, different mechanism: you can silence an existing stream, you cannot conjure one that was never created.

Failure two: the silent restart. Right after restart, the agent should speak first ("apologize for the glitch, pick up where the conversation left off"). A one-shot generate call for that recovery line races the fresh model stream coming up, times out, and the agent never speaks again. The fix is equally unglamorous: the recovery reply retries with backoff (immediately, then 1s, 2s, 3s) until the stream is ready.

Both fixes are a handful of lines. Both took a live call to find. The honest limits: recovery still depends on a full history replay, the caller does hear the gap, and the ordering constraints are specific to the speech model we run, so this is tuned recovery, not a general algorithm. The transferable part is the checklist, not the code.