Wiring up a phone agent looks like a weekend project. Twilio Media Streams gives you a WebSocket with raw audio, you push it into a streaming STT, you feed the transcript to an LLM, you stream the reply into a TTS and send the bytes back. A few hundred lines. It works on the first call.
Then you listen to a recording and the agent is talking to itself.
Agent: "Hello, how can I help you?"
STT: "hello how can i help you" ← its own voice
LLM: "Sure! What can I help you with?"
STT: "sure what can i help you with" ← and again
Nobody said a word. The call is in a loop.
This post is about the part that took the real time — not the signal path, but the state machine sitting on top of it. I run this in production on a German phone line, and every rule below exists because something broke on a real call.
The single-channel problem
A phone line is not a mixing desk. There is one channel, and your own output comes back into it: through the caller’s speaker, through network echo, through the conference bridge on the other end. Your STT does not know which words came from a human and which are your own TTS coming home.
So you need a gate. While the agent speaks, the microphone is closed and incoming transcripts are discarded. When the agent finishes, it reopens.
The whole difficulty is in the word finishes.
The obvious fix, and why it doesn’t hold
The first instinct is to close the microphone when TTS starts and reopen it when the TTS stream ends.
This is wrong, and it’s wrong in a way that hides from you.
The end of your TTS stream is not the moment the caller hears the sentence. Between the last audio chunk you send and playback at the caller’s ear sit the telephony platform’s buffers and the network: anywhere from a couple of hundred milliseconds to well over a second, depending on the connection.
Release on stream end and the microphone opens while the caller is still hearing your voice. That’s the feedback loop, right there.
And here’s the part that costs you a day: it never reproduces locally. On your machine the latency is a few milliseconds, so the window never opens wide enough to matter. It only shows up on a real call, over a real mobile network, ideally on the worst connection your caller has.
Playback receipts
Twilio supports a mark frame. You place one behind a block of audio, and Twilio sends it back to you when playback reaches that point.
That’s the only honest signal you have. Not “I finished sending” — “they finished hearing.”
So the rule becomes: the microphone opens when no unacknowledged mark is outstanding.
That is necessary but not sufficient. Four more conditions turned out to be load-bearing, each after a specific failure:
| Condition | What happens if you ignore it |
|---|---|
| All marks acknowledged | Feedback — the caller is still hearing the agent |
| Speech queue empty | Opens in the gap between two sentences |
| No TTS stream active | Race condition when more audio is pushed |
| LLM not generating | Opens while the next sentence is still forming |
| Not tearing down the call | The closing sentence gets cut off |
One design note that paid for itself many times over: the release check returns the reason alongside the decision, not a bare boolean. When something goes wrong on a live call, you get false: marks or false: queue in the log instead of a silent False, and you know immediately which of the five it was. Field logs are the only debugger you have on a phone call.
Sentence-by-sentence output
If you wait for the complete LLM response before speaking, every reply opens with a pause as long as the entire generation. On a phone call that’s unbearable — a second of dead air feels like the line dropped.
So the token stream gets cut at sentence boundaries, and each finished sentence goes straight into the speech queue. Output starts as soon as the first sentence is ready.
This is a clear win, and it’s also why the mark logic has to handle sets rather than a single value: one response produces several marks, and you need all of them acknowledged, not just the last one to arrive.
Barge-in: the same logic, inverted
An assistant you cannot interrupt is unusable. Humans interrupt each other constantly, and a caller who has to wait through a wrong answer will hang up.
But if you treat every incoming transcript as an interruption, you get an agent that never finishes a sentence — because some of those transcripts are its own voice.
Three hurdles, in order:
1. The agent must be speaking. Nothing to interrupt otherwise.
2. Minimum length. “yes”, “mhm”, “right” are backchannel signals. Humans emit them constantly while listening; they mean I’m still here, not stop talking. Three words turned out to be a reasonable floor.
3. It must not be echo. This is the hard one, because STT practically never returns your own output word-for-word. It arrives with words dropped, merged, or lightly mangled. So two criteria run in parallel:
- a contiguous subsequence of the agent’s own recent output — catches clean feedback
- word overlap above a threshold (0.75 by default) — catches noisy echo and partial transcripts
The comparison base is a rolling window over roughly the last 120 words the agent spoke. The window must be bounded. Unbounded, it grows across the call until eventually every caller utterance overlaps something the agent said twenty turns ago, and the agent goes deaf. That one is a slow, quiet failure — it doesn’t crash, it just stops listening halfway through a long call.
The expensive bug
Here’s the one I’d have paid money to know in advance.
On barge-in you send Twilio a clear frame, which discards the buffered audio. Reasonable — the caller is talking, you don’t want your queued sentences playing over them.
But playback now never reaches the marks you placed in that discarded audio. The receipts never arrive.
If you don’t flush pending marks on barge-in, the release check waits for the rest of the call on confirmations that do not exist. The microphone stays shut. The agent never hears the caller again. The connection is up, the line is quiet, and the conversation is dead.
What makes it nasty is the distance between cause and symptom. The barge-in itself works perfectly — the agent stops mid-sentence, exactly as designed. The failure surfaces seconds later and looks exactly like an STT outage. I spent an embarrassing amount of time reading Deepgram logs.
Related, in the same family: the cancel flag has to be set before the queue is drained. The other way round and the still-running LLM stream drops new sentences into the queue you just emptied, and the agent wakes up again after the interruption — while the caller is mid-sentence.
Make the decision logic pure
The single most useful structural choice: all of this lives in one module with no network I/O and no external dependencies. It takes state in, returns a decision and a reason.
Which means the entire state machine is testable without a phone line, without API keys, and without a network connection. There are 24 tests, and each one documents a failure that actually happened on a call — the test names describe the symptom, not the method.
That matters more here than in most projects. A state bug reproducible only on a live call costs several minutes and a phone connection per iteration, and you can only test it as fast as you can talk. Moving the logic out of the I/O layer turned a two-minute feedback loop into a two-second one.
What it sounds like
The repository has an unedited recording of a live call: the caller interrupts mid-sentence, the agent stops, and further down the call the caller goes quiet long enough that the silence watchdog fires and the agent asks whether they’re still there — then correctly discards the echo of its own question.
Code, sequence diagrams for all three state flows, and the recording:
https://github.com/bokatechsystems/realtime-voice-agent
MIT licensed. The gate logic is provider-agnostic — it assumes only that your telephony platform emits some form of playback receipt.
If you’ve built something similar and solved the timing differently, I’d genuinely like to hear about it. This is one of those problems where every implementation seems to arrive at its own set of five conditions.
