We run an AI receptionist on a real phone line. It greets the caller, says it is an AI and that the call is recorded, answers questions, quotes prices, books calls, takes leads, handles opt-outs and passes the call between seats with different voices. On November 20 and 21, 2026, I put four voice stacks behind that same line and called it 128 times to see which one I should trust with a caller.
The four were Phonic, xAI's grok-voice realtime, Google's Gemini Live (gemini-3.8-live) and Avartha, a cascade we route through our own gateway: separate speech-to-text, language model and text-to-speech. Same prompts, same tools, same scenarios. Only the stack changed.
Why task completion was not enough
Our earlier evals asked one question: did the agent do the thing? Book the slot, quote the right price, log the opt-out. That matters. But a receptionist that books every call and talks over every caller is not a receptionist I want on the line, and a completion score cannot see the talking over.
So half of each call's score was the task, and half was interaction: how fast the agent answered, whether it stopped when the caller cut in, whether it kept going through an "mm-hm", whether it waited out a pause in the middle of a sentence, and whether it left the caller sitting in silence. I wrote the rubric down and committed it before the first scored call. The pilot could fix measurement bugs. It could not move a weight.
How the calls were made
A caller bot dialed the line over Twilio. It played recorded caller lines, waited for 750 ms of agent silence before speaking again, barged in at fixed points, said "mm-hm" while the agent was talking, and paused for 1.2 seconds in the middle of a sentence.
All timing came from the dual-channel recording of the caller leg, one channel per side. Nothing a provider reports about itself was used, because each one reports differently and some report nothing. Tools ran in a sandbox where bookings and leads were mocked and logged, so the task checks read what the agent actually did. A Claude judge scored conversational quality from the transcript plus the timeline. Claude was not one of the stacks being tested.
Twelve scenarios, two calls each per stack, and four each for the interaction probes. 32 calls per stack.
What came out
| Stack | Overall [95% interval] | Interaction | Task | Median reply |
|---|---|---|---|---|
| Phonic | 0.88 [0.87, 0.89] | 0.78 | 0.98 | 920 ms |
| xAI grok-voice | 0.86 [0.85, 0.87] | 0.79 | 0.92 | 1,280 ms |
| Gemini Live | 0.76 [0.72, 0.79] | 0.69 | 0.82 | 1,780 ms |
| Avartha (ours) | 0.75 [0.70, 0.79] | 0.69 | 0.81 | 1,220 ms |
Phonic edged xAI by 0.02, and the interval on that gap runs from 0.003 to 0.041. It clears zero, barely. Gemini and Avartha could not be told apart.
The ranking is the least interesting part.
Each stack failed somewhere else
Gemini Live was the best interrupter and the worst listener. It went quiet 220 ms after the caller cut in, the fastest of the four, eight times out of eight. That same hair trigger fired on every backchannel: all seven times the caller said "mm-hm" mid-answer, Gemini stopped talking. It also had the slowest replies, 1,780 ms at the median, and 13 of its 32 calls had at least one stretch of 3 seconds or more of dead air while the caller waited.
Phonic was the fastest and the most eager. It answered in 920 ms at the median and completed both bookings in full, with a task score of 0.98. When the caller paused mid-sentence, though, Phonic started talking inside the pause on all four tries.
xAI was the best conversation partner and slow at handoffs. Its interaction score (0.79), turn-taking (0.81) and judged quality (0.70) were the best of the four. No backchannel tripped it, and it stopped on all eight interruptions. But each transfer between seats left 4.9 to 5.9 seconds of silence, on all four transfers measured.
Avartha, our own cascade, had the worst day. Twice its gateway could not reach the language model at call start and the call fell back to another provider. The rubric scores that as zero, and it should. It made no booking on either booking call, spoke into the pause four times out of four, and on the one transfer call it served it left 4.9 to 5.4 seconds of silence at each handoff. That one is on us.
The pause probe caught almost everyone. Three of the four stacks spoke into the caller's 1.2-second pause on at least three of four tries. Gemini waited it out three times in four.
What I take from it
The speed that makes an agent feel sharp is the same speed that makes it jump into a pause. The sensitivity that makes it yield cleanly to an interruption is the same sensitivity that makes it stop for an "mm-hm". These trade against each other inside each stack. Which way you lean depends on your callers. A line full of people reading out order numbers needs pause tolerance more than a fast reply.
None of this shows up if you only check whether the booking happened. Phonic's 0.98 task score is real, and so is its 4 for 4 on talking into a pause.
The limits
This is one company's line and prompts, on two days in November 2026, not a ranking of these vendors. The caller was synthetic, with one voice. 32 calls per stack is enough to separate most pairs here, not enough to promise how the order holds on a different script. The xAI calls mostly ran in one block on the second day instead of shuffled in with the others. The judge is one model family, and nothing measured how natural the voices sounded. Our crude check for "did the voice change at transfer" flagged Phonic as not switching voices; Phonic's own records showed it did, and the two voices were just close in pitch. A concurrent run of our own harness swapped providers on a few calls; those were set aside and re-run, never scored against the stack.
The full method, rubric, per-stack numbers and limits are in the white paper: Interaction, not just completion (PDF).
If you are picking a voice stack for a phone agent and want this run against your own line, get in touch.