Voice Agents That Do not Break at Enterprise Scale (Engineering Report 2026)

The Voice Agent, End-to-End

Whether the caller arrives over a phone line or a browser, the path looks the same once the call is on the wire. Audio enters the LiveKit SFU and a voice-agent worker is dispatched into the room. The worker handles the audio plane (VAD, STT, TTS, turn-taking, barge-in) and forwards each conversational turn to the Agent-Service Orchestration layer, which runs agentic orchestration — conversation state, tools, function calls — and makes its LLM calls through the LLM Router. The agent’s reply streams back through the SFU to the caller.

How We Benchmarked

All numbers in this post come from controlled, repeatable load tests against live SimplAI infrastructure — not vendor marketing material. We deliberately mixed test methodologies so each layer was measured both in isolation and as part of the end-to-end path.

Methodology in detail

SFU concurrency was driven with a synthetic 1:1 audio load generator ramping from 100 → 1,000 rooms in 5 rooms/sec increments, holding each step for 3–5 minutes. SIP inbound was driven with SIPp as a UAC, issuing standard INVITE → 200 OK → ACK → BYE flows at 5 calls/second. Call-setup latency and Voice Agent Response Time were captured from a single client that timestamps each protocol event (T0–T5) and detects the first audio byte via an RMS-energy threshold on the agent’s track. Agent-pipeline TTFT was measured by sending OpenAI-compatible streaming /chat/completions requests directly at each layer (LLM Router, agent-service end-to-end) at 5 → 1,000 concurrency. Barge-in was measured with a scripted client that injects a 2-second interrupt utterance 2.0 s into the agent’s welcome and times the silence-and-recovery sequence. STT and TTS layers were characterized using a combination of vendor-published benchmarks and field measurements from our deployment.

The Media Plane Holds: SFU at 1,000 Rooms

The first question for any voice platform is whether the media plane survives. We pushed the LiveKit SFU from 100 to 1,000 concurrent 1:1 audio rooms, with HPA configured at 80% CPU target and replicas bounded between 2 and 10.

SIP Inbound: The Worker Is the Limit

For our calling-agent customers, SIP capacity matters more than SFU capacity. We drove inbound calls at 5 calls/sec and varied voice-agent worker resources to find the binding constraint. Every failed call we observed traced back to the agent worker — never the SIP server, never the SFU. LiveKit’s load-based scheduling rejects new INVITEs when all workers are above its load > 0.7 threshold and the SIP server returns a protocol-correct 486 Busy Here.

The takeaway is simple: SIP capacity scales linearly with worker CPU. The SIP server, room admission, and SFU all stayed silent during stress.

The Single Biggest Latency Lever: Same-Region

VART — Voice Agent Response Time, from user end-of-speech to first byte of agent audio — is the metric users actually feel. We measured it in two topologies: voice-agent service and SFU in the same region, vs. across regions. All VART numbers in this post come from real-world scenarios we benchmark against — not curated demo runs. They are designed to mirror the conditions our enterprise customers actually run in, including realistic concurrency, prompt complexity, and network paths.

Several stages in the welcome and VART paths involve round-trips between the agent worker and the SFU. Each round-trip compounds inter-region RTT; co-locating the worker collapses the dispatch overhead from over a second to ~2 ms. Connection time remains ~3.0 s either way — that one is bounded by client-to-SFU transport, not by where we deploy the worker.

The < 2-second VART target, budgeted

We size each layer against a less than 2-second VART target, measured under real-world traffic — not demo conditions. Here’s how the budget breaks down across stages, anchored on the vendor benchmarks in Section 06 and the layer measurements in Section 07.

Picking the Right Speech Vendors

STT and TTS are the two layers we don’t build ourselves. We bias toward vendors that win on the combination of latency, accuracy, and conversational features (turn-taking, diarization), then we measure them in our own pipeline rather than trusting the marketing page.

STT — Deepgram Nova-3 / Flux

For speech-to-text, what matters for VART is finalize latency: the delay between the user finishing their sentence and the complete transcript reaching the LLM. Streaming partials don’t help if the finalize stalls. Deepgram Flux is purpose-built for voice agents, with native end-of-turn detection that makes turn-taking feel natural.

TTS — ElevenLabs Flash v2.5

For text-to-speech, the metric is FTTS — first-byte time from the LLM’s first token to the first audio sample. Equally important is voice naturalness (MOS). Flash v2.5 wins both axes for our use cases. The one caveat is region proximity: we route Mumbai traffic through the US endpoint today, which adds ~250 ms of network latency on top of the ~200 ms model-inference TTFB.

When ElevenLabs APAC GA lands, the Mumbai VART profile improves automatically. If the gap persists for any specific deployment, Deepgram Aura-2 is our pre-validated fallback for APAC traffic.

Inside the Hot Path: Agent Orchestration under Load

The most interesting bottlenecks aren’t where you’d expect. The SFU is fine. The LLM provider is fine. The compression happens inside the two services that sit between them. The Agent-Service runs the agentic orchestration: conversation state, tool calls, traces, the loop that turns a transcript into the next agent action. The LLM calls from the Agent-Service then flow through the LLM Router, which combines model routing with our Model Aggregator — a thin service that manages provider API keys and transforms requests into a uniform OpenAI-compatible format.

With LLM Router pods and Agent Orchestrator pods sized appropriately, we sustain 1,000 concurrent calls across 3,000 requests with zero failures at 40.7 calls/sec. The LLM provider itself contributes only a small fraction of that median; the bulk is orchestration overhead — routing, key management, request transformation, and the agentic loop.

The lesson we learned the hard way: pod sizing matters

The first time we ran the Model Aggregator portion of the LLM Router at 1,000 concurrency on its original sizing (1 core / 1.5 GB × 3 pods), it broke spectacularly: 33.75% failure rate, two pods restarting under resource exhaustion, a long tail well above 30 s. Doubling each pod to 2 cores / 3 GB resolved it cleanly. Same code, same traffic, completely different outcome.

Naturalness: Barge-In and Recovery

Latency makes voice agents fast. Barge-in is what makes them feel human. If a caller starts speaking while the agent is mid-sentence, a natural agent stops quickly, listens, and responds with a new answer. We measured this end-to-end across 20 sequential runs on our production stack.

The total spread across 20 runs was just 203 ms — p95 sat 31 ms above p50, p99 within 88 ms — pointing to a well-tuned VAD and TTS-cancellation pipeline with no significant tail. The barge-in pipeline includes a 300-ms silence-confirmation window that’s deliberately conservative; the underlying audio cut-off reaction is closer to 250–400 ms.

Recovery shares the same STT → LLM → TTS path as a normal turn, so its variance tracks the agent-pipeline characteristics in Section 07 — the cleaner we make the agent orchestration and LLM Router, the cleaner barge-in recovery feels.

Capacity Plan: 1,000 Concurrent Calls

Putting it all together, here’s what it takes to run 1,000 concurrent voice conversations on the SimplAI stack — sized from the actual measurements above, with realistic headroom.

Numbers above assume same-region deployment per our default topology. Voice-agent worker count dominates the cost line — and is the layer most amenable to next-quarter optimization (see Section 10).

What We Ship Next

Numbers like these are a checkpoint, not a finish line. The road map for the next two quarters keeps the same north stars — sub-second VART, human-feel barge-in, no tail failures at 1,000+ concurrent — and pushes them further.

The SimplAI Commitment

Voice is the highest-bar interface software has — there’s nowhere to hide a 4-second pause or a robotic interruption. The stack we’ve built holds 1,000 concurrent calls with zero packet loss, hits a 1.80-second VART end-to-end against real-world benchmarked scenarios (not demos), and lets the agent stop talking in 600 ms when a caller chimes in. Whether you’re deploying outbound calling agents at scale or embedding a conversational agent in your product, the same infrastructure carries the call.

We’re committed to keeping that bar moving. The benchmarks in this post are public so customers, partners, and our own teams can hold us to them. The next set of numbers ship with the next release.