Voice infrastructure for AI: The complete guide for developers and enterprises

Key takeaways:

• Are you accounting for network round-trip time in your voice agent’s latency budget?
• “We own our network” is the most overused claim in this space. An IP backbone and a PSTN network are not the same thing. And this distinction matters more for Voice AI quality and performance. 
• The best-performing AI deployments keep the network and the AI vendor decoupled for flexibility and control.

---

Industry median response latency across production voice agents is 1.4–1.7 seconds according to Hamming AI’s analysis of 4M+ production calls.^[1] That’s five times slower than the 200-400ms window human conversation runs on. Each 100ms above 800ms reduces completion by 4–6% and 73% of abandonments follow latency spikes above 1,200ms.^[2]

In short, you could have the perfect model behind your voice agent but your infrastructure could fail you.  Everyone’s focused on prompts, fine-tuning, and evaluation benchmarks. But the network layer doesn’t make it onto the sprint board until it’s the reason your project launch KPIs go wrong.

The network: what you’re actually building your voice AI stack on

Most developers think about voice AI in three layers: speech-to-text (STT), the LLM, and text-to-speech (TTS). That’s clean, logical, manageable. But there’s a fourth layer sitting underneath all of it that rarely makes it onto the architecture diagram.

Think of it as the Dial Tone layer: the telephony and the network. It’s the path a call takes from your user’s phone through the PSTN (public switched telephone network), into your application, and back again. It governs latency, reliability, global reach, and compliance before your STT engine processes a single syllable.

Here’s how the full stack actually maps:

Layer	Component	What it does
Dial tone	Voice infrastructure	PSTN connectivity, call routing, SIP signaling, codec negotiation, number management
Speech	STT	Converts audio to text for the LLM to process
Intelligence	LLM	Processes input and generates a response
Voice	TTS	Converts the LLM’s text output back to audio
Orchestration	Control layer	Coordinates the entire loop — turn-taking, interruption handling, context management

What works: Using separate STT and TTS APIs with your own orchestration layer. Most teams start here, and it’s a reasonable starting point. You get flexibility, you can swap providers, and you maintain control of the AI logic.

What elevates: Integrated voice infrastructure that handles the entire telephony layer: number provisioning, PSTN connectivity, call routing, and codec optimization. So your orchestration layer only has to think about AI logic. The network becomes invisible, and the AI becomes the experience.

Production requirements: What does a  “good” carrier layer for AI look like?

How much network latency can a voice AI agent tolerate?

Conversational AI lives and dies by latency. Research consistently puts the threshold for natural-feeling conversation at around 200ms total round-trip time. Beyond that, users start to notice. Beyond 500ms, they lose confidence in the system. Beyond 1.2 seconds, they start asking if anyone is there.

At worst, the 500ms budget has to cover dial tone through orchestration (See table above). By the time your STT, LLM, and TTS each take their share, the network layer needs to contribute as little friction as possible.

That’s a real infrastructure problem that a latency benchmark number won’t solve.

A latency number on a spec sheet tells you what a network did in a lab. It doesn’t tell you what happens when your call volume triples during a product launch, or when you need someone to explain a latency spike at 2 AM instead of waiting on a support ticket routed through a network aggregator.

Not all network ownership is the same

“We own our network” has become the default claim in this space. Take it with a grain of salt. 

An IP overlay network and a PSTN network are not the same thing, and for voice AI, the distinction matters more than most providers will tell you.

In the US, Bandwidth is a CLEC (a Competitive Local Exchange Carrier) licensed under the US Telecommunications Act of 1996. That designation isn’t marketing language. It’s a regulatory status that requires Bandwidth to interconnect directly with the PSTN at the local exchange level, subject to FCC oversight, state-level certification in every market we operate, and direct interconnection agreements with ILECs like AT&T. The same legal framework that governs how calls move across America governs how Bandwidth moves them.

The practical difference is: when your AI voice agent places or receives a call, Bandwidth originates and terminates that call on PSTN infrastructure we own and are legally accountable for. For calls on the Bandwidth network, there’s no upstream carrier waiting on someone else to fix it. You’re talking directly to the engineers who own the routing.

An IP backbone gets your data between data centers efficiently. CLEC status means you own the last mile into the telephone network itself, which is exactly where voice AI calls live and die.

That’s the real test of the network layers under your Voice AI: not the latency it shows on a good day, but what it shows at scale and who you can call when it doesn’t meet your requirements.

Why do voice AI deployments fail in production even when the network tested fine?

Metrigy’s AI Technology Foundation & Strategy 2026-27 study of 759 IT and CX leaders found that even among those who know their own infrastructure well, roughly a third still flag network congestion and latency as likely blockers for AI deployments, despite most organizations reporting general confidence in their network’s AI readiness.

So the gap isn’t rare or theoretical. When your infrastructure is resold or aggregated, closing that gap means filing a ticket with a company that’s also waiting on someone upstream. When you’re working directly with the network owner, you’re talking to the engineers who control routing, codec negotiation, and interconnection, not a support layer reading from the same dashboard you already have access to.

What’s the risk of bundling voice AI and telephony with one vendor?

Enterprises initially chased all-in-one convenience by wrapping UCaaS and CCaaS with bundled calling plans, then spent years decoupling the parts that didn’t fit. AI voice deployments are trending the same way, only faster. Chasing the conversational AI latency needs, teams are moving toward modularized call flows: network, AI, and orchestration controlled and swapped independently rather than bundled. Metrigy’s research backs this up: 30.6% of organizations utilize custom middleware or gateways to process and convert media streams before they ever reach the AI. Successful companies rely heavily on session border controllers to replicate and transcode these streams directly to their AI services. They’re setting up to work directly with whichever AI service they choose, keeping that layer deliberately separate.^[3]

Some CPaaS providers are betting on the opposite: network and voice AI sold as one native stack. It reads well on a latency slide, since one vendor controls both ends of the call. It’s also the same bet enterprises already lost with early UCaaS/CCaaS bundles. When something breaks, whether it’s a codec issue or a model hallucination, both land in the same queue, worked by the same vendor whose stack caused the problem. There’s no one else to call. Vertical integration gets you an impressive demo, not a way out when the bundle itself fails.

Voice AI platforms like Bookline have already recognized this problem:

What should you look for in a voice AI provider’s global network coverage?

A voice AI agent that works in Chicago but degrades in São Paulo is still a broken product. As AI-powered communications scale globally, infrastructure that relies heavily on patchwork connectivity through resellers introduces what we call yo-yo coverage: performance that expands and contracts unpredictably across geographies.

What to look for in a global provider:

• Direct PSTN connectivity (not resold minutes) in your target markets
• Local number inventory in countries where you operate
• In-country regulatory support, not just coverage
• SLA-backed uptime commitments that apply globally 

Compliance requirements you can’t ignore

Voice AI in healthcare, finance, and regulated industries is a technical and legal challenge. The infrastructure layer carries compliance obligations that can’t be delegated to your AI vendor.

HIPAA: Any voice infrastructure handling protected health information (PHI) requires a Business Associate Agreement (BAA).* That agreement needs to cover the telephony layer, not just the LLM. Many developers assume their AI provider handles this. It frequently doesn’t.
*Consult your legal counsel to confirm your specific BAA requirements and coverage across your stack

SOC 2 Type II: The minimum credibility bar for enterprise deployments. Look for annual third-party audits, not self-attestations.

GDPR and data residency: Verify whether your infrastructure provider supports regional data routing for EU traffic.

Total cost of ownership: what the per-minute rate doesn’t tell you

Per-minute pricing is the most visible number and usually the least important one. When evaluating total cost of ownership, account for:

• Developer time: How long does it take to provision numbers, configure SIP, and get a test call working? Measured in days, not hours, for some providers.
• Incident cost: When something breaks at 2 AM during a sales peak, what does downtime actually cost? What’s the SLA response time?
• Compliance overhead: If your provider doesn’t offer a BAA, you’re building that compliance layer yourself. That’s an engineering sprint you didn’t budget for.
• Scale tax: Some providers price predictably at low volume and surprise you at enterprise scale. Model your pricing at 10x your current volume before signing anything.

How do you connect voice infrastructure to an AI model API?

Here’s how it works with the Bandwidth Voice API:

1. If you’re a Voice AI platform provider

A consumer calls your enterprise customer’s contact center which forwards the call to a Bandwidth number you own. This routes to your AI agent. If the AI can’t resolve the query, it sends a blind REFER back to Bandwidth, which then initiates a new call to the enterprise’s contact center platform. The AI agent is removed from the media path entirely once escalation happens, so the contact center handles queuing and agent routing on a clean call, not a three-way relay.

The key advantage of going direct to our network in this call flow is that your Voice AI platform gets a consistent and repeatable front-door deployment, irrespective of the type of contact center platform sitting behind it. You can build once and deploy across cloud, on-prem, or hybrid contact center environments without re-architecting for each use case. 

2. If you’re connecting any conversational AI platform to your contact center platform

Main article: How to add conversational AI to your contact center

A. Via SIP Integration

The architecture for a production voice AI deployment looks roughly like this:

Inbound call → PSTN → Bandwidth SIP trunk → AI Orchestration Platform
→ STT (real-time streaming) → LLM API → TTS → Audio back to caller

If you’re already working with a conversational AI provider, SIP is the most direct way to connect it to Bandwidth. Advanced Call Routing (ACR) sends inbound calls straight to your AI platform’s SIP endpoints, which you get from your AI provider. 

From there:

• Your AI platform handles the SIP INVITE, establishes media, and runs the conversation. 
• As an optional component, the Conversational AI platform can specify User‑to‑User (UUI) data, which will be passed to the CCaaS platform in a User‑to‑User header within the SIP INVITE.

This standards-based setup is straightforward if your AI provider supports SIP. It allows you to connect any conversational AI platform to your existing contact center stack. 

B. Via Voice API with Media Streaming

Bandwidth’s Voice API supports real-time audio streaming via WebSocket, which lets your application receive audio as it arrives rather than waiting for the full call to complete. Bandwidth sends a webhook notification to the application for every inbound call. The application can then answer the call and establish bidirectional media streaming to receive the call’s audio. From there, the application uses the Conversational AI platform’s capabilities to deliver the customer experience the enterprise has defined for that phone number.

• In most cases, the call can be contained in this platform, and can end as soon as the customer’s request has been satisfied.  
• If the call needs to be routed to a human agent, the application sends an API request to Bandwidth to initiate a transfer to the selected CCaaS platform, typically using one of Maestro^TM’s CCaaS voice integrations. Context information for the interaction can be incorporated into the call control signaling using SIP User-to-User Information. 

The Programmable Voice APIs give the software application access to a variety of network services that can be optionally layered into the call, including User-to-User Information, network-based transcriptions, and call recording.

Getting started with Bandwidth for voice AI

Everything in this guide points to the same conclusion: the network and the AI should stay two separate decisions. That’s true whether you want the flexibility to swap Voice AI platforms in your contact center or build your own voice AI platform on top of the Bandwidth Voice API.

We’ve modeled our Voice API  to give you direct and immediate control over your network layer and the flexibility to custom-build your (or around your enterprise customer’s) call flows. Your AI, whether it’s a vendor you chose or one you built, stays a decision you keep making on your own terms.

1. Hamming AI, Voice Agent Evaluation Metrics: Definitions, Formulas & Benchmarks
2. Hamming AI, Voice Agent Drop-Off Analysis: How to Measure and Reduce Call Abandonment (2026)
3. NoJitter, To maximize AI performance, maximize network operations