Skip to main content

RFO - 20261111 | Voice JFK

Incident Window

Start: 2026-11-11 16:21 UTC

End: 2026-11-11 20:15 UTC

Impacted Services

A portion of inbound and outbound voice calls routed through the JFK data center did not complete during the incident window.

Summary

Beginning at 16:21 UTC on November 11, 2026, a portion of Bandwidth customers' inbound and outbound voice calls in the JFK data center did not complete. Bandwidth teams were engaged and began investigating.

Investigation determined that a software update to the call routing service in the JFK data center increased the number of database lookups made for each call. When responses from the supporting database slowed, the service could not keep up with the volume of waiting requests, exhausted its available resources and became unavailable. Calls that depended on routing attributes from this service could not retrieve them, and a portion of inbound and outbound calls was not completed.

To resolve the incident, Bandwidth removed the affected software version from the JFK data center and restored the previous stable version. Following the restoration, call processing returned to normal. As a precaution, Bandwidth also restored the previous version in the other data centers. Bandwidth confirmed full service restoration on November 11th at 20:15 UTC.

Root Cause

The incident was caused by a software update to the call routing service in the JFK data center that increased the number of database lookups made for each call. When responses from the supporting database slowed, requests accumulated faster than the service could process them, which exhausted its available resources and made the service unavailable. Without the service, voice calls could not retrieve the attributes required to complete, which affected a portion of inbound and outbound calls throughout the incident window.

Preventive Actions

Bandwidth is undertaking the following actions to improve system resilience and reduce the recovery time should this type of incident recur:

  • Bandwidth is strengthening how the call routing service handles slow responses from its supporting databases, to keep the service available during database slowdowns.

  • Bandwidth is expanding health monitoring for the call routing service.

  • Bandwidth is adding pre-release load testing that simulates database slowdowns as a required step before release.

  • Bandwidth is improving its deployment tooling to make deployment troubleshooting easier.

Did this answer your question?