Resolved
Resolved
After 8 hours and 8 minutes

Hello Friends,

This issue has been resolved, and we are seeing traffic flow normally.

From approximately 7:45am to 8:45am PST, we were impacted by a routing issue with one of our carrier partners. This caused inbound phone calls to fail before hitting the Skyetel network, and resulted in callers greeted with error tones.

One of the pillars of the Skyetel architecture is our redundant routing, and many of our customer's have asked why it failed today. We do our best to be transparent, and so here’s what happened.

As part of our interconnection agreements with our carrier partners, we require multiple redundant connections from physically separate networks that automatically redirect call flows from an unhealthy region to a healthy one on a call-by-call basis. Ordinarily, this would mean that if a carrier partner was having an issue in one of their data centers, Skyetel's traffic would be automatically shifted into a different data center instantly.

This morning, the routing table with our carrier partner which dictates how calls should be routed was mistakenly deleted by an authorized member of our carrier partner's team. This prevented our routing tables from being followed and resulted in the call failures observed this morning. As soon as this routing table was reestablished, we were able to re-enable our call routing and service was immediately restored.

We have been, and will continue to be, in talks at the highest level with our carrier partner about how this happened, and what safeguards are available to prevent this from happening again. Skyetel’s President is addressing this personally, and all options are on the table. He’s, uh, pretty upset.

It is important to note that we do not use this carrier partner for a lot of Skyetel's transit. As such, only 7-8% of our DIDs were impacted. Furthermore, no outbound calling or other features were impacted.

We know that issues like this are frustrating, and we sincerely apologize for any inconvenience this may have caused.

Avatar for
Recovering
After 1 hour and 14 minutes

Hooray! Looks like traffic is coming back. We will provide a more detailed report of what happened shortly.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 1 hour and 7 minutes

Hi All

A re-route has been deployed and we are testing to verify this re-route worked.

Avatar for
Identified
After 12 minutes

We have identified one of our upstream providers is currently experiencing an outage. This outage is impacting multiple carriers - not just Skyetel. We are working with the offending party to recover as fast as possible.

Avatar for
Investigating

We are receiving reports of call failures before they reach the Skyetel network. We are investigating and will provide an update to this notice as soon as possible.

Avatar for
Began at:

Affected components
  • PSTN Network