Backbone provider Level3 says an outage that knocked out VoIP service for much of the US Tuesday morning was the result of improperly configured equipment.

It seems the outage, which smashed call services offline for much of the country, was not the result of any fiber cuts or facility damage, but rather some classic bad switch settings.

As the network provider returned operations to normal, customers received a technical note from Level3, seen by The Register, describing the issue and its resolution:

Root Cause: Calls were not present in Level 3 voice switches, impacting voice services in multiple markets of the United States.

Fix Action: Configuration adjustments were implemented to allow calls to reach the switches and complete appropriately.

Reason for Outage (RFO) Summary: The Voice NOC investigated reports of voice service impact throughout multiple markets in the United States. The equipment vendor was engaged for assistance with investigations. No inbound or outbound calls were present in voice switches. The Voice NOC was able to implement configuration changes that were successful in allowing calls to reach the switches and process accurately. The Voice NOC will continue to fully evaluate the incident and appropriate actions will take place to ensure incidents of this nature do not recur in the future. Services are currently restored and stable.

Corrective Actions: We know how important these services are to our customers. As an organization, we are putting processes in place to prevent issues like this from recurring in the future.

To us, this sounds like Level3 – or a partner – misconfigured its network equipment to drop voice traffic. We asked the carrier if that was the case, and we were told the following:

On October 4, our voice network experienced a service disruption affecting some of our customers in North America due to a configuration error. We know how important these services are to our customers. As an organization, we’re putting processes in place to prevent issues like this from recurring in the future. We were able to restore all services by 9:31am Mountain time.

Make of that what you will.

Updated to add

Here's some more detail on the cock-up from Level 3 – the backbone biz has forwarded to us this advisory it sent out to customers:

Repair Area: Human Error Occurrence

Repair Action: Human Error

Repair Summary:

Reason for Outage (RFO) Summary: On October 4, 2016 at 14:06 GMT, calls were not completing throughout multiple markets in the United States. Level 3 Communications¿ call center phone number, 1-877-4LEVEL3, was also impacted during this timeframe, preventing customers from contacting the Technical Service Center via that phone number. The issue was reported to the Voice Network Operations Center (NOC) for investigation. Tier III Support was engaged for assistance isolating the root cause. It was determined that calls were not completing due to a configuration limiting call flows across multiple Level 3 voice switches. At 15:31 GMT, a configuration adjustment was made to correct the issue, and Inbound and outbound call flows immediately restored for all customers. Investigations revealed that an improper entry was made to a call routing table during provisioning work being performed on the Level 3 network. This was the configuration change that led to the outage. The entry did not specify a telephone number to limit the configuration change to, resulting in non-subscriber country code +1 calls to be released while the entry remained present. The configuration adjustments deleted this entry to resolve the outage.

Corrective Actions:

Level 3 Communications knows how important these services are to customers. As an organization, this incident is being evaluated at the highest levels to prevent reoccurrence. Process has been put in place to alert this specific Provisioning team of how this incident could have been avoided. Access restrictions have been made to mitigate the possibility of large-scale configuration changes, and a future process for these types of provisioning activities will be evaluated to involve additional technical support. System tools are being investigated to place additional guardrails against this type of trouble.

(via Register)

2016.10.31  - CenturyLink said Monday that it will pay about $24 billion to buy telecommunications company Level 3, in an effort to expand its services for businesses.

CenturyLink provides internet, TV and phone services to consumers, as well as data and IT services for businesses. Level 3, based in Broomfield, Colorado, provides data, video and other communication services to businesses and government agencies.

Level 3 shareholders will receive about $66.50 in cash and a portion of CenturyLink stock for each share of Level 3 they own. The companies valued the deal at $34 billion, when debt is included.

After the deal closes, which is expected to happen in the third quarter of 2017, CenturyLink shareholders will own about 51 per cent of the combined company and Level 3 shareholders will own about 49 per cent.

The combined company will be based at CenturyLink's headquarters in Monroe, Louisiana, and will keep a "significant presence" in the Denver area where Level 3 is currently based.

Shares of CenturyLink Inc. fell 9.8 per cent to $27.46 before the stock market opened Monday, while Level 3 Communications Inc. shares rose 5.5 per cent to $57.04.