Emergency Maintenance by ACS’ Service Provider Results in Outage

Event Occurrence: April 8, 2021, 09:00am AKDT

Background

The University of Alaska receives network services from ACS.  The ISP provides the long haul WAN circuits that comprise the University of Alaska’s core network from/to Fairbanks, Anchorage, Juneau, Seattle, and Portland.  The circuits from Fairbanks <-> Seattle and Anchorage <-> Portland comprise the University’s connection to Internet2, AWS, and some commodity internet sites.

Break Down of the Problem

On April 8th at 0900, ACS’ provider Wave Networks commenced repair work on many fiber-optic circuits in the Pacific Northwest.  The work was performed to replace failing equipment and in order to prevent an extended outage.  University traffic to/from the internet was impacted as a result.  All of ACS suffered from the same outage as we did.

Target State / Goal 

Restore services as soon as possible.

Root Cause Analysis 

Wave was performing further work in support of repair and restoration in the Portland area due to the severe inclement weather this past year.  Workers accidentally impacted more fiber circuits than intended resulting in this widespread issue.

Develop Countermeasures 

This issue was caused by ACS’ upstream provider, there are not many countermeasures that we can take to prevent this issue from happening again.  Short of provider diversification, which has historically been cost-prohibitive.