Date of Report: Friday, 30th of May 2014
Date/Time of Incident: Thursday, 22nd May 2014 9:34hrs
Date/Time of Restoration: Thursday, 22nd May 2014 10:04hrs
impact: Fibre users of international bandwidth and all ADSL/VDSL connections
Full Flavour buys fibre network capacity from FX Networks between our main network facility in Hamilton and our point-of-presence in Sky Tower where international bandwidth is handed over to Full Flavour from our upstream provider. We also have our Chorus copper traffic (ADSL/VDSL) transiting our POP at Sky Tower.
Full Flavour noticed a loss of network connectivity and immediately informed FX Networks.
Cause of Incident:
FX Networks Co-location at Citylink's Telco House Site
Citylink were completing maintenance on the power distribution cabinet to measure the load. The electrician clamped the meter to the power feed the Citylink racks, hosting FX Networks network equipment.
During this process the power feed from the power distribution cabinet to the rack, fell out. The electrician found the joint to be loose. He immediately reinstalled and tightened the power feed to the power distribution cabinet to restore service.
The UPS power feeds to the collation cabinet includes three phases, however the power rail in the cabinet was only connected to a single phase.
Telco House identified as a single point of failure, Telco House > Sky Tower link identified as a single point of failure.
Citylink have advised FX Networks that an additional PDL 56 series outlet is being provisioned to allow a dual power supply.
Full Flavour is investigating options with FX Networks to remove the single point of failure.