Date of Report: Friday, 30th of May 2014
Date/Time of Incident: Thursday, 29th May 2014 20:10hrs
Date/Time of Restoration: Friday, 29th May 2014 04:20hrs
impact: All Full Flavour services
Full Flavour operates it's core network centre in a data centre facility in Hamilton. We buy dark fibre services from Ultrafast Fibre Ltd that connect our core network to Ultrafast Fibre Ltd and our upstream network providers.
Cause of Incident:
UltraFast Fibre had scheduled fibre work to repair damaged fibre from 20:20hrs on Thursday, 29th May 2014 to 4:20hrs on Friday, 30th May 2014. Full Flavour and other service providers were not notified of this planned work.
Thursday, 29th May 2014
20:20hrs - Full Flavour monitoring system detects an alarm and commences investigation.
20:35hrs - Full Flavour advised by UltraFast Fibre that they have scheduled fibre work in Hamilton.
21:11hrs - Full Flavour receives confirmation that UltraFast Fibre is working on damaged fibre on the corner of Victoria street and Bryce street in Hamilton. The fibre repair has been scheduled as planned work. Completion is scheduled for 2:00hrs. Full Flavour have not received notification of this planned work.
21:13hrs - Full Flavour contacts customers to advise loss of service and restoration ETA
Friday, 30th May 2014
3:01hrs - Fibre repair still in progress.
4:20hrs - Fibre repaired. Full Flavour confirms service restoration.
Single point of failure exists between Full Flavour Hamilton facility and Ultrafast Fibre Hamilton facility. While alternative fibre is in place, real-time fail-over is not presently configured.
Full Flavour have raised the issue of not receiving a notification of this planned work with UltraFast Fibre management. Had a notification been received, Full Flavour would of scheduled a move to alternative fibre while the work was being completed.
UltraFast Fibre are to conduct an internal review regarding this matter.
Full Flavour investigating options for a real-time fail-over system onto the alternative fibre.