Network Interruption @ LON
RESOLVED
2015-JAN-31 10:07UTC (RESOLVED)
Our upstream provider has narrowed down the root-cause of today’s incident. It seems that a line-card was defective in one of our datacenter’s core network routers. The nature of the defect was that the hardware component had not completely failed, in which case it would have been automatically bypassed by failover, but was operating in an invalid manner causing serious disruption of service to the entire DC network and making it very difficult to pinpoint. Our upstream provider has assured our network engineering team that they have taken corrective actions so this type of incident is avoided in the future and we will review their RFO report to try to ensure that an event of this type is never repeated again.
2015-JAN-31 08:44UTC (UPDATE)
Full network connectivity has now been restored at our London datacenter. Our network engineering team is closely monitoring the situation to make sure that network operation is nominal. We are expecting root-cause analysis from our upstream provider shortly.
2015-JAN-31 07:26UTC (OPEN)
Our London Data Center is currently experiencing site-wide network interruptions and connectivity issues. Our network engineering team is working with our upstream provider to get these issues resolved as soon as possible. All websites hosted at London are currently experiencing degraded performance.
