Incident Report
Planned emergency maintenance was carried out last night. This was to remedy issues between our border routers and the core network. These issues were caused by a bug that originated back in August 2019. This was a recommended fix outlined by Juniper support after they identified a bug that can present itself under very specific circumstances.
The work was undertaken in two parts, firstly on the backup router, then proceed to update the primary router after checking backup change was successful. All work and outages occurred within the advertised maintenance window from 10pm to 2am.
Unfortunately the change to the link up to the backup router caused a downstream issue that affected all devices connected to the backbone network and the primary router. This initial outage occurred at 10:10pm and was resolved once connectivity was restored at 10:45pm.
At this time we were halfway through the required changes and Network team made the decision to complete the work so that we did not leave the system in an unstable condition, as noted by Juniper support. We completed the change on the primary router uplink at 11:45pm and this also caused some instability in the backbone network with the dynamic routing. This cleared at midnight.
There are still some outstanding instabilities, specifically around the Azure Stack environment and these are being raised with Juniper support this morning. The Network team are working through the individual issues to resolve them as soon as possible.
Network Team