Resolved - Summary
A combination of the routing daemon process crashing on the border router and high CPU utilization as the router rebuilt the global routing table from our multiple upstream providers.

End User Impact
Temporary degradation of external access to customers internet facing web services.

Incident Root Cause
Software & hardware issue with internet border routers.

Incident Resolution
Software automatically restarted and rebuilt routing tables, although this a time-consuming process and the hardware was under load for fifteen minutes while this occurred.

Timeline
20:57 Route process died on router and automatically re-started
21:03 Parings with upstream providers was re-established.
21:03 – 21:20 Route table rebuild and upstream distribution of routes.

What we learned
Automated self-correction worked as expected. The incident surfaced an issue within the Junos software that needs to be investigated.

Planned Actions
Logged ticket with hardware vendor
Upgrade of software for the border router

Recommendations
• Upgrade of software for the border router.
• Confirmation from hardware vendor that upgrade resolves issue.
Oct 21, 20:30 NZDT