On Tuesday, May 28th, 2019, the Inkling platform was down for 11 minutes starting at 11:27 AM PST. A hardware failure on one of our internal load balancers caused requests to time out and prevented automatic recovery failovers from succeeding. The engineering team was alerted to the issue and simultaneously worked to restore the load balancer and restart internal services which were still impacted by the load balancer failure. After investigating the root cause and why the outage lasted longer than expected, the team has documented how to deal with this kind of outage more quickly and will be improving tooling to rapidly identify the cause of major outages.
