On September 5, 2023, from 11:42 AM PT to 11:46 AM PT, the Inkling platform experienced a performance issue in our central authentication service, causing 4 minutes of downtime. The effect of the performance issue was compounded by the coincidental deployment of a patch intended to partially alleviate that problem. The site recovered following the deployment.
Engineering teams have responded to these issues in the following ways:
- Many of the known performance issues have been isolated and corrected.
- Hardware capacity has been added to this service to help support it during peak demand.
- Additional performance monitoring tools have been installed to help detect this type of issue going forward. This has already helped identify a number of places where performance of this critical service can be improved.
- Engineering work to identify and resolve this type of problem has been prioritized.
