Over the past month, Inkling systems have suffered a series of brief outages. Specifically, these outages have occurred between January 11, 2023 and January 25, 2023, and lasted an average of less than 7 minutes. After extensive research, it has been determined that all of these incidents were related to a particular database query that was generated in the area of our platform that handles user authentication. Although the logic of the operation was sound, the database query it generated was not optimized. One fix was to limit the number of processes generating the query. This throttling allowed the query to run, but not in such bulk as to create a problem. The other fix - a correction to the problematic query itself - was deployed to our production servers on January 30, 2023. This resulted in a significant improvement to database performance. We have also implemented more sophisticated techniques for observing the performance of our database and identifying problematic queries. We are engaging with 3rd party database experts to determine if further improvements can be made to our databases' configurations and how our applications interact with those systems.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Over the past month, Inkling systems have suffered a series of brief outages. Specifically, these outages have occurred between January 11, 2023 and January 25, 2023, and lasted an average of less than 7 minutes. After extensive research, it has been determined that all of these incidents were related to a particular database query that was generated in the area of our platform that handles user authentication. Although the logic of the operation was sound, the database query it generated was not optimized. One fix was to limit the number of processes generating the query. This throttling allowed the query to run, but not in such bulk as to create a problem. The other fix - a correction to the problematic query itself - was deployed to our production servers on January 30, 2023. This resulted in a significant improvement to database performance. We have also implemented more sophisticated techniques for observing the performance of our database and identifying problematic queries. We are engaging with 3rd party database experts to determine if further improvements can be made to our databases' configurations and how our applications interact with those systems.

This incident has been resolved.

Over the past month, Inkling systems have suffered a series of brief outages. Specifically, these outages have occurred between January 11, 2023 and January 25, 2023, and lasted an average of less than 7 minutes. After extensive research, it has been determined that all of these incidents were related to a particular database query that was generated in the area of our platform that handles user authentication. Although the logic of the operation was sound, the database query it generated was not optimized. One fix was to limit the number of processes generating the query. This throttling allowed the query to run, but not in such bulk as to create a problem. The other fix - a correction to the problematic query itself - was deployed to our production servers on January 30, 2023. This resulted in a significant improvement to database performance. We have also implemented more sophisticated techniques for observing the performance of our database and identifying problematic queries. We are engaging with 3rd party database experts to determine if further improvements can be made to our databases' configurations and how our applications interact with those systems.

This incident has been resolved.

Over the past month, Inkling systems have suffered a series of brief outages. Specifically, these outages have occurred between January 11, 2023 and January 25, 2023, and lasted an average of less than 7 minutes. After extensive research, it has been determined that all of these incidents were related to a particular database query that was generated in the area of our platform that handles user authentication. Although the logic of the operation was sound, the database query it generated was not optimized. One fix was to limit the number of processes generating the query. This throttling allowed the query to run, but not in such bulk as to create a problem. The other fix - a correction to the problematic query itself - was deployed to our production servers on January 30, 2023. This resulted in a significant improvement to database performance. We have also implemented more sophisticated techniques for observing the performance of our database and identifying problematic queries. We are engaging with 3rd party database experts to determine if further improvements can be made to our databases' configurations and how our applications interact with those systems.

This incident has been resolved.

We just resolved the nonexistant issue!

This is just a test of the Emergency Broadcast System.

At 7:14 AM PT on Wednesday, November 23, 2022, an elevated number of ordinary database queries generated a heavy disk write load on Inkling's user database. This had the effect of locking up an internal authentication service. Since other services rely upon that auth service, a ripple effect occurred where those other services became unable to process their load. During this time, a self-healing mechanism in our database configuration prevented complete overload of the database, allowing it to recover. Inkling engineers responded by restarting the internal authentication service as well as our core internal API service. These things combined, the services were able to restart successfully. The recovery spread out to the other affected services. In total, Inkling systems were affected for approximately 12 minutes. Inkling Engineering is pursuing options to prevent the issue from recurring, including optimizing database memory settings, refactoring the problematic query, and archiving unused data to reduce the query's pressure.

This incident has been resolved.