Over the past month, Inkling systems have suffered a series of brief outages. Specifically, these outages have occurred between January 11, 2023 and January 25, 2023, and lasted an average of less than 7 minutes. After extensive research, it has been determined that all of these incidents were related to a particular database query that was generated in the area of our platform that handles user authentication. Although the logic of the operation was sound, the database query it generated was not optimized. One fix was to limit the number of processes generating the query. This throttling allowed the query to run, but not in such bulk as to create a problem. The other fix - a correction to the problematic query itself - was deployed to our production servers on January 30, 2023. This resulted in a significant improvement to database performance. We have also implemented more sophisticated techniques for observing the performance of our database and identifying problematic queries. We are engaging with 3rd party database experts to determine if further improvements can be made to our databases' configurations and how our applications interact with those systems.

This incident has been resolved.