Around 10:38 PT on Thursday, August 18, 2022, Inkling's user database came under undue load from a series of database queries which processed a large amount of data, causing high resource consumption on those database servers. In response, a system which prevents excessive load from impacting the database held back further connections. This caused the HTTP requests generating that load to fail, and the database automatically recovered. Inkling Engineering restarted the services which were directly impacted by the database's unavailability, and service was restored after 11 minutes of downtime.
Engineering is investigating this problem from multiple angles, seeking ways of lessening the impact of these queries. These methods include breaking the query into smaller, less impactful subqueries; clearing erroneous data to reduce the amount of data processed by the queries; and implementing partial rate limiting for certain functions.
