At 11:31 AM PT on January 11, 2023, Inkling's user database received an unanticipated series of queries which generated significant disk activity. This slowed the database down and prevented other ordinary queries from succeeding. This effect rippled out, causing an outage.
Inkling engineers immediately detected and resolved the issue by restarting critical services. This allowed the user database to recover as well as other affected Inkling services. The outage lasted approximately 4 minutes in total.
Inkling is continuing to monitor these queries and will take further action to prevent this kind of impact in the future. Our primary strategy is to offload this type of query to a dedicated database replica. This would prevent these queries from affecting other services, leaving the site operational even under these adverse conditions.
