Inkling - Notice history

On April 9, 2024, Inkling restarted the central API services after adding additional capacity. Following the restart, the services that process central queue items failed to connect to the queue properly. This caused items in the queue to have their processing delayed. This was mostly seen in a delay publishing Inkdocs.

To resolve the issue, Inkling engineers reconnected the services to the queue. By 10:30 AM PT on April 10, 2024, the services processed the backlog and resumed normal service. No data was lost in this process, only delayed. Changes to the queueing system were made to ensure this does not reoccur.

All previously delayed project publications have been processed, and there are no longer any delays expected for new tasks.

At 11:58 AM PT on April 9, 2024, a scaling event reduced the number of Inkling's main API instances in response to an unusual reduction in traffic. When traffic increased, the scaling increased the number of instances but those took some time to come online. During this time, the remaining instances were overwhelmed, initially leading to some intermittent timeouts and ultimately to 8 minutes of platform downtime between 12:30 and 12:38 PM PT.

To mitigate the immediate problem, Engineering added capacity and restarted the core internal services. This restored the platform. The scaling logic has been updated to ensure sufficient capacity at all times.

At approximately 23:01 PT on Feburary 6, 2024, a cache node experienced a failure and went into an automated recovery process. When the node was restored, a brief failover occurred which caused the Habitat portion of Inkling's platform to become unavailable for approximately 90 seconds. Engineering was notified and responded immediately, but the outage self-resolved, requiring no further action. Engineering has investigated the situation and has determined that the scenario played out in the best possible way, with automated recovery actions being performed immediately upon failure and without the need for further intervention.

The Habitat performance issue has been resolved.

At approximately 3:00 AM PT on February 1, 2024, an AWS-automated process intended to maintain data redundancy while replacing defective hardware caused large amounts of data to be placed on a single server, resulting in its storage volume reaching capacity. This prevented new user and attribute data from being recorded. Inkling Engineering increased the storage capacity of these servers, which resolved the issue of saving incoming changes to this data. All of the jobs which had failed during the incident were re-run to restore the proper data state and ensure proper distribution of Inkdocs. All told, the issue persisted for approximately 5 hours and 36 minutes.

Monitoring surrounding this issue did trigger alerts, but they were set to a low priority which prevented them from alerting the on-call engineer. Engineering has added severity to certain of these alarms to improve response time in the future. Inkling is also investigating multiple approaches to managing the size of this data, which will make this sort of routine automation operation more efficient.

The incident has been resolved

We have resolved the issue and are monitoring.

We have identified the problem and are working towards resolution.