At approximately 23:01 PT on Feburary 6, 2024, a cache node experienced a failure and went into an automated recovery process. When the node was restored, a brief failover occurred which caused the Habitat portion of Inkling's platform to become unavailable for approximately 90 seconds. Engineering was notified and responded immediately, but the outage self-resolved, requiring no further action. Engineering has investigated the situation and has determined that the scenario played out in the best possible way, with automated recovery actions being performed immediately upon failure and without the need for further intervention.

The Habitat performance issue has been resolved.

At approximately 3:00 AM PT on February 1, 2024, an AWS-automated process intended to maintain data redundancy while replacing defective hardware caused large amounts of data to be placed on a single server, resulting in its storage volume reaching capacity. This prevented new user and attribute data from being recorded. Inkling Engineering increased the storage capacity of these servers, which resolved the issue of saving incoming changes to this data. All of the jobs which had failed during the incident were re-run to restore the proper data state and ensure proper distribution of Inkdocs. All told, the issue persisted for approximately 5 hours and 36 minutes.

Monitoring surrounding this issue did trigger alerts, but they were set to a low priority which prevented them from alerting the on-call engineer. Engineering has added severity to certain of these alarms to improve response time in the future. Inkling is also investigating multiple approaches to managing the size of this data, which will make this sort of routine automation operation more efficient.

The incident has been resolved

We have resolved the issue and are monitoring.

We have identified the problem and are working towards resolution.

At approximately 12:45 PM PT on December 13, 2023, a server which had been temporarily removed from service for maintenance was put back into rotation. A misconfiguration on that server prevented it from processing the requests it received, resulting in intermittent service of the Inkling reader. During this time - approximately 53 minutes - users of the platform may have noticed some pages responding slowly or returning errors. Refreshing the page may have shown ordinary performance.

To resolve this issue, engineers identified the incorrect configuration and restored it, which immediately brought the platform back to normal. The documentation concerning the maintenance procedure has been updated to include explicit instructions surrounding this issue to prevent any further interruptions. The incident generated a pattern in metrics which we have also set up additional monitoring around to improve response time for this and other potential issues which might present in the same way.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.