On September 21, 2023, at 7:18 AM PT, the Inkling service which processes analytics events came under a spike of abnormally heavy load. Inkling engineers responded by forcing the analytics application to temporarily store these events on the server instead of processing them immediately. This allowed the service to recover. By 9:45 AM PT, the analytics service was operating normally.

The events which were collected on the servers have been replayed; they only had delayed delivery. During the approximately two and a half hours of this incident, the platform remained functional, although users may have experienced slow behavior for short, intermittent periods.

To prevent this from happening in the future, additional capacity was added to the service. Engineers are also working on moving the processing of these events to a more robust system already in use by other platform components, which is designed to avoid this kind of load issue entirely.

This incident has been resolved. Please contact support if you have any outstanding issues.

We implemented a fix and are currently monitoring the result.

The problem has been identified, and engineers are currently working toward a resolution. The Inkling platform is operational at this time, but some backend event processing may be delayed.

We are continuing to work on a fix for this incident.

This incident has been resolved.

On September 13, 2023, from 7:14 AM PT to 7:22 AM PT, performance issues in Inkling's central authentication service created failures throughout the rest of the platform. The problems related to a dependency between the authentication service and another internal application which created latency in the ordinary handling of authentication requests.

Engineering teams have responded to these issues in the following ways:

• Many of the known performance issues have been isolated and corrected.
• Hardware capacity has been added to this service to help support it during peak demand.
• Additional performance monitoring tools have been installed to help detect this type of issue going forward. This has already helped identify a number of places where performance of this critical service can be improved.
• Engineering work to identify and resolve this type of problem has been prioritized.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

On September 7, 2023, from 10:17 AM PT to 10:18 AM PT, performance issues in Inkling's central authentication service created failures throughout the rest of the platform. This created a queue of requests in a load balancer, stalling requests until sufficient resources freed up to process them. This led to approximately 1 minute of downtime in the platform.

Engineering teams have responded to these issues in the following ways:

• Many of the known performance issues have been isolated and corrected.
• Hardware capacity has been added to this service to help support it during peak demand.
• Additional performance monitoring tools have been installed to help detect this type of issue going forward. This has already helped identify a number of places where performance of this critical service can be improved.
• Engineering work to identify and resolve this type of problem has been prioritized.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

On September 5, 2023, from 11:42 AM PT to 11:46 AM PT, the Inkling platform experienced a performance issue in our central authentication service, causing 4 minutes of downtime. The effect of the performance issue was compounded by the coincidental deployment of a patch intended to partially alleviate that problem. The site recovered following the deployment.

Engineering teams have responded to these issues in the following ways:

• Many of the known performance issues have been isolated and corrected.
• Hardware capacity has been added to this service to help support it during peak demand.
• Additional performance monitoring tools have been installed to help detect this type of issue going forward. This has already helped identify a number of places where performance of this critical service can be improved.
• Engineering work to identify and resolve this type of problem has been prioritized.

This incident has been resolved.

On the morning of August 31, 2023, the servers that process media when projects are published came under abnormally heavy load due to a defect in this service. At 7:22 AM Pacific Time, these servers became unresponsive. This resulted in some images not loading in Habitat and a small number of failed builds. Inkling Engineering responded by restarting the relevant services. The issues were resolved by 7:34. Additional capacity was added to these servers to further mitigate these issues while Engineering diagnosed the problem. A fix was later deployed to correct this problem permanently. All builds which failed during the time of the incident were rerun shortly after service was restored. All of those builds completed successfully.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

We are continuing to work on a fix for this incident.