Inkling - Habitat Performance Degradation – Incident details

Habitat Performance Degradation

Resolved
Degraded performance
Started about 1 year agoLasted about 6 hours

Affected

Habitat

Degraded performance from 3:36 PM to 9:18 PM

Updates
  • Resolved
    Update

    On September 21, 2023, at 7:18 AM PT, the Inkling service which processes analytics events came under a spike of abnormally heavy load. Inkling engineers responded by forcing the analytics application to temporarily store these events on the server instead of processing them immediately. This allowed the service to recover. By 9:45 AM PT, the analytics service was operating normally.

    The events which were collected on the servers have been replayed; they only had delayed delivery. During the approximately two and a half hours of this incident, the platform remained functional, although users may have experienced slow behavior for short, intermittent periods.

    To prevent this from happening in the future, additional capacity was added to the service. Engineers are also working on moving the processing of these events to a more robust system already in use by other platform components, which is designed to avoid this kind of load issue entirely.

  • Resolved
    Resolved

    This incident has been resolved. Please contact support if you have any outstanding issues.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Update

    The problem has been identified, and engineers are currently working toward a resolution. The Inkling platform is operational at this time, but some backend event processing may be delayed.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    Inkling has received reports of Habitat performance degradation and our team is currently investigating.