API

99.98% uptime
Apr 2023 · 100.0%May · 99.94%Jun · 100.0%
Apr 2023100.0% uptime
May 202399.94% uptime
Jun 2023100.0% uptime
Axis Web Reader
100.0% uptime
Apr 2023 · 100.0%May · 100.0%Jun · 100.0%
Apr 2023100.0% uptime
May 2023100.0% uptime
Jun 2023100.0% uptime
Inkling Web Reader
99.96% uptime
Apr 2023 · 99.88%May · 100.0%Jun · 100.0%
Apr 202399.88% uptime
May 2023100.0% uptime
Jun 2023100.0% uptime

Habitat

99.96% uptime
Apr 2023 · 100.0%May · 99.88%Jun · 100.0%
Apr 2023100.0% uptime
May 202399.88% uptime
Jun 2023100.0% uptime

InkForms

100.0% uptime
Apr 2023 · 100.0%May · 100.0%Jun · 100.0%
Apr 2023100.0% uptime
May 2023100.0% uptime
Jun 2023100.0% uptime

Learning Pathways

100.0% uptime
Apr 2023 · 100.0%May · 100.0%Jun · 100.0%
Apr 2023100.0% uptime
May 2023100.0% uptime
Jun 2023100.0% uptime

Notice history

Jun 2023

No notices reported this month

May 2023

New user searching, automated course assignment degraded
  • Resolved
    Update

    On Monday, May 8 around 6:00 am PDT, Inkling engineering was alerted to the fact that newly created users were not searchable. It was determined that two queues had backed up due to a long running internal process. The process which led to the backup was immediately stopped, and the capacity of a backend component upgraded to process the extra load. The queues began to recover, but not without impact. User updates and automated assignments experienced some delay starting May 4. The user data queue was cleared and caches repopulated fixing the user search issue by 7:05 PM PDT on the day of the incident. The second queue, one for Learning Pathways automated assignments, took until 4:29 PM PDT on the following day to clear. Unfortunately, when that queue temporarily filled, some Learning Pathways events were missed and code needed to be written to replay events. In some cases, this created duplicate assignments which Inkling engineers will delete once the replay is complete, upon customer approval. Duplicate assignments created by some customers as a workaround were deleted earlier this week. Affected customers have been contacted by their CSM. In response to the incident, Inkling has added comprehensive monitoring and alerting for the queues in question. We have also increased the data retention on the queues to reduce the likelihood of overflow. Additional capacity was added to the affected cache and related systems.

  • Resolved
    Resolved

    We have delivered most of the automated assignments that were previously delayed, and all new assignments will be delivered to learners as expected. Our Customer Success team has directly contacted all impacted customers with additional details.

  • Identified
    Identified

    The backlog of automated assignments is complete.

Apr 2023

Inkling Web Reader Performance Degradation
  • Resolved
    Update

    On April 20, 2023, Inkling services experienced an outage starting at 06:52 AM PT which lasted approximately 43 minutes. This is the root cause analysis: A change to the database schema for Inkling's central authentication service was applied overnight, as a first step in the delivery of a new version of the software.  Before that new software could be deployed, an unrelated transient issue appeared related to events emitted by the service. As a result, the authentication service suffered an outage beginning at 06:52 AM PT. Engineering immediately responded by restarting the affected service, which ordinarily would have resolved the issue. However, the service detected the inconsistency with the database schema and attempted to self-heal. A security feature preventing database tampering caused it to enter an infinite retry loop. Upon investigation of the system logs, Inkling personnel identified the root cause, rolled back the schema change to allow the software to initialize normally, and once again restarted services. This restored service to the platform after approximately 43 minutes of unavailability. To prevent a recurrence of similar issues, Inkling has identified several changes that are being implemented: * Add monitoring to detect services in rapid boot cycles, which indicate problems that may not be reported through our standard telemetry. * Improve the logging of the authentication service  * Automate / streamline the process around multi-step software changes & hand-off between the development and DevOps teams.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    The issue is resolved and Inkling engineers are continuing to monitor this closely.

Apr 2023 to Jun 2023