Analytics APIs Experiencing Degraded Performance in US-East-1

Incident Report for Learnosity

Postmortem

Affected Systems and Regions
On 2025-01-09, Learnosity experienced a brief partial outage affecting our analytics stacks, specifically the Reports API and Data API in the AMER region. The issue began at 13:30 UTC and was resolved at 14:06 UTC, lasting 36 minutes. All other API's were unaffected and there was no loss of data.

Investigation
We discovered that a large number of atypical, inefficient Data API queries requested by customers were taking too long to complete. This prevented other queries from running in a timely manner, creating a backlog. It was determined that an additional database index would significantly improve the response times of these types of queries.

Resolution
Immediately upon discovering this issue, the impacted database instances were successfully scaled up to ease the backlog of requests. The additional index was implemented and all remaining queries were processed quickly. Affected APIs returned to normal operations, and further monitoring ensured the issue was fully resolved.

Prevention
Following further testing, the new index is working well and has become a permanent part of the system. We are also adding new automated monitoring and regression testing to ensure similar requests perform as expected.

Posted Jan 14, 2025 - 12:29 EST

Resolved

As of 14:50 UTC, after a further 60 minutes of issue-free monitoring, we are resolving this issue affecting availability of session information via our Analytics stack. Reports API and Data API are fully operational.

Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required.

Please reach out if you have any questions or concerns.

Posted Jan 09, 2025 - 09:50 EST

Monitoring

As of 14:10 UTC, Learnosity has restored full service for all new requests to all Analytics APIs.

We are monitoring stability and analyzing all queued requests.

Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

Posted Jan 09, 2025 - 09:13 EST

Update

As of 14:00 UTC, Learnosity is working on identifying the cause of degraded performance in our Analytics APIs, Reports and Data.

Authoring and Assessment stacks remain unaffected

Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

Posted Jan 09, 2025 - 09:02 EST

Investigating

As of 13:30 UTC, We are currently investigating a possible performance degradation affecting Reports API and Data API results in US-East-1.

Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Posted Jan 09, 2025 - 08:33 EST

This incident affected: AMER || Analytics (Loading and rendering of reports, Availability of session information).