Issue affecting Scoring in US region

Incident Report for Learnosity

Postmortem

One of the databases that handle the backend storage of sessions blocked writes which caused a slowdown in availability of session data to the Data API. Failsafes built into Learnosity's system ensured that there was no data loss, and sessions that could not be written immediately remained in a queue for processing. Rebooting an RDS instance solved the issue and the remaining sessions were then quickly processed and the queue was cleared.

Because there was no data loss, this issue temporarily affected availability of session reports/reviews, had a possible impact on adaptive sessions where scoring decisions affected next item delivery and resuming sessions during this period, and may have contributed isolated effects to the assessment stack where custom implementations queried session results before follow-up actions could occur (such as determining which assessment to deliver based on the results of a preliminary diagnostic assessment). We apologize for any inconvenience this may have caused.

There were no log entries showing any failures, and other databases were still writing correctly. We are following up with AWS to see if there is any more information available. Thank you.

Posted Sep 01, 2020 - 12:30 EDT

Resolved

As of 8:00 UTC, We are have resolved the delays affecting scoring in the US region.

No known data loss occurred, and sessions previously delayed will now, or soon, be available to the assessment and analytics stacks.

Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required.

Please reach out if you have any questions or concerns.

Posted Aug 27, 2020 - 16:35 EDT

Update

As of 7:40 pm UTC, We are continuing our monitoring of delays previously affecting scoring in the US region.

All previously delayed sessions have been scored and should be available via the analytics stacks. Any assessment decisions based on scoring should also be functioning in client implementations of the assessment stack

Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Posted Aug 27, 2020 - 15:49 EDT

Monitoring

As of approximately 7:15 UTC, we are monitoring the scoring of sessions previously delayed in the US Region.

We are currently processing all queued submissions which will then become available via the Data and Reports APIs.

Learnosity Support and Systems Engineering teams are actively addressing the issue, and will follow on with an update and resolution as soon as possible.

Posted Aug 27, 2020 - 15:30 EDT

Identified

As of approximately 6:05 UTC, we have identified delays affecting scoring in the US Region.

This may also temporarily affect loading of Reports that are based on yet-to-be scored sessions, the manual updating of question scores, and possibly adaptive testing decisions based on score.

Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Posted Aug 27, 2020 - 14:30 EDT

Investigating

As of approximately 5:08 UTC, we are experiencing delays affecting session scoring in the US Region.

This may cause delays in session responses, and/or showing sessions as completed. All submissions are being recorded, so no data loss is expected.

Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

Posted Aug 27, 2020 - 14:24 EDT

This incident affected: AMER || Analytics (Loading and rendering of reports, Adaptive session information) and AMER || Data Centric (Updating session response scores, Scoring endpoint).