As of 16:55 UTC, after two hours of monitoring, all scoring queues remain empty as sessions are being scored promptly. We are resolving this issue.
Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required.
Please reach out if you have any questions or concerns.
Posted Oct 02, 2023 - 13:14 EDT
Update
As of 14:55 UTC, all queued sessions have completed scoring and new sessions are moving through the scoring process smoothly. Analytical data and Firehose messages should be available for all sessions.
No authoring or assessment flows were affected, all submissions were successfully saved without data loss, and the small subset of customers in the VA region that were affected should again be fully operational.
Learnosity Support and Systems Engineering teams will continuing to monitor the issue, and will follow on with an update and resolution as soon as possible.
Posted Oct 02, 2023 - 10:59 EDT
Monitoring
As of 14:30 UTC, we identified and corrected the issue. Our engineers are now monitoring the queue which is diminishing quickly.
Learnosity Support and Systems Engineering teams are monitoring the issue, and will follow on with an update and resolution as soon as possible.
Posted Oct 02, 2023 - 10:49 EDT
Investigating
As of 14:00 UTC, we are investigating an issue with asynchronous scoring delays affecting our US-East-1 region.
Note: Submissions are not affected, resulting in no data loss, and no other APIs are affected. Sessions are being successfully stored and submitted for scoring, going into a temporary backlog queue. This slows down availability of analytics, including notification delays in the otherwise fully functional Firehose service.
Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
Posted Oct 02, 2023 - 10:06 EDT
This incident affected: AMER || Analytics (Availability of session information) and AMER || Data Centric (Updating session response scores, Firehose).