Our investigation on September 4 found a slow query caused by sudden unusual use of a legacy part of the Learnosity product. The async scoring queue was populated by design and we were immediately alerted to its growth. We identified the slow query in play, and updated the query and database configuration to improve performance. The backup queue emptied quickly and this specific issue will not reoccur.
Posted Sep 08, 2020 - 14:35 EDT
Resolved
As of 14:39 UTC, we have resolved the delay affecting scoring in us-east-1 (VA).
Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required.
Please reach out if you have any questions or concerns.
Posted Sep 04, 2020 - 10:39 EDT
Monitoring
As of 14:35 UTC, we are continuing to monitor an issue affecting delays in scoring.
Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted Sep 04, 2020 - 10:37 EDT
Investigating
As of 14:00 UTC, we are currently investigating a possible slow down in asynchronous scoring in the us-east-1 (VA) region.
Performance is unaffected at the moment, and all saves and submissions are being successfully captured. We will continue to investigate.
Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
Posted Sep 04, 2020 - 10:10 EDT
This incident affected: AMER || Data Centric (Scoring endpoint).