As of 21:14 UTC, the backlog of messages affecting the availability of sessions in the US-East-1 (VA) region has been processed and all function are operating normally.
Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required.
Posted May 16, 2023 - 17:16 EDT
Update
As of 18:30 UTC, we have a correction of the earlier message, the Systems Engineering team is still working on backfilling some of the messages from sessions submitted during this incident. Some data may not yet be available as the backfilling process continues but all data is safely stored and awaiting processing, and there will not be any data loss resulting from this event.
Sessions currently submitted are being processed normally and will be available shortly after submission.
Posted May 16, 2023 - 14:36 EDT
Monitoring
As of 17:00 UTC, the scoring backlog has resolved and all sessions should be available via Data API and Reports API as normal.
Learnosity Support and Systems Engineering teams will continue to monitor the scoring queue for a time to ensure normal operations continue.
Posted May 16, 2023 - 13:01 EDT
Update
As of 16:30 UTC, the scoring backlog is continuing to decrease.
Learnosity Support and Systems Engineering teams are continuing to actively work on this issue, and will follow on with an update and resolution as soon as possible.
Posted May 16, 2023 - 12:35 EDT
Update
As of 16:00 UTC, the Systems Engineering team continues to work on remediation to allow the backlog to decrease as quickly as possible.
Learnosity Support and Systems Engineering teams are continuing to actively work on this issue, and will follow on with an update and resolution as soon as possible.
Posted May 16, 2023 - 12:09 EDT
Update
As of 15:30 UTC, we're seeing a reduction in the scoring backlog, our Systems Engineering team is taking further steps to speed up the recovery process.
Learnosity Support and Systems Engineering teams are continuing to actively work on this issue, and will follow on with an update and resolution as soon as possible.
Posted May 16, 2023 - 11:36 EDT
Identified
As of 15:00 UTC, the Systems Engineering team has identified the source of the resource starvation at the bottom of this issue and has began remediation steps.
Learnosity Support and Systems Engineering teams are continuing to actively work on this issue, and will follow on with an update and resolution as soon as possible.
Posted May 16, 2023 - 11:03 EDT
Update
As of 14:30 UTC, we are still investigating scoring delays of submitted assessments for some customers, causing session data to take longer than normal to become available in Data API and Reports API .
Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted May 16, 2023 - 10:30 EDT
Investigating
As of 13:50 UTC, we are investigating possible scoring delays of submitted assessments in the US-East-1 region.
Authoring and assessment functionality remains largely unaffected, and no data loss is evident. If confirmed, impact will be on availability of submitted sessions via the Data API and Reports API.
Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
Posted May 16, 2023 - 09:53 EDT
This incident affected: AMER || Analytics (AMER Analytics (Summary), Availability of session information).