At 13:40 UTC on 8 August 2025, we noticed a brief backup of asynchronous jobs affecting manual session updates in the US-East-1 (VA) region. Possible jobs including manual updates to response scores, session statuses, and session metadata. The queue mechanism worked as expected, and all requests were processed in short order. Out of an abundance of caution, we raised an alert while systems remained operational, but found nothing of concern during our investigation. We hope the unwarranted alert was not a significant distraction.
Posted Aug 08, 2025 - 10:35 EDT
Resolved
As of 14:10 UTC, we've found no negative behavior in the manual update processes and attribute the short burst of jobs to uncharacteristic testing of the new manual grading workflow. The jobs queue functioned as expected, ensuring no loss of requests, and the jobs were processed in a timely fashion.
Learnosity will continue to monitor this activity for a further 60 minutes, but we are calling this resolved at this time. We apologize for the unwarranted alert.
Please reach out if you have any questions or concerns.
Posted Aug 08, 2025 - 10:19 EDT
Update
As of 13:55 UTC, all manual update jobs have been executed and the jobs queue is empty. Requests to manually update session response scores, session statuses, and session metadata should execute as expected. We are still investigating possible reasons for the blip.
Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted Aug 08, 2025 - 09:57 EDT
Investigating
As of 13:40 UTC, we began investigating a possible issue affecting manual update jobs to session response scores, sessions statuses, and session metadata in the US-East-1 (VA) region. We saw a brief buildup in the queue for these jobs, the queue processed as expected, and is now empty. However, we're investigating further, out of an abundance of caution, to see if there are any unexpected causes for the short queue buildup.
Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
Posted Aug 08, 2025 - 09:55 EDT
This incident affected: AMER || Data Centric (Updating session response scores).