Possible issue affecting AWS availability in us-east-1
Incident Report for Learnosity
Resolved
As of 17:06 UTC, Learnosity is no longer seeing any notable issues arising from this AWS outage.

Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have received additional information from AWS.

Thank you, and please reach out if you have any questions or concerns.
Posted Oct 09, 2020 - 13:08 EDT
Monitoring
As of 16:20 UTC, we are moving our loading of assessments back to Operational and will continue to monitor the situation.

While the AWS issue remains open, we have seen a dramatic drop in errors, and sessions continue to be scored within operational limits.

Learnosity Support and Systems Engineering teams are continuing to monitor the issue, and will follow on with additional information as it becomes available.
Posted Oct 09, 2020 - 12:24 EDT
Update
As of 15:52 UTC, we are updating this incident to remove any impact on scoring.

All issues are improving as AWS continues to address the issue, but we're continuing to monitor the delivery of assessments.

Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted Oct 09, 2020 - 11:54 EDT
Update
As of 15:08 UTC, we are broadening the stated affect to Learnosity from the ongoing partial AWS outage, to include delays to scoring. This continues to affect a very small number of customers.

We are starting to see a small number of scoring delays and our asynchronous scoring queue is performing as designed and queueing affected sessions for rescoring. We will continue to monitor all performance and update when we know more. We are still trying to work with AWS to learn more, as well.

Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted Oct 09, 2020 - 11:14 EDT
Update
As of 14:44 UTC, we are still investigating the impact of a partial AWS outage.

We are starting to see some errors where AWS EC2 instances are slowing connections to the Learnosity databases. Attempts to span new EC2 instances are failing, consistent with AWS description of the problem.

This appears to still only affect a single AWS Availabilty Zone, so not all customers are being affected. However, we are continuing updates in case this issue grows.

Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted Oct 09, 2020 - 10:58 EDT
Update
As of 14:33 UTC, we are continuing to investigate the impact of a partial AWS outage.

As a precautionary measure, we're alerting degraded performance in availability of assessments but, according to AWS, this is allegedly only affecting a small subset of EC2 instances launched after October 08, 2020, at 9:37 PM PDT within a single Availability Zone (use1-az2) in the US-EAST-1 Region. So this may not be affecting all customers.

Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted Oct 09, 2020 - 10:35 EDT
Investigating
As of 14:00 UTC, we are investigating any possible impact of a reported AWS issue affecting the us-east-1 region.

We are seeing intermittent availability issues due to this problem..

Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
Posted Oct 09, 2020 - 10:22 EDT
This incident affected: AMER || Assessment (Loading and rendering of assessment UI) and AMER || Data Centric (Scoring endpoint).