As of 07:50 PST, AWS have updated that they have restored all kinesis traffic and resolved error rates, with immediate actions in place to prevent reoccurrence. As a result, Learnosity will mark this issue as resolved, and will update with any further information or postmortem should any subsequent information from AWS shed light on route causes, or mitigation strategies.
Posted Nov 26, 2020 - 04:53 EST
Update
We are continuing to monitor for any further issues.
Posted Nov 26, 2020 - 04:51 EST
Monitoring
As of 06:00am UTC, AWS have reported resolution for the service outage affecting the Kinesis stream, EventBridge and other related services though recovery efforts on their part remain ongoing.
* Events API - Available * Firehose Service - Available * Live Progress reporting - Available
Learnosity systems are now fully operational though our Support and Engineering teams will continue to monitor for further issues, many thanks for your patience and understanding.
Posted Nov 26, 2020 - 01:09 EST
Investigating
As of 12:50 am UTC, Learnosity continues to monitor steady improvements with the AWS service outage affecting the Kinesis stream and EventBridge, as well as other related services. We are delaying calling this resolved until Kinesis has recovered, though only the following Learnosity services are impacted to varying degrees:
* Events API - Unavailable * Firehose Service - Unavailable * Live Progress reporting - Unavailable
The following systems continue to work reliably:
* Authoring - Available * Assessment - Delivery and capture of student assessments - Available * Analytics - Reporting - Available (except live progress and firehose)
Learnosity Support and Systems Engineering teams are diligently monitoring the issue, and will follow on with an update and resolution as soon as possible. Thank you for your patience and understanding.
Posted Nov 25, 2020 - 19:54 EST
Update
As of 10:50 pm UTC, we continue to monitor the AWS outages.
Services with knock on effects impacting Learnosity, such as EvenBridge and Kinesis, have started to recover, and we've thus far seen no significant impact from other services. Our infrastructure team has successfully managed scaling when required, and we've had no reports of CloudFront failures. Because Firehose and the Events API (including Live Progress reporting) may still experience delays until the AWS issues are resolved, we will continue to monitor.
Learnosity Support and Systems Engineering teams are continuing to be vigilant, and will follow on with an update and resolution as soon as possible.
Posted Nov 25, 2020 - 17:55 EST
Update
As of 8:28 UTC, we are continuing to monitor the AWS Kinesis incident that is ongoing.
Our current status is as follows.
Affected areas: * Events API - Unavailable * Firehose Service - Unavailable * Live Progress reporting - Unavailable
The following systems are all working reliably - though we are observing reduced traffic as many of our customers are also affected by the AWS issue.
Unaffected areas: * Authoring - Available * Assessment - Delivery and capture of student assessments - Available * Analytics - Reporting - Available (except live progress and firehose)
Learnosity Support and Systems Engineering teams are continuing to actively monitor the issue, and will follow on with an update and resolution as soon as possible.
As of 6:52 UTC, Learnosity continues to monitor the widespread AWS service degradation.
Note that, while CloudFront has been added to the list of affected AWS services, only customer reporting is affected, not content delivery. Similarly, our infrastructure team is personally monitoring traffic to mitigate any affect on autoscaling. Only Learnosity's Firehose and the Events API have been affected this far, and all other APIs and services are operating normally.
Learnosity Support and Systems Engineering teams are continuing to actively monitor the issue, and will follow on with an update and resolution as soon as possible.
(Retriggered to send notifications.)
Posted Nov 25, 2020 - 14:03 EST
Investigating
As of 6:52 UTC, Learnosity continues to monitor the widespread AWS service degradation.
Note that, while CloudFront has been added to the list of affected AWS services, only customer reporting is affected, not content delivery. Similarly, our infrastructure team is personally monitoring traffic to mitigate any affect on autoscaling. Only Learnosity's Firehose and the Events API have been affected this far, and all other APIs and services are operating normally.
Learnosity Support and Systems Engineering teams are continuing to actively monitor the issue, and will follow on with an update and resolution as soon as possible.
Posted Nov 25, 2020 - 14:01 EST
Update
As of 4:06 UTC, AWS issues with Kinesis continue, but effects appear to be limited to Learnosity's Firehose and Event Bus services.
Basic events, scoring, and availability of session data is functioning normally, as are all other Learnosity services such as authoring, and delivery of assessments. We are again revising our degraded performance status to apply only to Firehose and Event Bus use.
Learnosity Support and Systems Engineering teams are continuing to monitor the issue, and will follow on with an update and resolution as soon as possible.
Posted Nov 25, 2020 - 11:11 EST
Monitoring
As of 3:13 UTC, continuing AWS issues may start to cause delays in availability of session information for all, not just those using Firehose.
Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted Nov 25, 2020 - 10:17 EST
Identified
As of 2:58 UTC, AWS is reporting increased error rates in Kinesis Data Streams APIs in the US-EAST-1 Region.
Learnosity expects this to have an effect on Learnosity Firehose, and possibly Events API, users, so we're moving proactively to alert for degraded performance in these areas. Note that the listed availability of Session Information is currently limited to Firehose users. No other functionality is expected to be affected at the moment. All other authoring, assessment, and analytics services are operating normally.
Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
Posted Nov 25, 2020 - 10:01 EST
Investigating
As of 2:45 UTC, we are currently looking at possible impact of reported degradation of AWS Kinesis performance. As of this posting, all Learnosity services are fully operational. We are looking for possible delays in Firehose messages that may be caused by this AWS issue.
Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
Posted Nov 25, 2020 - 09:50 EST
This incident affected: AMER || Analytics (Availability of session information, Live Progress (Live Activity by User) report) and AMER || Assessment (Sending and receiving Live Activity Status by User report events).