On the afternoon of 9 June 2020, we experienced an issue with AWS RDS Replication in the us-west-1 region that affected data reads only in Learnosity authoring environments. All data was persisted, and the assessment and analytics stacks were unaffected. We routed all read-only queries to a different region and have logged the issue with AWS for further analysis and remediation.
Posted Jun 09, 2020 - 14:04 EDT
Resolved
As of 5:55 pm UTC, We are have resolved the issue affecting authoring reads in the US west region.
Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required.
Please reach out if you have any questions or concerns.
Posted Jun 09, 2020 - 14:00 EDT
Monitoring
As of 5:15 pm UTC, we have rectified the situation partially affecting reads within the authoring environments.
We will be monitoring the situation in the short term before marking resolved. Assessment and analytics stacks remain unaffected, as does authoring via the Data API. We will follow on with an update and resolution as soon as possible.
Posted Jun 09, 2020 - 13:28 EDT
Update
We are continuing to investigate this issue. All data is still being persisted, all content remains available to the Data API and all published content remains available to the assessment stack. Only retrieval/display of content changes in the authoring UI, made after approximately 3:25 pm UTC, is being delayed. Downgrading to partial outage status.
Posted Jun 09, 2020 - 12:55 EDT
Update
We are continuing to investigate this issue.
Posted Jun 09, 2020 - 12:34 EDT
Investigating
As of 3:25 pm UTC, We are currently experiencing delays affecting authoring UI display of items recently created or edited. This is affecting all regions.
This delay is not affecting the data saved/edited. New items/changes are still available through the assessment and analytics stacks, including the Data API. This has only started to affect updates to the authoring UI.
Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.