As of 17:55 UTC, we identified an adaptive Redis cache that was intermittently refusing a small percentage of connections. Self-hosted adaptive makes extensive use of this cache due to the need to store more state information supplied by the self-hosted client. We replaced the cache database, routed all traffic to the new instance, and saw an immediate, dramatic reduction of refused connections. We've monitored for approximately 90 minutes and all activity indicates normal operation of all APIs. We are resolving this issue and will continue to monitor customer comms.
Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required.
Please reach out if you have any questions or concerns.
Posted Dec 05, 2023 - 14:35 EST
Investigating
As of 17:25 UTC, we are investigating a possible issue affecting Self-Hosted Adaptive testing in the US-East-1 region. Due to the very small number of customers affected, we've not yet confirmed if this is something related to Learnosity APIs or the self-hosted algorithm portion of the solution. Most customers using this feature appear to be unaffected, but we're raising this alert out of an abundance of caution.
Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
Posted Dec 05, 2023 - 12:29 EST
This incident affected: AMER || Assessment (Self-Hosted Adaptive services).