Incident
Slow responses from the the platform
Affected: TwentyThree Platform, Webinars
Resolved### Summary We are following up on the outage on **March 2nd** from 08:59 UTC to 09:23 UTC. The outage was caused by a capacity issue on an internal database responsible for our Audience service, used when joining webinars. When the problem started, we saw a steep rise in traffic to the database, which caused it to respond slowly due to a lack of resources. The drop in response time caused queueing of requests to the TwentyThree platform and an unresponsive experience throughout. We know that issues such as this one significantly impact our customers – and want to use this occasion to apologize for the problems caused. In addition, we are committed to using the findings to further improve our service. ### Timeline At 08:59 UTC on Wednesday, March 2nd, the capacity issue detailed above started presenting. At 09:00, our monitoring infrastructure registered the issues and alerted on-duty personnel to respond to the problem. At 09:02, the responding team acknowledged the problem, and work commenced to identify and resolve the issue. At 09:13, the incident was communicated to customers through our status page at https://status.twentythree.com. At 09:15, We directed traffic away from the stressed database, and we started seeing recovery across all affected services. At 09:23, all services were fully available to all customers. ### What will TwentyThree do to ensure the problem doesn't re-occur? We have subsequently scaled up the internal database significantly, and we are investigating a predictive scaling of the databases. Additionally, we have made initial changes to our application to avoid blocking the user experience where one service's response time affects the complete platform, and we are planning further changes based on these findings. Together this will prevent similar incidents from happening in the future.
ResolvedThis incident has been resolved.
MonitoringA fix has been implemented and we're monitoring results.
InvestigatingWe are investigating an issue on the platform causing slow loading. We will update this thread when we know more.