Incident
Issues connecting to a webnar streaming server
Affected: Webinars, Live streaming
Resolved# Summary We are following up on an outage on our webinar platform on Thursday October 6th from 06:28 UTC to 07:15 UTC. The outage was caused by one of our webinar streaming servers becoming unavailable due to an issue with our cloud provider, AWS. This issue unfortunately meant that we we not immediate able to restart capacity, and full service for webinars was only restored by 07:15 UTC. During the outage, some webinars were able to run using a failover live server, but any webinar room started before 07:15 UTC may have experienced severe issues. We know that issues such as this affected our customers in significant ways. We want to use this occasion to apologize for the problems caused, and we are committed to using the findings to further improve our service. # Timeline At 06:28 UTC on Thursday, October 6th, a core webinar streaming server stopped responding. At 06:33, our monitoring infrastructure registered the issue and alerted on-duty personnel to respond to the problem. At 06:41, the issue on the streaming server was diagnosed and steps were initiated to start the service on other hardware. At 06:44, steps were initiated to route affected webinars to a failover webinar streaming server. At 07:06, new webinar rooms starting using failover webinar streaming servers. At 07:15, the issues on the failing webinar streaming server were fully resolved and the service back online. # What will TwentyThree do to ensure the problem doesn't re-occur? Based on the experience outlined above, we will be moving the affected webinar server to different hardware, to ensure both performance and quick recoverabilty. Additionally, we identified a need for quicker failover between streaming servers and will be updating our product and internal produces around this within coming weeks. Together this will prevent similar incidents from happening in the future.
ResolvedThis incident has been resolved.
MonitoringA fix has been implemented and we're monitoring the services. We will post full information about the outage this morning later today. We sincerely apologize for any inconvenience caused.
InvestigatingWe are investigating an issue on the platform causing problems in connecting to a webinar server. We will update this thread when we know more.