Incident
Issues connecting to the platform
Affected: TwentyThree Platform, Webinars
ResolvedOn March 17th 2021, the TwentyThree platform suffered issues caused by a network outage from the internet provider servicing our data center in Copenhagen, Denmark. This location serves as the origin for much of the data and live streaming traffic used with both TwentyThree Video Marketing Platform and TwentyThree Webinars, and as a result both products were severely affected by the outage. The outage started at 7:37am UTC and was ultimately resolved by the provider at 10:14 UTC. During this interval, the network dropped several times and the service was unavailable for a total of 44 minutes. Live streams and webinars throughout this period were affected and cut off. The TwentyThree platform relies on internet connectivity from Sentia and TDC. Our team was notified about the problems by monitoring at 7.37am UTC. The issue was diagnosed as a problem in internet connectivity by 7.42am UTC and then promptly escalated to the provider. During the outage, we used the status page at [https://status.twentythree.net](https://status.twentythree.net) to communicate updates to customers, and upon learning about the severity, our specialist team reached out proactively to customers planning live webinars. Learning from this event, we are establishing produres to provide such proactive updates more quickly to customers, and we will be updating our products to better communicate critical status information. In the wake of the outage, we are also moving up our already-made plans for upgrading our cloud infrastructure in order to enable quicker recovery from severe network problem by failing over to a different provider or geographic location. Yesterday, these problems put our customers in a rough situation and we’re incredibly sorry about the effects of the interruption. Please reach out to our team at [support@twentythree.net](mailto:support@twentythree.net) with any questions should you need further clatification.
ResolvedThis incident has been resolved.
MonitoringA fix has been implemented and we are monitoring the results.
IdentifiedThe issues on accessing both the video and webinar platform this morning (CET) are caused by a failure from one of the Danish internet providers for the core data center where the TwentyThree platform is hosted. During the last two hours, some customers will have been unable connect to the product. We are working to get more clarity of the root causes and will update in a full post mortem after the fact. The will include steps to ensure that outage the issues doesn't repeat. We sincerely apologize for any inconvenience caused.
MonitoringWe are continuing to monitor for any further issues.
MonitoringWe're continuing to monitor network problems on the platform, with one on the core network providers reporting about issues connecting into the data center. All services are operational, but we will update this thread as we know more about the outage and its causes.
IdentifiedThe issue has been identified and a fix is being implemented.
MonitoringA fix has been implemented and we are monitoring the results.
IdentifiedWe're continuing to monitor network problems on the platform, with one on the core network providers reporting about issues connecting into the data center. All services are operational, but we will update this thread as we know more about the outage and its causes.
IdentifiedThe issue has been identified and a fix is being implemented.
MonitoringA fix has been implemented and we are monitoring the results.
IdentifiedWe're starting to see full recovery of the platform, with all network connectivity coming through. We will keep this thead updated.
InvestigatingWe're experiencing an network connectivity problem on our core hosting location and are actively investigating the problem