An infrastructure provider is having network issues
Incident Report for TwentyThree
Postmortem

On June 8th 2021 between 9:54 UTC and 11:02 UTC we have been seeing network issues on our platform that were caused by a global service outage of our content delivery provider Fastly. We use Fastly's global network of servers to optimise the delivery of videos, players and web pages.

The issue affected a large number of their services and as a consequence made it impossible to reach our services. This could mostly be seen through an increase in HTTP 5xx level error messages, such as “500 Domain Not Found” and “503 Service Unavailable”.

During the outage, we used our status page at https://status.twentythree.com to communicate updates to customers, and upon learning about the impact, our specialist team reached out proactively to customers planning live webinars.
We are sorry about the effects of the interruption. Please reach out to our team at support@twentythree.com with any questions should you need further clarification.

Fastly has identified a list of process improvements to ensure outages like this one won't reoccur, and beyond this we have intensified already ongoing work to introduce more content delivery networks to our service.

For further information about the incident from Fastly please refer to the following page: https://www.fastly.com/blog/summary-of-june-8-outage

Posted Jun 17, 2021 - 15:08 CEST

Resolved
This incident has been resolved.
Posted Jun 08, 2021 - 16:30 CEST
Monitoring
Fastly has resolved the problem on their network, and we're seeing full service restored to TwentyThree. We will continue to monitor availability and performance through the next hours. As we know more about the root cause and the mitigation steps, we will update this page.

We sincerely apologize for any inconvenience caused by the outage and invite you to reach out to support@twentythree.com with any questions.
Posted Jun 08, 2021 - 13:02 CEST
Update
The connection issues driven by the Fastly outage is continuing to affect the TwentyThree platform along with other significant internet product and media sites. If you're a planning webinars within the next hour, please reach out to our support team at support@twentythree.com who can assist.
Posted Jun 08, 2021 - 12:49 CEST
Update
We are continuing to monitor this outage, with updated status also being available from Fastly through https://status.fastly.com, including which regions are affected.
Posted Jun 08, 2021 - 12:13 CEST
Identified
We're currently seeing connectivity problems to the platform in some regions, including Scandinavia. The problem is caused by out outage from our global content delivery provider, Fastly, and we're looking at mitigation steps.
Posted Jun 08, 2021 - 12:01 CEST
Update
We are continuing to investigate this issue.
Posted Jun 08, 2021 - 11:58 CEST
Investigating
We are currently monitoring a network issue from our infrastructure provider for global video delivery. An incident is reported for their location in Copenhagen, Denmark, and we will update this thread with more information.
Posted Jun 08, 2021 - 11:54 CEST
This incident affected: TwentyThree Platform, Live streaming, and Webinars.