99.99%
No incidents reported.
Slow responses from the TwentyThree platform
ResolvedOn the 27th of February between 20:36 UTC and 20:39 UTC the TwentyThree platform responded with high latency due to a surge of traffic on the platform, which increased faster than the automated solutions for handling more traffic. The higher latency affected the real-time parts of the system such as: * Admin interface * Engagement functionality on webinars It did not affect: * Video playback * Ongoing live streams
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
Webinar registrations from HubSpot workflows not working
ResolvedThe issue relating to processing webinar registrations from HubSpot workflows has been resolved.
InvestigatingWebinar registrations from HubSpot workflows are currently not working.
Slow responses from the TwentyThree platform
ResolvedOn the 9th of February between 09:39 and 09:49 UTC and later between 09:59 and 10:01 UTC the TwentyThree platform would reject a large portion of incoming traffic and it would respond with latency going up to 30 seconds on the accepted traffic. The issue was caused by a large amount of traffic invoked by a bogus configuration of a 3rd party tool. During the issue we received the same amount of traffic during 10 minutes as we would normally receive over the span of a few days. During the incident: * VMP management would not be available or the latency would be very high * Some TwentyThree-hosted webinar Hubs would not be accessible and some embedded video players would not be accessible, based on the status of CDN cache * Ongoing webinars kept streaming, but people would not be able to join the webinars for the duration of the incident Timeline: * 09:39 UTC: we noticed unusually high amount of traffic * 09:40 UTC: we provisioned more resources to handle incoming traffic * 09:41 UTC: we identified the cause, and started working on an application-layer remediation while trying to limit the incoming traffic * 09:47 UTC: we started recovering * 09:49 UTC: we fully recovered * 09:59 UTC: we saw another uptick in incoming traffic, causing the platform to not respond * 10:00 UTC: we started deploying application-layer remediation * 10:01 UTC: with application layer fully deployed we recovered fully To avoid the same issue in the future: * We are putting stricter rate limiting in place * We are continuing our effort to introduce asynchronous behaviour to parts of our system that can put a lot of pressure on the TwentyThree platform when they receive a lot of traffic
ResolvedThis incident has been resolved.
MonitoringThe issue has been resolved. We will provide more information about the incident in the post-mortem section.
MonitoringA fix has been implemented and we are monitoring the results.
IdentifiedThe issue has been identified and a fix is being implemented.
MonitoringA fix has been implemented and we are monitoring the results.
InvestigatingWe are investigating an ongoing issue causing the TwentyThree platform to respond with more delay than usual. We will update this incident once we know more.
No incidents reported.
No incidents reported.
Google Firebase Realtime Database is reporting issues
ResolvedThe issue has been resolved by the vendor.
InvestigatingGoogle Firebase Realtime Database is used in our webinar product to connect status between hosts and speakers. The service is reporting issues on their European cluster, https://status.firebase.google.com/incidents/aJrR8Pu9PFeepbgkWkLm, and we are monitoring status actively.
MaintenanceScheduled Maintenance
CompletedThe scheduled maintenance has been completed.
In progressScheduled maintenance is currently in progress. We will provide updates as necessary.
ScheduledWe will be updating different components on our infrastructure. Different parts of the platform will be severely limited for up to 10 minutes at a time during the maintenance window.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
Availability is (seconds up + seconds unmonitored) ÷ all seconds in the Europe/Copenhagen calendar month; only measured downtime counts against it, Maintenance included.
