Incident
Loss of video transmission for the audience of a small number of webinars.
ResolvedBetween 14:37 UTC and 14:41 UTC on the 22nd of May we observed a loss of video transmission on a number of webinars live at the time. The issue was caused by the introduction of a new feature around the mixing of webinar streams. The introduction of this feature, although in beta and not accessible to the majority of our users, required an overhaul of the inner workings controlling how images and video are displayed on screen. This is what ultimately introduced the issue that some users experienced. The issue would've been felt by users performing a specific sequence of mixing actions during the 4 minute response window, or after this period, provided they had been in the webinar room during that timespan. A partial amount of webinars managed to be manually recovered and resumed without issues, but unfortunately, for some of the affected users, it resulted in the webinars ending up in an unrecoverable state. During the whole process, both video and audio transmission managed to reach our servers, as well as other speakers and producers, but only audio ended up reaching the webinar audience. As such, integral recording of all the webinar sources are available and can be recovered using our RawMode feature. Timeline: At 14:37 UTC we received the first report of the issue At 14:38 UTC, post initial diagnostic steps the issue got escalated to engineering Between 14:39 UTC and 14:40 UTC we attempted to recover the affected webinars while attempting to identify the cause of the issue At 14:40 UTC we identified the recently made release as the primary suspect for the outage At 14:41 UTC we rolled back on the introduction of this feature, ensuring the issue wouldn't spread further We spent the following minutes attempting to restore the state of affected webinars with partial success.