Incident
23 Video is up
Resolved23 Video suffered significant downtime over the weekend, totaling almost an hour of measured unavailability Saturday evening and Sunday morning. For much of this time, embedded players, video playback and video websites were accessible to visitors -- but response times were high and performance lackluster. The problems were caused by a combination of two factors: First, we saw an order of magnitude increase in the requests made to a specific API endpoint, which was determined to be run in a non-malicious fashion by an automated script from a 23 Video customer. When this was diagnosed, the site was brought offline and the service response time and availability restored. We have since deployed upgrades to our infrastructure that will detect and shield against such incidents. Second, a misconfiguration in analytics reporting on a set of video players was revealed by the initial incident. This caused requests to be routed in an inefficient manner than worsened the effects of the first issue. We have also deployed an update to the configuration to avoid further incident. We sincerely apologise for the inconvenience caused by this, and are available for further questions by mail or phone.