ResolvedThe server issues have been resolved and full service restored. At around 14:20 a routine restart of a web server caused a cascade of failures across all of our web servers. We were notified immediately and set to work on restoring the service. Around 14:40 we started to see normal service on the platform at a degraded performance and over the following 10 minutes restored service to normal performance. Steps are being taken to avoid similar failures in the future.
IdentifiedWe are slowly restoring service across the platform, but performance may be temporarily affected while we get every server fully up and running.
InvestigatingWe are seeing issues with our web server affecting video site and backend access. We are investigating and will provide an update once we know more.
History
Product Availability
99.95%
Past Events
29 Feb 2016
No incidents reported.
28 Feb 2016
No incidents reported.
27 Feb 2016
No incidents reported.
26 Feb 2016
No incidents reported.
25 Feb 2016
24 Feb 2016
No incidents reported.
23 Feb 2016
No incidents reported.
22 Feb 2016
No incidents reported.
21 Feb 2016
No incidents reported.
20 Feb 2016
No incidents reported.
19 Feb 2016
No incidents reported.
18 Feb 2016
No incidents reported.
17 Feb 2016
No incidents reported.
16 Feb 2016
No incidents reported.
15 Feb 2016
No incidents reported.
14 Feb 2016
No incidents reported.
13 Feb 2016
No incidents reported.
12 Feb 2016
No incidents reported.
11 Feb 2016
No incidents reported.
10 Feb 2016
Resolved23 Video was affected by a major outage in at round 14:30 CEST caused by a change in traffic patterns that wasn't well-handled by our internal caching systems. The meant that undue stress was put on the core service responsible for serving data requests -- and requests to the platform either become slow or timed out entirely. The problem was resolved with a deployment of fix for the local caching issue, and service was restored after 12 minutes of unsteady delivery.
9 Feb 2016
No incidents reported.
8 Feb 2016
No incidents reported.
7 Feb 2016
No incidents reported.
6 Feb 2016
No incidents reported.
5 Feb 2016
No incidents reported.
4 Feb 2016
No incidents reported.
3 Feb 2016
ResolvedAll new uploads are now being processed correctly. If a previous upload is not present in your video backend within the next couple of minutes, please contact support (support@23company.com) and we will have it resolved in no time.
MonitoringA congestion issue caused all uploads to be delayed, but it has been resolved. We will keep this issue open while we monitor the continued performance of video uploads and processing.
InvestigatingWe are seeing a number of errors with video uploads across different sites. We are investigating and will provide an update once we have identified and mitigated the error.
ResolvedThis incident has been resolved.
MonitoringThe database backup has been successfully restored, but we are still reprocessing the data between the backup date (February 1st) and now. Data prior to February 1st is not affected. Our current estimate is 6 hours for the data to be fully up-to-date.
IdentifiedOur backup procedure is running and our current estimate is that restoration + reprocesing of the missed data will be done by around midnight tonight (February 3rd). Individual video statistics as well as site-wide statistics are still available from the "Video" tab in the meanwhile.
IdentifiedOur analytics database server crashed unexpectedly, causing a corruption with the current data. We are restoring the data from a backup, but due to the volume of data it will take some time.
2 Feb 2016
No incidents reported.
1 Feb 2016
No incidents reported.
Availability is (seconds up + seconds unmonitored) ÷ all seconds in the Europe/Copenhagen calendar month; only measured downtime counts against it, Maintenance included.