Last night we experienced some issues in our data center that affected a portion of our clients. We wanted to take a moment to let you know what happened.
At approximately 6:22 PM MST our primary Data Center experienced an electrical transient. An Automatic Transfer Switch on one of our legs of power failed, causing some servers to lose power. Our systems are designed with multiple, redundant sources of power that automatically transfer as needed to ensure servers stay up and running. In this instance, however, the switch itself failed. It also sent a spike back to one of our transformers, causing additional problems. Our technical teams of Data Center Technicians, System Administrators, and power experts responded immediately and worked continuously to bring all our customers back online.
Our data center utilizes many redundant systems including backup generators and 4 different UPS legs of power. These are tested every two weeks and regular maintenance is performed on all systems in order to provide the most reliable service possible. However, tonight our services did not live up to your—or our own—expectations. We are evaluating every aspect of this particular failure to eliminate the possibility of recurrence.
We pride ourselves on delivering outstanding reliability and an unbeatable customer experience. Any feedback on how to improve our platform and services, as well as suggestions on how we can better communicate with you in the future, is welcome and appreciated. Thank you for your continued patience and support as we resolve this issue.
