Server Outage

5 Apr 2021

We apologize for the extended outage, but there has been a fire at the WebNX datacenter in Utah where many of our servers are located.

 

We have just received this update from their communications team:

 

***

Now that we have a better understanding of what happened we would like to give everyone an update.

 

One of our old generators that have worked for years and was recently load tested had a mechanical failure and caught fire resulting in power being cut to our core routers and fire suppression system controlling the fire. Unfortunately, the fire department opted to cut power to the rest of the building as a precaution even though the power systems were independent. We are currently waiting for an emergency inspector to arrive to give the all-clear so we can bring most of the servers in Ogden back online.

 

We would like to thank you for your patience and know that we are doing everything we can to get everyone back online.

***

 

We will continue to update this announcement as information becomes available.

***

April 6/2021 :

Our datacenter is reporting some progress in restoring power to the building. They have announced that by 5:00 PM EST today about 95% of the servers should be restored to active duty. We are optimistic that this includes our cluster, but our team is maintaining communication with the datacenter to keep this interruption of services as short as possible. We will keep you updated.

***

April 7/2021:

latest update from our DC

As of this moment, all of the servers not directly affected are back online. If you are unable to access your server, it can be assumed it is currently down and will remain so until it can be inspected for any damage in the coming weeks. We are optimistic that the vast majority of data stored on these servers is recoverable. Several servers racked in the vicinity of the incident have resumed operation without trouble.

Our focus going forward is to restore functionality and connectivity to as many servers as possible. Every hour, more and more servers are brought back online, and we will continue in that effort over the coming weeks. Timelines are very hard to predict at any stage and many clients’ servers won’t be back online for several weeks.

***

 

Thank you for your understanding.

 

****

 

Newest update from the datacenter:

We are quickly approaching the halfway mark with server inspections. Our rough estimates place about 20% of servers in need of some repairs. These repairs range from something as simple as a battery replacement to a new CPU. Data from these machines appear to mostly have remained intact, with few exceptions. Servers that are still down should be up and running at the very latest by Friday. Thank you so much for all your patience throughout this time. We truly appreciate the loyalty we have seen from our customers and members of the online community.

*****

 

Server forever456 has been restored to working order

******

Server forever078 has been restored to working order