Important Server issues all servers.

Status
Not open for further replies.

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
You won't be able to until your server is completed.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
There's really not much new in regards to the progress, but I did get a bit information about the cause of it all.

Due to a bug in the cloud software we use we suffered a failure to our underlying storage system. This caused data corruption on the partition containing the user accounts on some of the servers (x11 and x15 got away unharmed).

The restore process is tedious, there's a lot of data involved, but we do have full backups.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
There was a discovery of a DNS issue that mainly affects new accounts (but also anyone changing/adding domains in their account).

This issue basically leads to the domains never being made public and are therefor not able to resolve to the correct IP (think of DNS as a huge phonebook, where domain is the listed name and the IP is the phone number, if this fails the number is never known to the one searching, be it a browser or ftp client, so they can not connect to it).

We are working on getting this resolved as quick as possible alongside with restores of servers x12 and x14.

Due to this DNS issue we have also halted new account creations for the time being.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
The DNS issues should be sorted, anyone that created an account the last few days and have a domain that gives an error about not resolving should begin to resolve properly fairly soon, if it is not already doing so.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
Official statement of what's currently going on and why it is taking such long time to get everyone back up:

Unfortunately, we suffered a failure of our underlying storage system due to a bug in the cloud software we use. This has caused data on the /home partition to become corrupted. We keep multiple daily backups using Acronis of all our systems and immediately implemented our disaster recovery plan to restore the lost data. We have run into another bug that is preventing us from restoring the full set of data all at once, which is the normal and fastest method of restoration. At this time we're stuck doing manual restores of batches of files, which is much much slower. We have this issue escalated to the highest levels at Acronis and hope to have a resolution soon. We have full backups of all data, it's just a matter of getting it back on the servers in place, which we're working on as fast as we can to get done.

This has never happened in over 10 years of being in business and we certainly expect it to not happen in the future.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
Current status:

We have about 50% accounts restored on x12

All accounts but a handful that we had issues with should now be working on x14.
Please start a new support thread if you are on x14 and your site is still not loading.
 
Status
Not open for further replies.
Top