Important Server issues all servers.

Status
Not open for further replies.

Eric S

Administrator
Staff member
Messages
836
Reaction score
119
Points
43
ALL SERVICES SHOULD BE RESTORED. IF YOU ARE STILL HAVING ISSUES WITH YOUR ACCOUNT PLEASE MAKE A POST AT https://community.x10hosting.com/community/forums/free-hosting.9/

WE ARE CURRENTLY EXPERIENCING SERVER ISSUES WITH X12 AND X14.

YOU CAN TELL WHICH SERVER YOU ARE ON IN YOUR ACCOUNT DASHBOARD IT WILL STATE USERNAME @ SERVER.

Once an update is available we will update this post.

New signups are disabled at this time.

You may see missing files, missing emails and more but please do know this is normal as we are working to restore all services.
 
Last edited:

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
Had a word with the lead tech, unfortunately not really much new to tell.

They are working as fast as possible to get everyone that is affected back online.

Rest assured that your data is secure and backed up properly.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
x11, x13 and x15 should work properly now.

x12 and x14 are still underway and will show irregularities such as missing files during the process.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
There's really not much new in regards to the progress, but I did get a bit information about the cause of it all.

Due to a bug in the cloud software we use we suffered a failure to our underlying storage system. This caused data corruption on the partition containing the user accounts on some of the servers (x11 and x15 got away unharmed).

The restore process is tedious, there's a lot of data involved, but we do have full backups.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
There was a discovery of a DNS issue that mainly affects new accounts (but also anyone changing/adding domains in their account).

This issue basically leads to the domains never being made public and are therefor not able to resolve to the correct IP (think of DNS as a huge phonebook, where domain is the listed name and the IP is the phone number, if this fails the number is never known to the one searching, be it a browser or ftp client, so they can not connect to it).

We are working on getting this resolved as quick as possible alongside with restores of servers x12 and x14.

Due to this DNS issue we have also halted new account creations for the time being.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
The DNS issues should be sorted, anyone that created an account the last few days and have a domain that gives an error about not resolving should begin to resolve properly fairly soon, if it is not already doing so.
 

Anna

I am just me
Staff member
Messages
11,733
Reaction score
578
Points
113
Official statement of what's currently going on and why it is taking such long time to get everyone back up:

Unfortunately, we suffered a failure of our underlying storage system due to a bug in the cloud software we use. This has caused data on the /home partition to become corrupted. We keep multiple daily backups using Acronis of all our systems and immediately implemented our disaster recovery plan to restore the lost data. We have run into another bug that is preventing us from restoring the full set of data all at once, which is the normal and fastest method of restoration. At this time we're stuck doing manual restores of batches of files, which is much much slower. We have this issue escalated to the highest levels at Acronis and hope to have a resolution soon. We have full backups of all data, it's just a matter of getting it back on the servers in place, which we're working on as fast as we can to get done.

This has never happened in over 10 years of being in business and we certainly expect it to not happen in the future.
 
Status
Not open for further replies.
Top