Important Server issues all servers.
- Thread starter Eric S
- Start date
- Status
If your service is on x12 or x14 you will be experiencing these issues until everything is fully resolved.
- Messages
- 11,750
- Reaction score
- 581
- Points
- 113
There's really not much new in regards to the progress, but I did get a bit information about the cause of it all.
Due to a bug in the cloud software we use we suffered a failure to our underlying storage system. This caused data corruption on the partition containing the user accounts on some of the servers (x11 and x15 got away unharmed).
The restore process is tedious, there's a lot of data involved, but we do have full backups.
Due to a bug in the cloud software we use we suffered a failure to our underlying storage system. This caused data corruption on the partition containing the user accounts on some of the servers (x11 and x15 got away unharmed).
The restore process is tedious, there's a lot of data involved, but we do have full backups.
- Messages
- 11,750
- Reaction score
- 581
- Points
- 113
There was a discovery of a DNS issue that mainly affects new accounts (but also anyone changing/adding domains in their account).
This issue basically leads to the domains never being made public and are therefor not able to resolve to the correct IP (think of DNS as a huge phonebook, where domain is the listed name and the IP is the phone number, if this fails the number is never known to the one searching, be it a browser or ftp client, so they can not connect to it).
We are working on getting this resolved as quick as possible alongside with restores of servers x12 and x14.
Due to this DNS issue we have also halted new account creations for the time being.
This issue basically leads to the domains never being made public and are therefor not able to resolve to the correct IP (think of DNS as a huge phonebook, where domain is the listed name and the IP is the phone number, if this fails the number is never known to the one searching, be it a browser or ftp client, so they can not connect to it).
We are working on getting this resolved as quick as possible alongside with restores of servers x12 and x14.
Due to this DNS issue we have also halted new account creations for the time being.
- Messages
- 11,750
- Reaction score
- 581
- Points
- 113
Official statement of what's currently going on and why it is taking such long time to get everyone back up:
Unfortunately, we suffered a failure of our underlying storage system due to a bug in the cloud software we use. This has caused data on the /home partition to become corrupted. We keep multiple daily backups using Acronis of all our systems and immediately implemented our disaster recovery plan to restore the lost data. We have run into another bug that is preventing us from restoring the full set of data all at once, which is the normal and fastest method of restoration. At this time we're stuck doing manual restores of batches of files, which is much much slower. We have this issue escalated to the highest levels at Acronis and hope to have a resolution soon. We have full backups of all data, it's just a matter of getting it back on the servers in place, which we're working on as fast as we can to get done.
This has never happened in over 10 years of being in business and we certainly expect it to not happen in the future.
Unfortunately, we suffered a failure of our underlying storage system due to a bug in the cloud software we use. This has caused data on the /home partition to become corrupted. We keep multiple daily backups using Acronis of all our systems and immediately implemented our disaster recovery plan to restore the lost data. We have run into another bug that is preventing us from restoring the full set of data all at once, which is the normal and fastest method of restoration. At this time we're stuck doing manual restores of batches of files, which is much much slower. We have this issue escalated to the highest levels at Acronis and hope to have a resolution soon. We have full backups of all data, it's just a matter of getting it back on the servers in place, which we're working on as fast as we can to get done.
This has never happened in over 10 years of being in business and we certainly expect it to not happen in the future.
- Messages
- 11,750
- Reaction score
- 581
- Points
- 113
Current status:
We have about 50% accounts restored on x12
All accounts but a handful that we had issues with should now be working on x14.
Please start a new support thread if you are on x14 and your site is still not loading.
We have about 50% accounts restored on x12
All accounts but a handful that we had issues with should now be working on x14.
Please start a new support thread if you are on x14 and your site is still not loading.
- Messages
- 11,750
- Reaction score
- 581
- Points
- 113
All services should now be restored. We had about a dozen accounts that had corruption issues. If you are on x12 or x14 and do not see your account working please do make a post in https://community.x10hosting.com/community/forums/free-hosting.9/
- Status