Important Server issues all servers.

Discussion in 'News and Announcements' started by Eric S, Sep 24, 2021.

Thread Status:
Not open for further replies.
  1. Eric S

    Eric S Administrator

    Messages:
    95
    Likes Received:
    9
    Trophy Points:
    8
    ALL SERVICES SHOULD BE RESTORED. IF YOU ARE STILL HAVING ISSUES WITH YOUR ACCOUNT PLEASE MAKE A POST AT https://community.x10hosting.com/community/forums/free-hosting.9/

    WE ARE CURRENTLY EXPERIENCING SERVER ISSUES WITH X12 AND X14.

    YOU CAN TELL WHICH SERVER YOU ARE ON IN YOUR ACCOUNT DASHBOARD IT WILL STATE USERNAME @ SERVER.

    Once an update is available we will update this post.

    New signups are disabled at this time.

    You may see missing files, missing emails and more but please do know this is normal as we are working to restore all services.
     
    Last edited: Oct 8, 2021
    • Informative Informative x 1
    • Useful Useful x 1
  2. Anna

    Anna I am just me Staff Member

    Messages:
    10,950
    Likes Received:
    500
    Trophy Points:
    113
    Had a word with the lead tech, unfortunately not really much new to tell.

    They are working as fast as possible to get everyone that is affected back online.

    Rest assured that your data is secure and backed up properly.
     
    • Like Like x 3
    • Informative Informative x 1
  3. Anna

    Anna I am just me Staff Member

    Messages:
    10,950
    Likes Received:
    500
    Trophy Points:
    113
    x11, x13 and x15 should work properly now.

    x12 and x14 are still underway and will show irregularities such as missing files during the process.
     
    • Like Like x 2
  4. Anna

    Anna I am just me Staff Member

    Messages:
    10,950
    Likes Received:
    500
    Trophy Points:
    113
    There's really not much new in regards to the progress, but I did get a bit information about the cause of it all.

    Due to a bug in the cloud software we use we suffered a failure to our underlying storage system. This caused data corruption on the partition containing the user accounts on some of the servers (x11 and x15 got away unharmed).

    The restore process is tedious, there's a lot of data involved, but we do have full backups.
     
    • Like Like x 1
    • Informative Informative x 1
  5. Anna

    Anna I am just me Staff Member

    Messages:
    10,950
    Likes Received:
    500
    Trophy Points:
    113
    There was a discovery of a DNS issue that mainly affects new accounts (but also anyone changing/adding domains in their account).

    This issue basically leads to the domains never being made public and are therefor not able to resolve to the correct IP (think of DNS as a huge phonebook, where domain is the listed name and the IP is the phone number, if this fails the number is never known to the one searching, be it a browser or ftp client, so they can not connect to it).

    We are working on getting this resolved as quick as possible alongside with restores of servers x12 and x14.

    Due to this DNS issue we have also halted new account creations for the time being.
     
    • Like Like x 1
    • Informative Informative x 1
  6. Anna

    Anna I am just me Staff Member

    Messages:
    10,950
    Likes Received:
    500
    Trophy Points:
    113
    The DNS issues should be sorted, anyone that created an account the last few days and have a domain that gives an error about not resolving should begin to resolve properly fairly soon, if it is not already doing so.
     
    • Like Like x 2
  7. Anna

    Anna I am just me Staff Member

    Messages:
    10,950
    Likes Received:
    500
    Trophy Points:
    113
    Official statement of what's currently going on and why it is taking such long time to get everyone back up:

    Unfortunately, we suffered a failure of our underlying storage system due to a bug in the cloud software we use. This has caused data on the /home partition to become corrupted. We keep multiple daily backups using Acronis of all our systems and immediately implemented our disaster recovery plan to restore the lost data. We have run into another bug that is preventing us from restoring the full set of data all at once, which is the normal and fastest method of restoration. At this time we're stuck doing manual restores of batches of files, which is much much slower. We have this issue escalated to the highest levels at Acronis and hope to have a resolution soon. We have full backups of all data, it's just a matter of getting it back on the servers in place, which we're working on as fast as we can to get done.

    This has never happened in over 10 years of being in business and we certainly expect it to not happen in the future.
     
    • Informative Informative x 2
    • Like Like x 1
  8. Eric S

    Eric S Administrator

    Messages:
    95
    Likes Received:
    9
    Trophy Points:
    8
  9. Anna

    Anna I am just me Staff Member

    Messages:
    10,950
    Likes Received:
    500
    Trophy Points:
    113
Thread Status:
Not open for further replies.

Share This Page