Important Server issues all servers.

Discussion in 'Free Hosting' started by Eric S, Sep 24, 2021.

Thread Status:
Not open for further replies.
  1. Anna

    Anna I am just me Staff Member

    Messages:
    11,050
    Likes Received:
    512
    Trophy Points:
    113
    You won't be able to until your server is completed.
     
    • Friendly Friendly x 2
  2. Eric S

    Eric S Administrator

    Messages:
    178
    Likes Received:
    13
    Trophy Points:
    18
    If your service is on x12 or x14 you will be experiencing these issues until everything is fully resolved.
     
    • Like Like x 2
    • Useful Useful x 1
  3. Anna

    Anna I am just me Staff Member

    Messages:
    11,050
    Likes Received:
    512
    Trophy Points:
    113
    There's really not much new in regards to the progress, but I did get a bit information about the cause of it all.

    Due to a bug in the cloud software we use we suffered a failure to our underlying storage system. This caused data corruption on the partition containing the user accounts on some of the servers (x11 and x15 got away unharmed).

    The restore process is tedious, there's a lot of data involved, but we do have full backups.
     
    • Useful Useful x 2
  4. Anna

    Anna I am just me Staff Member

    Messages:
    11,050
    Likes Received:
    512
    Trophy Points:
    113
    There was a discovery of a DNS issue that mainly affects new accounts (but also anyone changing/adding domains in their account).

    This issue basically leads to the domains never being made public and are therefor not able to resolve to the correct IP (think of DNS as a huge phonebook, where domain is the listed name and the IP is the phone number, if this fails the number is never known to the one searching, be it a browser or ftp client, so they can not connect to it).

    We are working on getting this resolved as quick as possible alongside with restores of servers x12 and x14.

    Due to this DNS issue we have also halted new account creations for the time being.
     
  5. Anna

    Anna I am just me Staff Member

    Messages:
    11,050
    Likes Received:
    512
    Trophy Points:
    113
    The DNS issues should be sorted, anyone that created an account the last few days and have a domain that gives an error about not resolving should begin to resolve properly fairly soon, if it is not already doing so.
     
  6. Anna

    Anna I am just me Staff Member

    Messages:
    11,050
    Likes Received:
    512
    Trophy Points:
    113
    Official statement of what's currently going on and why it is taking such long time to get everyone back up:

    Unfortunately, we suffered a failure of our underlying storage system due to a bug in the cloud software we use. This has caused data on the /home partition to become corrupted. We keep multiple daily backups using Acronis of all our systems and immediately implemented our disaster recovery plan to restore the lost data. We have run into another bug that is preventing us from restoring the full set of data all at once, which is the normal and fastest method of restoration. At this time we're stuck doing manual restores of batches of files, which is much much slower. We have this issue escalated to the highest levels at Acronis and hope to have a resolution soon. We have full backups of all data, it's just a matter of getting it back on the servers in place, which we're working on as fast as we can to get done.

    This has never happened in over 10 years of being in business and we certainly expect it to not happen in the future.
     
    • Like Like x 1
    • Informative Informative x 1
    • Useful Useful x 1
  7. Anna

    Anna I am just me Staff Member

    Messages:
    11,050
    Likes Received:
    512
    Trophy Points:
    113
    Current status:

    We have about 50% accounts restored on x12

    All accounts but a handful that we had issues with should now be working on x14.
    Please start a new support thread if you are on x14 and your site is still not loading.
     
    • Informative Informative x 1
    • Useful Useful x 1
  8. Anna

    Anna I am just me Staff Member

    Messages:
    11,050
    Likes Received:
    512
    Trophy Points:
    113
    • Like Like x 1
Thread Status:
Not open for further replies.

Share This Page