- Messages
- 34,551
- Reaction score
- 205
- Points
- 63
This thread is now obsolete, please read http://x10hosting.com/forums/news-announcements/130105-overall-update.html
Hi All,
This will be a semi-long read as I'm going to explain the issues, our thought process on it and where we're at now and what we're going to do.
As far as I know we're the largest free host that offers cPanel... with a very good reason, cPanel does not scale well past a few thousand accounts. When you get 1,000 a day this becomes an issue very quickly. In order to keep up with the growth we decided to move cPanel into a semi-clustered setup using a large very expensive server as a central place to manage cPanel while offloading the web services, MySQL, and Email to other servers. This would allow for easier management of accounts and also allow redundancy for serving pages since this is also another bottleneck, especially when a few abusive users cause an issue.
We started off this idea by setting up a cluster of virtual private servers and interconnecting them via private vlan. From here we simulated a large hosting setup just the way we pictured our final setup would be, instead using the virtual servers instead of real servers. After A LOT of testing and configuration we came to a solution we believed would work.
From here we started implementation by moving Cossacks and Lotus to the new setup and also putting new signups on the same server(Fris). As always there were issues with moving the accounts, for whatever reason maybe 1 out of every 20 accounts does not completely restore properly from the cPanel transfers, whether it be passwords being corrupted, files missing, etc... this took the better part of a month to do. At the end of the transfer process we were at 23,000 accounts on Fris.
This is where things started to go downhill. Misc reports of people receiving 503 errors when trying to login to cPanel started to come in. Shortly after data inconsistencies between our backend started to show up, saying accounts were suspended when they were not, saying addon domains were added when they were were not, etc... after doing some additional testing we found that cPanel was denying connections to the backend and also to users. We spent some time searching for any type of limitation in cPanel's forums and docs, after coming up with nothing we decided to open a ticket with cPanel to see if there was a value somewhere that we were missing.
Now, in past threads I have stated some of what we were told by cPanel. At first we were told there was a 200 simultaneous connection limit, that was later corrected to 160. Now for some reason some people that have no experience in web hosting and do not understand what they're talking about thought this had to do with the maxclients limit with Apache. Apache and cpsrvd are completely separate services and hence have completely separate settings that control them. The connection limit that controls cpsrvd(cPanel) is closed source and not editable by us. I have taken a screen shot of my ticket with cPanel regarding this issue instead of typing it all out. I have removed some of my replies as it contained server information that I do not want public.
http://img189.imageshack.us/img189/4603/cpticket.png
The final reply to this ticket was on July 9th and was:
To sum it up, we hit an undocumented connection limit in cPanel and while they made a 'feature request' to fix it they have no ETA of when that would be. Unfortunately that leaves us in the middle of a bunch of transfers and hardware that is no longer viable to use. The past few days we have been devising plans and writing scripts to come up with a solution to this mess. As of right now we have a ton of data inconsistencies and half messed up accounts due to all the transfers and problems caused by cPanel denying requests. We tried one other solution over the past 2 days with Chopin using NFS and a different approach to clustering... however the only way we could get it to work was to turn off some major features, and without proper testing we do not feel comfortable moving forward with it.
Moving accounts again is not an option, the last move took roughly a month to do and we're still fixing issues caused from it. In order to get things back to running properly in the shortest amount of time we have decided the best course of action will be to wipe the free servers, fit them with some new hardware then restore accounts back to them. This will allow us to clear all old accounts, and also cleanup all of our records which have data going back 6 years. Unfortunately this will cause some downtime, however we believe the downtime caused by this will be far better than consistent problems over a month or two while moving accounts between servers.
Please remember that a lot of work and money has gone into this failed setup and also will be going into fixing it. We have thousands upon thousands of accounts and we're not talking about a single server setup where it's easy to move stuff around and manage. There are a lot of servers involved and a lot of work has to go into making sure they're secured and optimized properly for free hosting. Everyone here has been and will continue to work hard to come up with a stable solution that supports cPanel.
This is what will happen as of right now:
We're in the process of packaging up every free hosting account to one of our backup servers. After our last transfers we're not going to fully rely on the cPanel packages this time... after the packages are complete we will also be taking a full copy of the /home dir on each server, this will make sure no files are missed in the package and also give us a more up to date copy of files as the packages will quickly be out of date due to how long it takes to package all accounts. I see no way any files should be lost during this but it is ALWAYS good to keep a local backup of your files. Once packages and copies have been completed we will be taking the server down, adding additional hardware, formatting it and setting it back up. Right now I do not have a good ETA on any of this. Chopin will be the first server to go and a news update will be posted with live updates for each server as the time comes.
After we do the first server we'll be able to give a better estimate of how long each will take. Chopin's packages should complete by tomorrow morning, an rsync of the whole /home may take a full day to do so this may push the start of the restore until Thursday.
Once we start taking servers down I will be closing the support forums and tickets... most of the issues people are facing will be resolved when the accounts are restored.
I'm going to tentatively leave this thread open for questions as this is a huge update and affects all free users. There will absolutely be no support provided here and your post will be deleted if you post a support question. I've posted some Q&As below and will add to them as people ask questions. Please check this post as the bottom will be updated with answers as they come in.
Q. Will I lose any files?
A. You shouldn't but it's always good to keep a local backup. Remember that MySQL is stored on a separate server and backed up daily, the MySQL server will be unaffected by this.
Q. What is going to change with the server setups?
A. We are increasing the speed of the hard drives and will be doing a load balanced web solution still. We're hoping this will solve the two major bottlenecks that started all this in the first place.
Q. How long do you think it will take?
A. As of right now we're not going to give estimates. After the server comes down we estimate it will take a few hours for our DC to replace the hardware and get it formatted. Generally it takes us 3 to 4 hours to setup a free server properly. Once done we will start restoring two accounts at a time, one from A-Z and another from Z-A. Also, we will only be restoring unsuspended accounts, people that are suspended will have to wait and request a restore once unsuspended accounts are restored.
Q. Did you proofread this post?
A. Not really, I have work to get back to. I'm sure some of the staff\CAs will be happy to point out my spelling * grammar mistakes, expect some edits.
Q. What will happen to new signups? *Updated 7/16*
A. Starting tonight new signups will be queued and then created once the new servers are online. This will most likely be 24-48 hours from now.
Update threads below, as we start additional servers they will be added
Chopin: http://x10hosting.com/forums/news-announcements/129434-chopin-update-thread.html
Fris: http://x10hosting.com/forums/news-announcements/129435-fris-update-thread.html
Boru: http://x10hosting.com/forums/news-announcements/129437-boru-update-thread.html
Starka: http://x10hosting.com/forums/news-announcements/129498-starka-update-thread.html
Stoli: http://x10hosting.com/forums/news-announcements/129499-stoli-update-thread.html
Support will be closed until all restores have been started for at least 24 hours. Posting support requests in other parts of the forum will get you a ban.
This thread has been closed due to the inability of people to read before posting.
Hi All,
This will be a semi-long read as I'm going to explain the issues, our thought process on it and where we're at now and what we're going to do.
As far as I know we're the largest free host that offers cPanel... with a very good reason, cPanel does not scale well past a few thousand accounts. When you get 1,000 a day this becomes an issue very quickly. In order to keep up with the growth we decided to move cPanel into a semi-clustered setup using a large very expensive server as a central place to manage cPanel while offloading the web services, MySQL, and Email to other servers. This would allow for easier management of accounts and also allow redundancy for serving pages since this is also another bottleneck, especially when a few abusive users cause an issue.
We started off this idea by setting up a cluster of virtual private servers and interconnecting them via private vlan. From here we simulated a large hosting setup just the way we pictured our final setup would be, instead using the virtual servers instead of real servers. After A LOT of testing and configuration we came to a solution we believed would work.
From here we started implementation by moving Cossacks and Lotus to the new setup and also putting new signups on the same server(Fris). As always there were issues with moving the accounts, for whatever reason maybe 1 out of every 20 accounts does not completely restore properly from the cPanel transfers, whether it be passwords being corrupted, files missing, etc... this took the better part of a month to do. At the end of the transfer process we were at 23,000 accounts on Fris.
This is where things started to go downhill. Misc reports of people receiving 503 errors when trying to login to cPanel started to come in. Shortly after data inconsistencies between our backend started to show up, saying accounts were suspended when they were not, saying addon domains were added when they were were not, etc... after doing some additional testing we found that cPanel was denying connections to the backend and also to users. We spent some time searching for any type of limitation in cPanel's forums and docs, after coming up with nothing we decided to open a ticket with cPanel to see if there was a value somewhere that we were missing.
Now, in past threads I have stated some of what we were told by cPanel. At first we were told there was a 200 simultaneous connection limit, that was later corrected to 160. Now for some reason some people that have no experience in web hosting and do not understand what they're talking about thought this had to do with the maxclients limit with Apache. Apache and cpsrvd are completely separate services and hence have completely separate settings that control them. The connection limit that controls cpsrvd(cPanel) is closed source and not editable by us. I have taken a screen shot of my ticket with cPanel regarding this issue instead of typing it all out. I have removed some of my replies as it contained server information that I do not want public.
http://img189.imageshack.us/img189/4603/cpticket.png
The final reply to this ticket was on July 9th and was:
To sum it up, we hit an undocumented connection limit in cPanel and while they made a 'feature request' to fix it they have no ETA of when that would be. Unfortunately that leaves us in the middle of a bunch of transfers and hardware that is no longer viable to use. The past few days we have been devising plans and writing scripts to come up with a solution to this mess. As of right now we have a ton of data inconsistencies and half messed up accounts due to all the transfers and problems caused by cPanel denying requests. We tried one other solution over the past 2 days with Chopin using NFS and a different approach to clustering... however the only way we could get it to work was to turn off some major features, and without proper testing we do not feel comfortable moving forward with it.
Moving accounts again is not an option, the last move took roughly a month to do and we're still fixing issues caused from it. In order to get things back to running properly in the shortest amount of time we have decided the best course of action will be to wipe the free servers, fit them with some new hardware then restore accounts back to them. This will allow us to clear all old accounts, and also cleanup all of our records which have data going back 6 years. Unfortunately this will cause some downtime, however we believe the downtime caused by this will be far better than consistent problems over a month or two while moving accounts between servers.
Please remember that a lot of work and money has gone into this failed setup and also will be going into fixing it. We have thousands upon thousands of accounts and we're not talking about a single server setup where it's easy to move stuff around and manage. There are a lot of servers involved and a lot of work has to go into making sure they're secured and optimized properly for free hosting. Everyone here has been and will continue to work hard to come up with a stable solution that supports cPanel.
This is what will happen as of right now:
We're in the process of packaging up every free hosting account to one of our backup servers. After our last transfers we're not going to fully rely on the cPanel packages this time... after the packages are complete we will also be taking a full copy of the /home dir on each server, this will make sure no files are missed in the package and also give us a more up to date copy of files as the packages will quickly be out of date due to how long it takes to package all accounts. I see no way any files should be lost during this but it is ALWAYS good to keep a local backup of your files. Once packages and copies have been completed we will be taking the server down, adding additional hardware, formatting it and setting it back up. Right now I do not have a good ETA on any of this. Chopin will be the first server to go and a news update will be posted with live updates for each server as the time comes.
After we do the first server we'll be able to give a better estimate of how long each will take. Chopin's packages should complete by tomorrow morning, an rsync of the whole /home may take a full day to do so this may push the start of the restore until Thursday.
Once we start taking servers down I will be closing the support forums and tickets... most of the issues people are facing will be resolved when the accounts are restored.
I'm going to tentatively leave this thread open for questions as this is a huge update and affects all free users. There will absolutely be no support provided here and your post will be deleted if you post a support question. I've posted some Q&As below and will add to them as people ask questions. Please check this post as the bottom will be updated with answers as they come in.
Q. Will I lose any files?
A. You shouldn't but it's always good to keep a local backup. Remember that MySQL is stored on a separate server and backed up daily, the MySQL server will be unaffected by this.
Q. What is going to change with the server setups?
A. We are increasing the speed of the hard drives and will be doing a load balanced web solution still. We're hoping this will solve the two major bottlenecks that started all this in the first place.
Q. How long do you think it will take?
A. As of right now we're not going to give estimates. After the server comes down we estimate it will take a few hours for our DC to replace the hardware and get it formatted. Generally it takes us 3 to 4 hours to setup a free server properly. Once done we will start restoring two accounts at a time, one from A-Z and another from Z-A. Also, we will only be restoring unsuspended accounts, people that are suspended will have to wait and request a restore once unsuspended accounts are restored.
Q. Did you proofread this post?
A. Not really, I have work to get back to. I'm sure some of the staff\CAs will be happy to point out my spelling * grammar mistakes, expect some edits.
Q. What will happen to new signups? *Updated 7/16*
A. Starting tonight new signups will be queued and then created once the new servers are online. This will most likely be 24-48 hours from now.
Update threads below, as we start additional servers they will be added
Chopin: http://x10hosting.com/forums/news-announcements/129434-chopin-update-thread.html
Fris: http://x10hosting.com/forums/news-announcements/129435-fris-update-thread.html
Boru: http://x10hosting.com/forums/news-announcements/129437-boru-update-thread.html
Starka: http://x10hosting.com/forums/news-announcements/129498-starka-update-thread.html
Stoli: http://x10hosting.com/forums/news-announcements/129499-stoli-update-thread.html
Support will be closed until all restores have been started for at least 24 hours. Posting support requests in other parts of the forum will get you a ban.
This thread has been closed due to the inability of people to read before posting.
Last edited: