Updates, Upgrades, Etc.

Status
Not open for further replies.

Corey

I Break Things
Staff member
Messages
34,551
Reaction score
204
Points
63
Hi,

For some reason some of our posts going to the status site (status.x10hosting.com) are not showing up so I'm making an additional post here as to what is going on, status, etc...

Because of our rapid growth over the past year and some problems we were facing with our provider we decided it was in our best interest to move data centers. This decision was not made lightly as we have quite a bit of servers and terabytes upon terabytes of data including hundreds of IPs that we would have to leave behind. x10Hosting is a lot larger than just free hosting now, in fact our free hosting servers make up the minority of the servers we have online.

The move to Chicago from Dallas is only one of the many things that are going on right now. As most of you may have noticed, a new account panel has been put online. With this account panel a whole new backend system to manage free hosting was also turned on, this system has been in development for over a year and is one of the things that really sets x10 apart from the rest putting most of the account control in the hands of the user and the parts that are not are mostly automated. There are also 3 new websites in the works and changes going on to our paid services along with our free hosting. Some additional changes will be announced by the end of April.

These past few months we've been testing different setups for free hosting to find the ultimate combination of server, software, and compatibility with our own systems. Because the software we use such as cPanel is not setup to handle the amount of accounts we put on a server there is never any way for us to test these things in an 'offline' environment. Unfortunately this means we had to test on live systems and we really had to do a lot of testing to make sure we got it right as making a mistake would cost us thousands of dollars, and free hosting is not all that profitable ;). Because of all the testing, stability of the free servers has not been the best these past few months. However we did feel it was necessary to get it out of the way so we could succeed going into the future.

So what are we currently doing? With roughly 90% of our paid services moved over to Chicago including our own website we have started the free hosting migrations. Unfortunately there is no easy way to move 50,000+ accounts. The second we start moving accounts the host server usually chokes and starts spitting out errors. Again, this is mostly due to the fact the software we use such as cPanel is not meant to handle this many accounts. It's either that or no free hosting with cPanel though ;). Luckily we rarely have to move data centers, and if for some reason we ever had to again I think we'll pay to ship the hard drives instead of migrating accounts. For this migration it is not currently an option available to us though. Everyone keeps asking for an estimate on when we will be done transferring accounts and when things will go back to normal. Well, this is very hard to do. When transferring an account there are so many variables involved that an account that takes 1 minute to move right now may take 5 tomorrow.

I'm going to break out in some basic math now so here is a new paragraph to keep you focused. I can tell you we are currently averaging between 6-10 accounts per minute. Let's use 50K accounts in our math to keep it simple.
High end estimate: 50,000 Accounts divided by 6 per minute will give us how many minutes it will take, this comes to 8,333 minutes or 138 hours or 5.7 days.
Low end estimate: 50,000 Accounts divided by 10 per minute will give us 5,000 minutes, or 83 hours, or 3.5 days.

Now the above is if no problems occur and everything runs smoothly through out the whole transfer process -- this never happens. We constantly have to deal with issues from the transfers including rebuilding configuration files, restarting transfers, and restarting services. I know everyone would love to have an ETA as to when we'll be done but I really can not give a firm number. I can tell you I HOPE to have transfers 100% completed by this time next week. I can tell you they will not be done before 3.5 days from now. I can tell you we are very confident we will be able to consistently obtain 98% uptime or better for free hosting once this move is completed.

All we ask is that you be patient a little bit longer, we have some amazing updates coming soon and I believe everyone will think it was worth the wait.

I'm going to tentatively leave this thread open for questions as a lot was explained above but I'm sure I didn't cover everything. I will not provide support in this thread and even though I'm sure a lot of people will either want to thank us or flame us please do not post it here. Anything that is not a question about this update will be deleted.
 

Corey

I Break Things
Staff member
Messages
34,551
Reaction score
204
Points
63
I cleaned the thread up so people can find information easier. I will answer some of the questions that were asked though.

Will Chicago be faster?
The servers that we have at Chicago are faster than the ones we had at Dallas. Everyone that has been moved has also reported up to a 40% increase in round trip time even from out of country.

Will the accounts be unavailable the whole time?
No, even though we are stressing the servers out with multiple transfers there will be periods of up and down time for most of the servers. We've been moving Stoli separately due to the overall size of the accounts. Stoli used to be our ad-enhanced server when that was popular so it is almost double the size of our other servers. To move it we did have to take it down but it should be back up soon.

Why do we stay with cPanel?
For the most part cPanel is excellent software for the end user, sometimes it is a nightmare for the admin side but all of our custom scripts and backend software are made to work with cPanel. As far as I know there is not really an alternative that is better and right now we do not have time do make something custom.

Why not image the servers then move them?
We are moving to completely different hardware and unfortunately this would not work. Also each account has entries in at least 20 different config files that are located in different places on the server so there is no way to move just an account's files with cPanel. Each account needs to be packaged up, transferred over, unpackaged while cPanel re-creates all the necessary entries for each one.

Why not put a message up for what is going on?
This actually is not something that was ever considered. If we do this again in the future we can look into the best way to do it. Right now everything is setup using that primary IP that the accounts are on so we would not be able to redirect it. DNS wise I can think of a few ways we could do it but somethings would have to be modified otherwise it would cause some major issues with the transfers.

Absolut: 69.175.121.74
Boru: 69.175.121.66
Chopin: 69.175.104.34
Cossacks: 69.175.121.58
Lotus: 69.175.120.10
Starka: 69.175.120.122
Stoli: 69.175.122.178

Paid service issues?
Some people think free hosting and paid hosting are the same or have the same issues. This is not true. Paid hosting does not go down during migrations and generally we can do a whole server within a few hours, 99% of the paid users will not notice there was even a move due to the forwarding we're able to do and individual attention each account is given.

Are accounts moved in Alphabetical order?
Yes and no. Each server has multiple transfers running. One from A to Z, one from Z to A and usually between 1 and 3 others starting at random points in the list.

How does upgrading to paid hosting work?
You can still upgrade to paid hosting during this time however it can take a day or two depending what is going on with the server you are coming from since we need to be able to access it to get an account copy over to the paid servers. Just sign-up for paid hosting through the main website then open a support ticket in the paid hosting client area with the details of the account you would like moved and upgraded.

Will I lose any mail?
If the server is not online to accept mail the sending mail server should retry again, some retry for 24 hours but most retry for up to 72 hours before calling a message completely undeliverable.

Will the name servers change?
No, they will not. They will still be ns1.x10hosting.com & ns2.x10hosting.com

Is there a way to get status on the transfers?
No, it's not that we're trying to keep people in the dark on the status but the second we say something is almost done people start assuming things and telling other people bad information. When everything is done and is working like it should we will announce it. You can see if your site has been moved by pinging it and comparing it to the IPs I posted earlier in this thread. When your site has been moved it does not mean everything is all better, the server you moved to is having tons of data copied to it which may cause issues for it.
 
Last edited:

allofus

New Member
Messages
183
Reaction score
2
Points
0
I have been moved to Dallas already. Both VPS and Corporate shared hosting and now that the servers have been configured to their Chicago setup max_packet / memorl limits, etc I absolutely have noticed a better response time.

Our VPS is certainly working @ a 20% reduced load. aka 20% faster under certain conditions.

I downloaded a 'Full backup' off of Skyy yesterday and it was streaming over the Atlantic to England (me) @ 1.1MiB's according to WinSCP. Impressive.

Corey I personally want to thank you for the time you have dedicated to our account in the past week or so. One-2-one support in real time instigated by you and seen through to the end is a genuinely decent thing to do and that is why x10 is the only host I will be using and the only host I recommend also.

I think we will all be better off when this 'period' is over with.

Keep it real and remember where you came from when you get to Dallas.
 

Corey

I Break Things
Staff member
Messages
34,551
Reaction score
204
Points
63
There will be an update tonight with the status on transfers.
 

Corey

I Break Things
Staff member
Messages
34,551
Reaction score
204
Points
63
Update:
Lotus - Done
Chopin - Done
Starka - Done

In order to finish next:
Cossacks
Stoli
Boru
Absolut

10,000 accounts total left to transfer. While Chopin, Lotus, and Starka are done and should run somewhat smoothly there is still a massive amount of work we have to do once the migrations are complete to make sure they do run properly.

The math I did in the first post was an estimate if nothing went wrong and there were no problems, which does not happen. We also have more than 50,000 accounts. I picked 50,000 because it was easier to do the math on. We expect to have the migrations and the additional work to bring the servers back to being stable by the end of this week.
 

Corey

I Break Things
Staff member
Messages
34,551
Reaction score
204
Points
63
Another Update:

Cossacks has finished, Stoli and Boru will hopefully be done by tomorrow. Absolut is still trucking along.
 

Corey

I Break Things
Staff member
Messages
34,551
Reaction score
204
Points
63
Stoli and Boru have finished. Absolut is the only one left.

Our goal is to have the free hosting servers back to a stable state by the end of the weekend. Looking good so far.
 
Status
Not open for further replies.
Top