Global Update(MySQL, Issues, Etc.)

Discussion in 'News and Announcements' started by Corey, Jan 5, 2008.

Thread Status:
Not open for further replies.
  1. Corey

    Corey I Break Things Staff Member

    Update, Please Read Entire Post

    Summary(For those with short attention spans).
    1.) MySQL is messed up, we are moving it back to local.
    2.) We are replacing our secondary 250GB drives with SCSI drives solely for MySQL.
    3.) We are taking this time to format the servers and set them up efficiently and correctly.
    4.) I ordered another server solely for backups for this occasion.
    5.) I will update the bottom of this post with the progress of our actions.
    /End Summary
    Our original goal was to move all databases from the free server to a single remote server to handle all MySQL processes. MySQL is the #1 resource hog on each server. By doing this we would eliminate the number one cause of lockups and stability issues. The implementation looked easy, the cost was only $250\mo, easy decision. Wrong. We started having issues from the start. We knocked them out slowly one by one. One issue took a whole week, going back between our Datacenter and cPanel. We were finally were told that it's a cPanel bug and may be fixed in the next 3 months. Obviously not a good answer, we investigated and came up with a work around.

    This brings us to our very last and most recent issue, PHPMyAdmin. Since we are a free hosting company and on occasion stress software to it's breaking point with our highly unusual usage, it makes it very hard for us to get support from the developers of software. Specifically MySQL and PHPMyAdmin. We were told many different things, tried A LOT of different things, and in the end decided to open a paid support ticket with our Data Center to see if they could figure it out. This brings us to the past two days. The data center had a collapse of communication between their technicians and downgraded MySQL on our database server to 4.1(We were at 5.0). This caused huge issues, on top of that the tech did not make a backup of the databases before doing so.

    I exchanged some words with one of the managers yesterday who assured me he would take care of it and everything would be fixed. At this point I just wanted the databases back online, the PHPMyAdmin issue could wait. Well, another catastrophic miscommunication occurred. They made some stupid decisions and delayed the process of fixing things for hours. I got a call from Bryon last night while I was out to dinner for a friend's birthday(I have a life sometimes) informing me of the ticket updates. I then spent 45 minutes on the phone outside going over everything with a tech. They finally got the database server back up this morning. There are of course issues with some databases because of the downgrading and upgrading. We have about 15 backups from the past week that are floating around, we will get everyone databases back up and running when we resolve this.

    Where we are at now? Without a definitive answer for PHPMyAdmin in sight I have made the decision to move MySQL back locally, but replace the secondary 250GB HDDs we have with 73GB 15K SCSI drives at an extra cost and have MySQL run solely off these drives easing the burden on the I\O of the main drives. I have also decided that it would be best to format each server one by one and start fresh with all updated software. We've had so many updates, patches, fixes, etc(Apache 1.3->Apache2.2;ASP.NET Support, MySQL 4.1->5.0) over the past few months that these servers are not running anywhere close to efficient. By doing this each server will be 100% optimized and should be good to go for a long time to come.

    Implementation. I have just purchased a server solely for backups. We are going to do this one server at a time by migrating all accounts to the backup server, formatting the current server, optimizing and securing then moving the accounts back. I'm hoping we can finish all this in the next two days for all servers. This is obviously not what I really want to do as it is going to cause downtime. But I feel that the overall improvement that we will get out of this will be worth it

    So, please be patient, I am working as hard as I can on this. For those of you who have been with us a while, you know we will work as hard as possible until we solve these issues. We have a great team of staff members, and lots of drinks with caffeine, we will be around to answer questions until we are done. I will also try to keep everyone updated on our progress as much as possible. Thank you for sticking with us, we'll be back to 100% soon.

    -Corey
    You will not lose your data\accounts, this does not affect paid members, please excuse typos.

    Progress:
    Waiting for backup server to come online, ETA 2 hours.
    Backup server is online, recompiling Apache, setting up MySQL, and updating cPanel to prepare for the account moves.
    Server setup complete, copying accounts from Absolut over to backup server.
    Final backup of whole hard drive before format
    Format and reinstall in progress.
    Copying Absolut backup accounts back to Absolut.
    Absolut back online.
    Stoli in the process of being backed up to the backup server.
    All accounts were successfully copied to the backup server.
    Accounts are being copied from the back up server back to Stoli.
    Stoli has completed the copy back. Please post all move issues in the server move forum.
    Starting Cossacks transfer to backup tonight 1/24/08
    Cossacks completely transferred to backup.
    Cossacks to be formatted in the next 2 hours.

    All server moves are complete!

    Any remaining issues from the ABSOLUT, STOLI OR COSSACKS transfers should be posted here:
    http://forums.x10hosting.com/server-move-issues/
    Read the sticky threads before posting!

    Upgrade Optimization Order:
    Absolut Done
    Stoli Done
    Cossacks Done

    F.A.Q.
    Q: Will this move everything back to localhost?
    A: Yes, all config files and such will have to be changed back to localhost

    Q: Will I lose anything?
    A: You should not lose anything, we have multiple backups of everything. Worst case you might lose a few hours of updates from MySQL while your account was being copied between servers.

    Q: How will I know when it is done?
    A: I will update this post as we complete stages.

    Q: Does this affect paid services?
    A: No, these move and updates are only going on for free servers.
    Last edited: Jan 31, 2008
  2. Corey

    Corey I Break Things Staff Member

    Do not post support requests here, I will not answer them. Legitimate questions are okay to ask.
  3. MicrotechXP

    MicrotechXP New Member

    Also, please do not post topics about this, we will tell you when this issue is fixed!
  4. Nikolasr

    Nikolasr New Member

    Good corey thanks for this
  5. Smith6612

    Smith6612 I ate all of the x10Pizza Community Support

    Thanks for the update Corey. I already backed up all of my data this morning when MySQL was up and running long enough for the download to finish, along with all of my data so I should be set if something were to happen with my site. Well, this is an unexpected bump in the road, but it should improve the service a lot once this is done.
  6. Momiji

    Momiji New Member

    Thank you very much Corey.

    You guys are trying your hardest and people can't seem to grasp that.
  7. animerth

    animerth New Member

    If your moving MySQL back on to the servers what will be the performance cost even with the optimization

    Thanks for all your work
    Last edited: Jan 5, 2008
  8. Smith6612

    Smith6612 I ate all of the x10Pizza Community Support

    There shouldn't be much of a performance issue with the servers. Despite SQL making the servers get busy, just as long as there is enough memory, bandwidth and CPU, which there usually is, this should be fine. The site when I first got it here on x10 before the SQL move had very good page generation times and were very consistent. On the current setup which is about to be trashed and moved back to what Corey said, page generation times were all over the place.
  9. online87

    online87 New Member

    Corey & the guys,

    #1 - Seriously, thank you VERY MUCH for the detailed and honest update alert. It's nice to be told bout stuff.

    #2 - I notice... From 250Gb HDD to a 75Gb HDD (Raptor by any chance?)... that's a pretty big space sacrifice! Although the good thing about SCSI is the ability to link a fairly large number of drives to one controller with no performance hit. Is that your eventual intention?

    #3 - I'd like to thank you guys for providing a great free service, but unfortunately it is getting to the point where sometimes I just think "For ****'s sake they're down again.... Jesus..." - as you have to admit you have had an unacceptable amount of down time (even for the average free service), BUT...

    #4 - It's really great how dedicated you are to providing the fastest, most up-to-date, 'cool' free hosting service out there. I've tried others and they lag months behind with updates, and don't really care about their free customers. Although it often pisses me off, it's nice to be a customer of such dedicated and professional personnel.

    Keep it up, I hope it all goes as smoothly as possible, and sorry if I'm cheeky but let's hope this really DOES solve your problems, not just for my sake but for yours too.
  10. coolv1994

    coolv1994 New Member

    Well good luck with that and thanks for the update it sound like a load of fun.
    Last edited: Jan 5, 2008
  11. Evan1

    Evan1 New Member

    You guys are really something else :)
  12. Brandon

    Brandon Former Senior Account Rep Community Support

    On your #2, All the servers currently have 2 250 gig hard drives, one is unused on the servers, we will be replacing that with the 73. Also I think there Cheetah's, that's what's in the SQL server atm.
  13. antidoto

    antidoto New Member

    Don't worry be happy lool.

    Thanks a lot for your free service and support, no other freehost that I know is so supporting. I am honest yes.

    Keep up.
  14. Archkronos

    Archkronos New Member

    Will you ever move MySQL off to a seperate server in the near future, or will you always keep it local?
  15. Corey

    Corey I Break Things Staff Member

    I think after this mess we will probably always keep it local. We'll just add new servers if it becomes an issue.

    -Corey
  16. tonecas

    tonecas New Member

    Thank you very much corey =) This is the best and you are making for it ;)
    go ahead ;)
  17. Momiji

    Momiji New Member

    Corey I think that is a good idea considering all the problems that you have had.
  18. skeets

    skeets New Member

    I also think keeping it local would be a good idea. I'd hate (and I'm sure the admins would also) to go through this all again six months from now if x10 were to try again.
  19. DizzyTechnology

    DizzyTechnology New Member

    You guys are beyond awesome. New Apache, new SQL? Sweet.

    Now, a legitimate question or two: In what order will servers be done? Will the server IP's change, and will the backup server be accessible? How long does it usually take for each server (not whiny, just reference)? And, will PHP be compiled with GD and mbstring this time?

    Damn... I guess that was more than a question or two.
    Last edited: Jan 5, 2008
  20. kamcraft

    kamcraft New Member

    since u are moving mysql back to local does that mean you are goign to use that server for another webserver?
Thread Status:
Not open for further replies.

Share This Page