Global Update(MySQL, Issues, Etc.)
04:15 PM on January 5th
Update, Please Read Entire Post
Summary(For those with short attention spans).
1.) MySQL is messed up, we are moving it back to local.
2.) We are replacing our secondary 250GB drives with SCSI drives solely for MySQL.
3.) We are taking this time to format the servers and set them up efficiently and correctly.
4.) I ordered another server solely for backups for this occasion.
5.) I will update the bottom of this post with the progress of our actions.
/End Summary
Our original goal was to move all databases from the free server to a single remote server to handle all MySQL processes. MySQL is the #1 resource hog on each server. By doing this we would eliminate the number one cause of lockups and stability issues. The implementation looked easy, the cost was only $250\mo, easy decision. Wrong. We started having issues from the start. We knocked them out slowly one by one. One issue took a whole week, going back between our Datacenter and cPanel. We were finally were told that it's a cPanel bug and may be fixed in the next 3 months. Obviously not a good answer, we investigated and came up with a work around.
This brings us to our very last and most recent issue, PHPMyAdmin. Since we are a free hosting company and on occasion stress software to it's breaking point with our highly unusual usage, it makes it very hard for us to get support from the developers of software. Specifically MySQL and PHPMyAdmin. We were told many different things, tried A LOT of different things, and in the end decided to open a paid support ticket with our Data Center to see if they could figure it out. This brings us to the past two days. The data center had a collapse of communication between their technicians and downgraded MySQL on our database server to 4.1(We were at 5.0). This caused huge issues, on top of that the tech did not make a backup of the databases before doing so.
I exchanged some words with one of the managers yesterday who assured me he would take care of it and everything would be fixed. At this point I just wanted the databases back online, the PHPMyAdmin issue could wait. Well, another catastrophic miscommunication occurred. They made some stupid decisions and delayed the process of fixing things for hours. I got a call from Bryon last night while I was out to dinner for a friend's birthday(I have a life sometimes) informing me of the ticket updates. I then spent 45 minutes on the phone outside going over everything with a tech. They finally got the database server back up this morning. There are of course issues with some databases because of the downgrading and upgrading. We have about 15 backups from the past week that are floating around, we will get everyone databases back up and running when we resolve this.
Where we are at now? Without a definitive answer for PHPMyAdmin in sight I have made the decision to move MySQL back locally, but replace the secondary 250GB HDDs we have with 73GB 15K SCSI drives at an extra cost and have MySQL run solely off these drives easing the burden on the I\O of the main drives. I have also decided that it would be best to format each server one by one and start fresh with all updated software. We've had so many updates, patches, fixes, etc(Apache 1.3->Apache2.2;ASP.NET Support, MySQL 4.1->5.0) over the past few months that these servers are not running anywhere close to efficient. By doing this each server will be 100% optimized and should be good to go for a long time to come.
Implementation. I have just purchased a server solely for backups. We are going to do this one server at a time by migrating all accounts to the backup server, formatting the current server, optimizing and securing then moving the accounts back. I'm hoping we can finish all this in the next two days for all servers. This is obviously not what I really want to do as it is going to cause downtime. But I feel that the overall improvement that we will get out of this will be worth it
So, please be patient, I am working as hard as I can on this. For those of you who have been with us a while, you know we will work as hard as possible until we solve these issues. We have a great team of staff members, and lots of drinks with caffeine, we will be around to answer questions until we are done. I will also try to keep everyone updated on our progress as much as possible. Thank you for sticking with us, we'll be back to 100% soon.
-Corey
You will not lose your data\accounts, this does not affect paid members, please excuse typos.
Progress:
Waiting for backup server to come online, ETA 2 hours.
Backup server is online, recompiling Apache, setting up MySQL, and updating cPanel to prepare for the account moves.
Server setup complete, copying accounts from Absolut over to backup server.
(716 Accounts Copied out of 4,566 as of 1:30AM EST(-5GMT))
(905 Accounts Copied out of 4,566 as of 2:15AM EST(-5GMT))
(1830 Accounts Copied out of 4,566 as of 12PM EST(-5GMT))
(2500 Accounts Copied out of 4,566 as of 5:30PM EST(-5GMT))
(3484 Accounts Copied out of 4,566 as of 7:20AM EST(-5GMT))(1-7-07)
F.A.Q.
Q: Will this move everything back to localhost?
A: Yes, all config files and such will have to be changed back to localhost
Q: Will I lose anything?
A: You should not lose anything, we have multiple backups of everything. Worst case you might lose a few hours of updates from MySQL while your account was being copied between servers.
Q: How will I know when it is done?
A: I will update this post as we complete stages.
Q: Does this affect paid services?
A: No, these move and updates are only going on for free servers.
The servers will go down between 9 and 11 CST tonight, 1/6/07 to add the new 15K hard drives.
