Some Server Issues \ Updates

[Corey]

Hi All,

We've had some issues with a few servers this past week, after conferring with cPanel we believe the issues to be OS related and once again due to our unique free hosting setup.

For the more technical people out there, we believe that using ext4 the way we are with the amount of users is causing cPanel to crash and burn and take the server with it, it is quota related. We're currently using Cent 5.6 which has just got proper support of quotas on ext4 recently.

There is no way to downgrade a system from Ext4 (that I'm aware of). What we will be doing is creating a new volume on the SAN and copying everything from the current one that is ext4 to the new one which will be ext3.

We will be doing Stoli first... there should not be any downtime involved with this and we have stopped touching quotas all together which should keep the server online and working.

I'll make an update as we make progress.

 

[Corey]

Stoli is about 70% done the first pass. It will probably finish sometime late tonight. Once done we will do a second pass which will be much, much faster. More updates to follow.

 

[Corey]

Stoli is doing its second pass, I have already swapped out the partitions.

If you are missing files they will re-appear in the next 2-3 hours. You may also inadvertently see files that you have deleted re-appear.

I'll post another update later on today as more work is performed.

 

[Corey]

Stoli completed fine.

We're working on Chopin right now... having some issues and will update in a bit.

 

[Corey]

I have finished testing on Chopin. Sorry for the unexpected reboots, the server did not respond like I expected it to.

I'm going to proceed with switching to ext3 as we did on Stoli. There should not be any further downtime or reboots needed besides the few minutes to swap the volumes, I will update when we get there though.

 

[Corey]

I have also started Boru due to it being afflicted by the same bug. Rsyncs are off and going, they will probably take ~24 hours or so to finish. We will do a 2nd pass once done, then swap the volumes. No further downtime is expected. We will be leaving Starka as-is for now, it does not seem to be affected by the bug in question.

For those curious, this is the bug we have ran into that has caused some downtime over the past week for free hosting: https://bugzilla.redhat.com/show_bug.cgi?id=696545#c2

 

[Corey]

Just an update... Boru, and Stoli are both swapped back to ext3 with quotas updated. Chopin has been swapped and quotas are updating. Starka is indeed affected by this bug so we will be swapping it sometime this week. Downtime is not expected besides a brief one minute interruption when we swap volumes. Also some files may re-appear that have been deleted due to the volumes being out of sync.