A tale of three-phased hardware upgrade

14th May 2011 by BlueShadow

Squidwolf > Blog > Dev Blog > A tale of three-phased hardware upgrade

Now the second part of our three-phased hardware upgrade is complete. This phase included the upgrade of all electrical mains and fuses to 64 Amps from 16 Amps. This was need as the additional disks, YEP!!! nodes and the new Squidwolf SQL server, drain much more power than previous setup.

The part that we completed today was to move the transaction log of the Squidwolf SQL cluster over to our FiberChannel array, from an SCSI array that it had been using. This was need as we learned during the first phase that our SCSI array and an active-standby setup of Mac OS X 1.06 Snow Leopard Server cause it to enter re-synch mode each time there is a considerable change in the cluster.

Now this was all completed at around 14:00 today, but when we start up February, performance was way off. We of course suspected that the total number of spindles servicing the transaction log disk-volume was insufficient, so we upgraded it to 6 disks servicing the log drive. That didn’t change any thing and the problem was eventually chased down to a deadlock fault in the procedure that handles the loading of stations and automatic roll back of failed item exchanged sessions. As the hardware part was the biggest change we didn’t spot it straight away, I apologies for this.

Now all I/O and power upgrades are completed and we are ready to replace the SQL cluster and add the additional YEP!!! nodes next week, thus having doubled February to handle future growth.

Again I apologize for yet again having exceeded our downtime estimations.

Comments are closed.