As I’m sure you have noticed, the health of February, our dear live system, hasn’t been up to specs the last weeks. The reason is a common phenomenon in MMOG games called “Hotspots”. With us these are centered around our Collaboration services.
For short, it’s when you have centralized points where you have a large percentage of the population in the same place. Our mistake was to have to few “Highway” systems, which wasn’t really apparent when they were first introduced.
The reason this affects us now is that the core of these hotspots are now utilizing a full node (a full cpu) when its peaking. This affects transaction time with the SQL and it also affects it’s sibling nodes if it gets too much out of timesynch. We can’t solve this with “throwing more hardware at it” since we would need a nice shiny 3.02GHz Intel XEON CPU for each of the hotspots. But as a pleasant sidenote, we have already planned hardware upgrades to the cluster. Never hurts to have more hardware, does it 🙂