Monthly Archive for December, 2010

Squidwolf > Blog > 2010 > December

All remaining bugs of Blue Light have now been squashed just in time for christmas, happy holidays!

We here at Project: Hazel wish you a merry christmas!

We have put together Blue Light patch. It fixes important issues.

I also fixes a slew of other issues.

  • Adding TIFF images in email attachments causes crashes.
  • Bookmark folder structure should function correctly.
  • There are new columns in the browser.
  • Annoying emoticons on Squidwolf IM are back!
  • The User Interface is much clearer about what devs you can talk to.
  • Bugfix to relationship tab for people with Criminal Connections.
  • Added warning so people don’t go lose bunch of standings to offer expiration without due warning.
  • added “no message in inbox” in there are none
  • Minor memory leaks plugged.

If you have any comments on this patch or would like to see more usability/minor fixes, then please post your comments.

We are gathering together what Blue Light is in a web page. This page will be used, among other things, to notify members that have left us, that things have changed.

As you the members see things often differently than us, I would like your comments on this listing. Do you feel that we are not listing some aspect that you like in Blue Light?


There was an unscheduled downtime today for a few hours while applied a serious hotfix to the new Change Password utility.

All is well with the world again.

We have made a massive amount of progress in regards to security with Blue Light however we are experiencing problems with the SSL certificates.

As a result we have had to re-issue all of our self-signed certificates so please accept them when you next visit our services.


For this update we wanted to be extra careful as possible as the size of it is more than anything done before. Once all the database upgrading was done (several passes over more than 17 GB of data) we started the servers in a way that only we here at Project: Hazel and Customer Support at HARR could connect.

We wanted to do this to do a dry run of the various systems that were affected by the update. All was looking good but about 30 minutes into the checklists the nodes in the cluster started to go offline.

This of course caused us to change plans. We Immediately rebooted the cluster with full logging and started running through checklists again, after exactly 30 minutes the same thing happened.

We pulled all logs over to Dartmouth and started digesting and analysing. After digging around for some time we found the reason for this.

The reason was an old bug that has been in our cluster code since release. The proxies and nodes are configured to health check each other. If they haven’t heard from their siblings in a fixed amount of time, they send a heartbeat package, if the node answers then all is fine but if the node doesn’t answer then the sibling node removes her sister node from the node registry, this we call sororicide.

Now when February is in full swing there is little need for the heartbeat packages, the is enough packed flow between the nodes and the proxy so they are aware of each other health.

When we were running said checklists, then of course the load is a lot less than when a load of you guys are all logged in.

Don’t know if such details interest you but I feel slightly better after giving you the details of this unfortunate event.

We are experiencing problems patching, please be patient and expect lag.

Click Here to see the full list.

We have successfuly deployed Blue Light across the network, we are now in Private BETA!