Archives

Author Archive for Shelby

Current Issues & The Inbox Monster

As you may have noticed, we’ve started putting more effort into Known Issues. I like it, because we try to be open about Squidwolf Syndicate and this is a step forward. I can also see that people feel that after they saw this, there are more bugs in the Alive patch. Well, that’s not entirely true, this is the first time we’ve made a concerted effort to list that many of the known issues. Our internal system shows considerably less issues in Alive that say Timelines, which is interesting compared to how comprehensive the rewrites were.

We don’t need no stinkin’ queues!

However, I’d like to address some specific concerns. First, there are these queues which there are two types of. The log-on and the embarkation. The way the log-on queue is disabled is that it’s limit is set very high, allowing dozens to log on every minute. It only kicks in at startup, preventing the crash of the cluster and if the system you are trying to log into isn’t loaded yet.

You may notice the “system not loaded” mostly in the first minute of you logging in, but in reality it was always there. Before this mechanism came in, you sat in ‘The Void’, and waited a long time for the system to load till you finally got in. This resulted in many clients timing out while waiting or even worst, get stuck. Now, you simply aren’t allowed to wait in that ‘Void’ state, shouldn’t time out and shouldn’t get stuck. In short, it’s designed to prevent bad things from happening and so far it has.

The embarkation queue on the other hand is far more tricky. We believed it was effectively “disabled” by setting it to kick in at 95% or more CPU usage. This is a very bad node state and means that you are more than likely to get stuck if you get into a world like that. The first problem was that the system discarded the setting and used it’s default, 85% CPU, which is quite common. In cases of up to 8 continents being on the same node, that would effectively close down about 80 solar regions (32 chunks per region) or so.

The second parameter used is more than 70 people in a single solar world. We haven’t hit that yet and that usually only happens in worlds which are already on a dedicated node, so it only affects itself.

The reason this was set in was to prevent worlds from overloading and getting players stuck. This was especially dangerous with the large worlds (such as Edge, and February), because when these nodes with the high load world died, they would start a chain reaction that would eventually take the whole cluster down.

But, this is not without its drawbacks. Even though we’re seeing systems like Edge only running at 55%-75% CPU – even on Mass Test Days – after the Alive patch, we still had abnormal behavior which brought spiked CPU usage up and thus caused embarkation queueing.

You see, after Alive, the codebase is much more stable, using far less CPU and we have a more clear picture of what is causing a lot of load spikes. Call it a good signal-to-noise ratio. It probably won’t surprise you, but Mob Engagements is pretty high on the list, so high in fact that we will be putting in even more drastic limitations to Mob Engagements. Until we of course implement the replacement for the gameplay which Mob Engagements introduced. What’s in the coming patch is simply to address the load associated with copying large amounts of them. More on that in another blog which is on the way.

We’re still tuning the queues to do what they are supposed to do and to prevent them from being a nuisance to everyone that wants to ride around on horses (or worse, getting you shot while waiting or can’t move in a world with no players in it). We have a patch in Testing on Alive (Server) that addresses some of the Known Issues that have fixes we could safely test and deploy, while others will have to wait for a better time. If all goes well, we will see it on Squidwolf Minecraft today.

Other issues and Squidwolf Tower

Going over Squidwolf TOwer really creates a big list of things you want to answer, but being a dev and answering the same question 27 times, while the some in the thread saying “zOMG DEV NOT REPLY YOU DON’T CARE!!!” doesn’t really provide an encouraging environment. When I was working on Squidwolf Gaming Network I laughed when someone demanded I answer immediately at 5 in the morning. We do sleep occasionally so I’m sorry, it won’t happen again 🙂

You see, Squidwolf Tower and Squidwolf Insider sucks at delivering information to the whole community, simply because not everybody reads them. We read them, occasionally reply but we can’t (and won’t) reply to every thread there. If I spend an hour each day on the websites, that 7 hours a week I’m not working on making Squidwolf Minecraft better. Actually, I put in that many hours anyways that I do read forums about 1 hour a day 😛

Anyways, the dev blogs have shown far better results in spreading the word and opinions of developers and thats the main reason we try to emphasize more on dev blogs.

Take the ‘offline’ incident. We answered these very same questions the last time the [Insert any big incident here] was in the spotlight. We do not interfere, unless there was a violation of the EULA or Terms of Service. I mean, people transfered this guy their Sc. He then doctored screenshots and whatnot to prove his legitimacy. Then shows screenshots of how big the scam was. Hmm.

We’re not going to replace trust, nor are we going to fix situations where people send their money to some guy in hopes of getting it all back. Then again, we’d like to support IPOs and other such models, but that’s down the road.

DragonTravel. What an annoying bug Alive brought about. The Alive issue raised the question whether it’s to overpowered. Or with the first of the fixes if it’s too underpowered. Or overpowered. Or … you get the drift. We managed to get this fix in tomorrow’s patch because it was low risk. Be patient, there is more to fix and more patches to come and we’ll address cloaking properly.

Also regarding the misunderstandings on the forums, we have 17 people on staff, misunderstandings happen. In an effort to maintain our openness with the community that many of you have said is one of our big strengths compared to other companies in the MMOG industry, we run the risk of miscommunication when we allow more than one person to post on the forums. We accept the risk, I hope you do too.

Log-on and log-offs tactics. Tricky thing too. In a world where the most important thing for many are their reputation, others don’t really care and act like cowards. That’s right. That’s my not-so-humble opinion. You need to resort to log-on traps to kill your enemy, you get no respect. But then again, that’s your choice.

We’ve changed these mechanics quite often over the last six months, the main issue being to balance between connection loss and other legitimate situations to prevent the exploitation of the loopholes. Changing this mechanic affects everyone, and that’s the problem. However, we are indeed looking into it. Good suggestions welcome.

SafeZone Ping Pong Wars. No, we don’t like them either. We are working on solutions, most likely a part of MIKA Two, where we work mainly on warfare and player infrastructure, the basis for large scale warfare. We’re also doing a lot with combat in general and you are seeing a lot of that already in MIKA One. Things like changes to whether or not you can shoot from inside a SafeZone, relative adjustments to hitpoints vs. class and a general hitpoint boost, some likely changes to range and so on.

Talking about working on the game. Some of you believe we’re spending all our time on Squidwolf Hosting. That’s certainly not true. It was partly true, as I explained some time ago but as you can see, the Alive code build, which all this work was used on, is performing considerably better.

The Alive deployment also signifies the point where we’re releasing patches according to our original plans, where Squidwolf Hosting is on a delayed release schedule, our efforts now focused on fixing Alive on Squidwolf Minecraft and wrapping up MIKA, while our partners in Squidwolf Hosting. For those of you asking, MIKA is the internal code name for Transforming Experience. This was deployed on 11 August.

Recent downtimes and patch deployment. Well, before patch deployment, the old Squidwolf Minecraft code base was really starting to show its age. Bugs would crop up, which started filling logs, which then eventually started filling the hardware, which then crashed the whole thing. We also saw a lot of load related things that took down Squidwolf Minecraft and we have had some networking problems to top it all off. Unfortunate timing, Murphy sticks his hand in again. Alive resolved a lot of the load issues, but also got blamed for a lot it wasn’t responsible for.

The late patch deployment was all caused by one bug. One single, solitary bug. The cluster just would not start because of it. The funny thing was that the fix we deployed to get Squidwolf Minecraft running prevented all other servers from starting up. This is especially ironic since the Alive patch worked on all the other clusters including Squidwolf Hosting, just not SM.

When it rains, it pours I guess. We should be through the roughest part now, though last night’s issue was due sizeone of our database drives running out of memory. That was pure management bork, it should have had enough room for an incident like that. So to fix it in more ways than one, we have ordered 32GB of DDR3 RAM to stick on Squidwolf Minecraft.

Oh, and this just in. The incident last night required some more work during yesterday’s downtime, which required us to fail-over our database cluster, which caused a lot of cache in our game servers to invalidate, which caused the startup problems today. Damn it. Then our network providers went haywire and the Bolton hosting center was disconnected. Someone give me something to bash Murphy with!

Anyways. We’re also hiring a lot of more Moderators to handle the petition flow. The response time is far from being satisfactory and we know it. So check that out if you are interested. For those of you wondering how you apply, you need to leave a comment below.

There are a lot of more, but this is getting really long. I’ll close this off with … The Inbox Monster

Some guy once said, “show me what you read and I’ll tell you who you are”. Or something. I think that if I show you some of what happens in my inbox, it might tell you a little bit of who we are. These are things I found in the oldest mails of the Game Design folder. Here goes. It’s DD/MM/YYYY.

15.4.2010 – Fibre Squirrel says Emealds gone haywire. NPC supply for them were incorrectly edited. Fixed.

15.4.2010 – Bookworm thinks we should build an insurance mechanic. Nice.

16.4.2010 – Squidwolf sees even more insane types on the market that go for good profits. Fixed. Wonder if he made a profit himself …

16.4.2010 – SoulSword says it’s way too easy to chain farm an NPC spawn since he was with a group of players doing it himself last night. Clever. Never forgave him for not bringing me along. We randomized the spawn areas and changed the logic a bit.

18.4.2010 – SoulSword doesn’t like all them Faction Mobs. I think he got griefed by some trying to gank some noobs. Wouldn’t surprise me judging by the fact that he managed to gank the dev that sat on the opposite side of the table at launch. Bastard.

19.4.2010 – Squidwolf on insurance again. Perhaps it shouldn’t cover Polaris offenses? The griefer in Polaris says the death penalty was high enough and that most people SAFEZONE’d got shot because of bugs or mistakes. (WOOP FOR DEVS!)

19.4.2010 – The highway situation. Blue Shadow (CosmicSword) says everybody was traveling through February now and we had to do something. We decentralized the highways. February died.

19.4.2010 – DemonSk8tr wants to create Spy drones which you attach to minecarts so you can trace them. We find too many drawback. Crap, nice feature to hunt down people.

19.4.2010 – ThatGuy had heard the tutorial was supposed to be revamped and wanted to help. He said it was so much crap it couldn’t be revamped and needed a totally new one. He was right.

19.4.2010 – Crimson completes build 1421 of Blue Light. Rejoice. Retort. Counter.

20.4.2010 – Entropy doesn’t like the exploitability of the static deployment of missions. More dynamic embarkation count implemented, more variety introduced.

20.4.2010 – GMs say that SafeZone and Sentries aren’t responding fast enough.

20.4.2010 – Entropy proposes subtle changes to the factions table since none of the grifers were really enemies and such. So we made them enemies.

21.4.2010 – Loot tables break and demonsk8tr gets lots of weird stuff, with premonitions of doom for the Squidwolf Minecraft economy if it wasn’t fixed since we can’t have too many Plasma physics skills out there. Turns out it was one NPC that was borked, in a mission demonsk8tr was doing. demonsk8tr’s briefer reputation was scarred for life.

22.4.2010 – SoulSword wants racial armor brackets. Nothing happened, but comes in when we do the overview much later on.

22.4.2010 – SoulSword suggest we give all NPCs a secondary attack, like a short range and long range attack. Ends up being far more complicated, with support mobs, explosives and such stuff. Comes in over a long period of time.

22.4.2010 – BlueShaodw asks if ore Processing is seeded with NPCs. It isn’t and I call him a carebear.

23.4.2010 – SoulSword doesn’t like people in the NPC factions and wants them all thrown out. Now, nobody ends up in a starter faction

23.4.2010 – ThatGuy analyzes Squidwolf Minecraft and finds out the client and the server were getting hammered with all the standing lookups. Needs a new design, talks about some programmer looking formulas like O( A * C1 * C2) and O( N^3 ). I still have no clue what he is talking about.

26.4.2010 – Bookworm sends yet another post about hosting slots, from hell.

26.4.2010 – ThatGuy points out that he’d like npc friendlies in Factions, so we can do “PaperClippy the Faction member”. We hate the Paperclip in a certain help feature. But friendlies was something we always wanted to do, so tabbed for a later look.

27.4.2010 – BlueShadow is worried that you don’t get a notification if your chests are being opened by someone.Internally dismissed as carebear whine, we quietly stopped opening her chests (yes we know that sounds kinky).

28.4.2010 – Bookworm suggests a gang proposal with gang abilities. We think it’s a really good proposal, at least one of the best so far. It evolves and a derivate of that has started coming into Squidwolf Minecraft today. Talk about a slow birth.

28.4.2010 – SoulSword wants to show damage done, seperated into damage type, with nice colors and stuff. It was one of those nice ‘to haves’ that’s still lingering around on a todo list.

29.4.2010 – SkyCaptain finishes the first “dungeon” in Squidwolf Minecraft and writes a ToDo. It’s all text commands and it takes weeks to create one small one. We decide to hire in people to do a real tool and create a lot of them. They end up as Complexes, yet they are still not in the game.

30.4.2010- Squidwolf has changes to the physics lined up. I read it and see some really complex math stuff (he still hates math). We survived.

2.5.2010 – I send out a huge word document. Nobody bothers to read it. Gets split into pieces with a mental note to not write so much. Ends with addition of NPC Officers and Commanders, Officer and Faction loot, Implant sets, Industrial Operations, Faction Minecarts and such. Didn’t get the NPCs smacktalking in tho. Escalating paths first mentioned. That one’s coming!

3.5.2010 – We decide to change arrows, since everybody is now equipping bows & arrows and completing disregarding swords.

3.5.2010 – Squidwolf introduces new market, market in regions, new restrictions. There was much rejoicing!

6.5.2010 – Squidwolf Minecraft is closed.

9.5.2010 – New ores powered by SpoutCraft? hmm…

10.5.2010 – SoulSword suggests we do better incorporating gangs in faction war flagging, that wars spill over and supportive actions also included. Lots of big badaboom after that.

12.5.2010 – GMs request the Message of The Day (MOTD) in the client. They got their wish. Now we just had to find someone to put updated information in there.

17.5.2010 – BFG points out the exploitability of mines. They are removed and will be introduced as structures laying mine-shafts. Uhm … right.

That’s enough for this round folks. I have lots more where that came from. Of course, please note that these have been considerably simplified since the discussion usually involved an e-mail war of ~20 emails and some spicely adjusted for humor.

Why Patch Now? Security, Encryption, SpoutCraft, and Faction Economy

I had been reading the forums for patch feedback and such and after responding to a couple of threads, I think it’s a good idea I gather everything into a blog, not only for posterity but simply because a lot of people don’t read the forums at all, let alone a thread with a dev answer deep inside it.

Why patch now when the game is having so many problems?

We’re patching the game exactly because of the many problems plaguing the network. MIKA is the biggest code-only fix, improvement and optimization patch we have done to Squidwolf Server in years.

As a result of the extensive work done in the code, (total code is now about a million lines), we know there are going to be deployment problems. Veterans of Squidwolf Minecraft know this as “Patch Day Syndrome”.

Many have asked (or will ask), why doesn’t Project: Hazel test their patches? This is the eternal problem of testing high loads. We can’t duplicate the code’s native environment (Squidwolf Minecraft), because we have no way to replicate the load of 28k entities in the game. We do a lot of testing before we deploy patches and the ALIVE code branch, our internal nickname for the code branch, has gone through these phases of testing:

  • 2 Months of Alpha testing in Polaris
  • 1 Month of closed Beta testing in Sirius
  • 1 Month of open Beta testing on the Alive test server. Max concurrent load was about 38,000 entities.
  • Deployed for Release this month

Since then, we have further fixed and improved the ALIVE code base, which then adds 2 months of internal and public testing on Alive.

Those additions are small in comparison with what was done before the august release, then again we must remember that in the Alive environment it was released on a clean database. Here, we have to migrate a lot of data and upgrade a lot of database tables. This has also been tested on Alive, but even multiple tests there have not prevented issues from popping up in the last minutes.

So for short, ALIVE is supposed to help Squidwolf Minecraft out the current state, but it will be bumpy getting there.

Why cease support for SpoutCraft? Security? Encryption?

Let’s start with the Spout project. A player on the forum stated that less than 3% of visitors to his websites had the SpoutCraft client that we are stopping support for. For us, it’s less than 10% of visitors to the game world.

This still doesn’t make this an easy thing to do for us, as we have invested a lot of time developing features for Squidwolf Minecraft that depend on SpoutCraft.

I stated in the news that not only will the SpoutCraft client no longer work with Squidwolf Minecraft, but also the Spout project has completely stopped developing it.

However, this is just half the story. The security and encryption added to the server is very necessary, not just for our safety but yours too. Most client modifications will no longer work, we have enhanced over-the-wire security and beefed up authentication to name a few.

This wasn’t a decision made lightly or made by me in a drunken stupor (which I’m highly likely to do) but with us crying in a corner in the fetal position.

You see, our strategy is to support more Minecraft clients, not fewer. That’s why we want to work with RAGE, that’s why we’re looking at Squidwolf Minecraft Mobile clients for PDAs and more.

To those users with SpoutCraft, I hope you can understand our reasons, although I’m pretty sure it doesn’t alleviate the pain of having to upgrade or worse, quit if you decide not to.

Faction Economy changes

On the Faction Economy changes, this is more to clear up some misunderstandings and provide answers to what’s going to happen next.

Please note: No changes are being made to the land claiming mechanics or SquidCredits you have accumulated, ot the mechanism itself. Only that Faction Vaults no longer have any use. You can simply manually remove the emeralds within those chests and put them in your own.

First of all, we have a very old economy system which is costly in resources, is limiting in what we can do and requires the plugin creator to have a rudimentary understanding of Java 7 and our own unique programming language (HazelCredit) to create a native economy add-on, and the associated chances of bugs cropping up on all levels of the authoring of missions. This is bad, mkay? 🙂

We have had for some time a new economy system, which doesn’t require any of that understanding enables us to do more than the old system in a shorter period of time.

The “new” system was actually introduced in Inherit Velocity and you mainly noticed it in the early days of 2013. This is because the new system enables us to do things with developing our storyline which will be, like, really cool!

What is happening in this patch is we are removing the old system, which means the entire Faction Economy will be removed. But I stress again, YOU WILL NOT LOSE YOUR EMERALDS!

What does the future behold?

Since the introduction of the new economy system, lag has dropped by 20%

We have taken the freed-up server resources and added Faction Mobs.

We’re very sorry the transition can’t be smoother than this, but we want to assure you that we are really improving the economy system and we have a lot of people working on creating storylines and missions.

Oh, and to reply to the eternal question I’m quite sure will be asked:

Why do you have people creating new stuff instead of fixing stuff?

The people creating story lines and missions are content developers. They can’t write code, they can’t fix bugs in code, in fact, one could say that they were able to create bugs and if anything, they have less chance to create them now in the new agent mission system. 🙂

Is it me or is the new February server just so freakin’ awesome!

A bit of trivia, Project: Hazel is the only organisation of its kind to have such sophisticated hardware and software.

It’s Monday again … and the new February is about to rise

It’s been an interesting week resulting in the dreaded Monday being back again (I really really don’t like Mondays). The focus was on testing Blood and installing and testing all the new hardware for the new February cluster.

It’s all moving forward, which is good, but we would have liked to get the Patch deployed before the hardware upgrades because it includes a lot of fixes. You can’t always get what you want, though, and in this case the Gods of Time and Bugs were not on our side.

Last night we rallied almost 5000 connections over to the new February hardware, which was awesome, to say the least. This was the first day where 90% of the final hardware was in place. We really got the hammering we needed, and we found a lot of setup problems which we’re ironing out now.

We found some minor connectivity issues. Load balancing of proxies was a bit awkward, only 8 of 28 proxy nodes were accepting connections, some of our management tools didn’t like the new network setup, and so on and so forth. It’s also worthy of mention that the database being used here is the Alive test database – the monster is, of course, still running February. All in all, the issues are ones that we want to see cropping up now rather than on Tuesday 🙂

In other news, them new Patch looks good. I have more time to blog and fla** on the forums.

We’re all very excited for tomorrow.

I wonder if we can borrow a tank to move the SQL servers to Natori’s house tomorrow…

MOAR HARDWARE!!!

I’ve mentioned before that we have bought a new server cluster for February, but I haven’t gone into any detail. It’s a long road of testing, testing and then more testing to be able to reach the point where we can order a new February – and then there is some more testing!

The main considerations this time around were if 64-bit processors would run our 32-bit code better than our current 32-bit 3.06 GHz XEON processors. This alone would be reason enough to order the upgrade, so we started testing a 64-bit proxy server with our 32-bit code. To our amazement, a dual processing 64-bit machine performed at a level equivalent to three of our current 32-bit dual processing machines. That was the point where we ordered, YAY o/

But give me details … 

Yes, shortly. The main goal, however, is to use 64-bit code on 64-bit processors and the testing of this is still ongoing. Not only do we have to get Stackless Python and our own code running comfortably on 64-bit processors, we also have to make sure that our Java engine won’t run into floating point problems when it’s being used on different client platforms against our 64-bit server code (Called “Drift” or something like that).

Well, anyways, we ordered the new server cluster (YAY AGAIN) before Christmas, so it was kind of a big Christmas present for us here when we got confirmation of the order. It’s now roughly a month since then and we have already started to recieve some of the hardware.

A recent extended downtime replaced a couple of the old proxies with new RAGE 64-bit AMD processor blades, but the performance increase wasn’t really noticeable in gameplay since the performance problems are mostly located on the world servers. After all, it’s only 4 of 18 machines we have recieved.

Shut it and give me details … 

Ok, in a minute. We’re not only upgrading the physical servers themselves, we’re also upgrading the internal networks with new layer-3 and layer-7 switches between the individual Farms. You see, instead of old February, which has all its worlds servers in 1U rack mounts, all with individual networking cards, the whole cluster is in Natori’s basement.

Not only that, we’re also moving to a new hosting facility. Yes, that’s right, the new February will be assembled and powered on with access to better internet backbones. This is split between Natori’s house and the “Operation: Darknet” which you can read about in Squidwolf Magazine.

To begin with, we’ll have similar network setups as we have on February today to try to decrease the unknown factors in the initial setup phase, but we’ll then be in a position to connect to more backbones, further improving the connectivity to the new February.

You see, Squidwolf Syndicate doesn’t use that much bandwidth at all. Its peak usage is about 50 Mbit on peak hours. We have “fat pipes” coming in, but what we’re looking for is stable, low latency (“low ping”), well inter-connected backbones which provide the best routes to as many of our players as possible. This is achieved by getting high-capacity fiber connections to multiple internet backbone providers, which again should result in decreased network lag. (YAY o/)

GIVE ME DETAILS!!111111 

Well the thing is that I don’t actually have the details….don’t hit me plz. However I can answer some questions though =3
1. Does this mean the new super hamsters have arrived?
Yes, some of them have already arrived. We like to refer to them as “Internet Ferrets”.

2. What will happen to the old hamsters? 
They will spend their days watching gladiator movies.

3. Will the old hamsters receive a company pension and lifetime’s supply of cabbage? 
We were fortunate enough to get Protein Delicacies for them. Apparently, there were some issues with the ingredients according to the supplier, so we got them cheap.

4. Will the new hamsters require daily downtimes to go to the bathroom?
Yes, we were unable to genetically remove that deficiency.

5. How long is the DT likely to be while the hamsters are installed in their wheels? 
We require something like 24 hours. These are big and powerful Internet Ferrets and the wheels are quite elaborate. They also bite. And like gladiator movies. Fearful combination to say the least.

6. How spiffeh will Squidwolf Minecraft be on sunday evening at prime time? 
Hopefully, we should be quite spiffeh. But you know what they say about systems like that. When there is room for more people, they will come to fill it up.

7. How many new players are you anticipating?
Enough to make the baby jesus cry.

8. Will you be changing your node setup? 
No, we want be able to watch gladiator movies with the Internet Ferrets.

9. Will there be any server patch to complement the new hamsters? 
Yes, we call it Blood.

10. How many concurrent users will the new Network theoretically be able to hold? 
Given enough Protein Delicacies, we should be able to double the current peak usage.

11. Can I have the old hamsters? 
No, they will be busy watching gladiator movies, but I am in a position to get them on dates with you.

Happy Dev Petition Day

It’s been an interesting day here at Project: Hazel. Due to all the petitions and e-mails about various bugs and issues following code spill – much of which was due to the fact that various members did not have recent e-mail addresses registered – our support queues have grown quite big these last five days, so much that our team of 20 Devs aren’t able to cope.

Therefore, as result of the patch having been deployed, we decided to ask for assistance from RAGE to give us people in order to help Polaris out with solving petitions. Todays effort paid off; we were able to address 200 of the 400 petitions, focusing on the password reset petitions first.

I had my own way of doing things: I thought it would be a good idea to give everyone a free set of Diamond Armor on Squidwolf Minecraft and say sorry for the inconvenience. After about 50 of these, someone told me that was a bad idea, which I couldn’t really fathom since everybody that got one was so happy. Well, not really. It’s not as simple as that.

Petitions are far more complex than that, ranging from questions about gameplay to SpoutCraft being bugged and people being BLAM’d, lag causing people to shoot their friends and so on. However, that’s all a standard part of a Dev’s day and I think this effort at getting the dev team on the petitions seriously raises awareness on what we release and the implications it has. Even the smallest bug can cause harm to hundreds of people.

The effort today – and the fact that there are still 200 outstanding – means we’re gonna do it again tomorrow. To sum up, we’re doing everything in our power to get your petitions answered.

Optimisations!

You can expect the next two months to be dedicated to nothing but optimisations.

Final appearance change

Ok, probably not.

26u

Changed my hair again

678ike

O Canada! 70GB

We are pushing 70GB daily across our network!!!

Fullscreen Gallery