Current Issues & The Inbox Monster

30th August 2013 by Shelby

Squidwolf > Blog > Dev Blog > Current Issues & The Inbox Monster

As you may have noticed, we’ve started putting more effort into Known Issues. I like it, because we try to be open about Squidwolf Syndicate and this is a step forward. I can also see that people feel that after they saw this, there are more bugs in the Alive patch. Well, that’s not entirely true, this is the first time we’ve made a concerted effort to list that many of the known issues. Our internal system shows considerably less issues in Alive that say Timelines, which is interesting compared to how comprehensive the rewrites were.

We don’t need no stinkin’ queues!

However, I’d like to address some specific concerns. First, there are these queues which there are two types of. The log-on and the embarkation. The way the log-on queue is disabled is that it’s limit is set very high, allowing dozens to log on every minute. It only kicks in at startup, preventing the crash of the cluster and if the system you are trying to log into isn’t loaded yet.

You may notice the “system not loaded” mostly in the first minute of you logging in, but in reality it was always there. Before this mechanism came in, you sat in ‘The Void’, and waited a long time for the system to load till you finally got in. This resulted in many clients timing out while waiting or even worst, get stuck. Now, you simply aren’t allowed to wait in that ‘Void’ state, shouldn’t time out and shouldn’t get stuck. In short, it’s designed to prevent bad things from happening and so far it has.

The embarkation queue on the other hand is far more tricky. We believed it was effectively “disabled” by setting it to kick in at 95% or more CPU usage. This is a very bad node state and means that you are more than likely to get stuck if you get into a world like that. The first problem was that the system discarded the setting and used it’s default, 85% CPU, which is quite common. In cases of up to 8 continents being on the same node, that would effectively close down about 80 solar regions (32 chunks per region) or so.

The second parameter used is more than 70 people in a single solar world. We haven’t hit that yet and that usually only happens in worlds which are already on a dedicated node, so it only affects itself.

The reason this was set in was to prevent worlds from overloading and getting players stuck. This was especially dangerous with the large worlds (such as Edge, and February), because when these nodes with the high load world died, they would start a chain reaction that would eventually take the whole cluster down.

But, this is not without its drawbacks. Even though we’re seeing systems like Edge only running at 55%-75% CPU – even on Mass Test Days – after the Alive patch, we still had abnormal behavior which brought spiked CPU usage up and thus caused embarkation queueing.

You see, after Alive, the codebase is much more stable, using far less CPU and we have a more clear picture of what is causing a lot of load spikes. Call it a good signal-to-noise ratio. It probably won’t surprise you, but Mob Engagements is pretty high on the list, so high in fact that we will be putting in even more drastic limitations to Mob Engagements. Until we of course implement the replacement for the gameplay which Mob Engagements introduced. What’s in the coming patch is simply to address the load associated with copying large amounts of them. More on that in another blog which is on the way.

We’re still tuning the queues to do what they are supposed to do and to prevent them from being a nuisance to everyone that wants to ride around on horses (or worse, getting you shot while waiting or can’t move in a world with no players in it). We have a patch in Testing on Alive (Server) that addresses some of the Known Issues that have fixes we could safely test and deploy, while others will have to wait for a better time. If all goes well, we will see it on Squidwolf Minecraft today.

Other issues and Squidwolf Tower

Going over Squidwolf TOwer really creates a big list of things you want to answer, but being a dev and answering the same question 27 times, while the some in the thread saying “zOMG DEV NOT REPLY YOU DON’T CARE!!!” doesn’t really provide an encouraging environment. When I was working on Squidwolf Gaming Network I laughed when someone demanded I answer immediately at 5 in the morning. We do sleep occasionally so I’m sorry, it won’t happen again 🙂

You see, Squidwolf Tower and Squidwolf Insider sucks at delivering information to the whole community, simply because not everybody reads them. We read them, occasionally reply but we can’t (and won’t) reply to every thread there. If I spend an hour each day on the websites, that 7 hours a week I’m not working on making Squidwolf Minecraft better. Actually, I put in that many hours anyways that I do read forums about 1 hour a day 😛

Anyways, the dev blogs have shown far better results in spreading the word and opinions of developers and thats the main reason we try to emphasize more on dev blogs.

Take the ‘offline’ incident. We answered these very same questions the last time the [Insert any big incident here] was in the spotlight. We do not interfere, unless there was a violation of the EULA or Terms of Service. I mean, people transfered this guy their Sc. He then doctored screenshots and whatnot to prove his legitimacy. Then shows screenshots of how big the scam was. Hmm.

We’re not going to replace trust, nor are we going to fix situations where people send their money to some guy in hopes of getting it all back. Then again, we’d like to support IPOs and other such models, but that’s down the road.

DragonTravel. What an annoying bug Alive brought about. The Alive issue raised the question whether it’s to overpowered. Or with the first of the fixes if it’s too underpowered. Or overpowered. Or … you get the drift. We managed to get this fix in tomorrow’s patch because it was low risk. Be patient, there is more to fix and more patches to come and we’ll address cloaking properly.

Also regarding the misunderstandings on the forums, we have 17 people on staff, misunderstandings happen. In an effort to maintain our openness with the community that many of you have said is one of our big strengths compared to other companies in the MMOG industry, we run the risk of miscommunication when we allow more than one person to post on the forums. We accept the risk, I hope you do too.

Log-on and log-offs tactics. Tricky thing too. In a world where the most important thing for many are their reputation, others don’t really care and act like cowards. That’s right. That’s my not-so-humble opinion. You need to resort to log-on traps to kill your enemy, you get no respect. But then again, that’s your choice.

We’ve changed these mechanics quite often over the last six months, the main issue being to balance between connection loss and other legitimate situations to prevent the exploitation of the loopholes. Changing this mechanic affects everyone, and that’s the problem. However, we are indeed looking into it. Good suggestions welcome.

SafeZone Ping Pong Wars. No, we don’t like them either. We are working on solutions, most likely a part of MIKA Two, where we work mainly on warfare and player infrastructure, the basis for large scale warfare. We’re also doing a lot with combat in general and you are seeing a lot of that already in MIKA One. Things like changes to whether or not you can shoot from inside a SafeZone, relative adjustments to hitpoints vs. class and a general hitpoint boost, some likely changes to range and so on.

Talking about working on the game. Some of you believe we’re spending all our time on Squidwolf Hosting. That’s certainly not true. It was partly true, as I explained some time ago but as you can see, the Alive code build, which all this work was used on, is performing considerably better.

The Alive deployment also signifies the point where we’re releasing patches according to our original plans, where Squidwolf Hosting is on a delayed release schedule, our efforts now focused on fixing Alive on Squidwolf Minecraft and wrapping up MIKA, while our partners in Squidwolf Hosting. For those of you asking, MIKA is the internal code name for Transforming Experience. This was deployed on 11 August.

Recent downtimes and patch deployment. Well, before patch deployment, the old Squidwolf Minecraft code base was really starting to show its age. Bugs would crop up, which started filling logs, which then eventually started filling the hardware, which then crashed the whole thing. We also saw a lot of load related things that took down Squidwolf Minecraft and we have had some networking problems to top it all off. Unfortunate timing, Murphy sticks his hand in again. Alive resolved a lot of the load issues, but also got blamed for a lot it wasn’t responsible for.

The late patch deployment was all caused by one bug. One single, solitary bug. The cluster just would not start because of it. The funny thing was that the fix we deployed to get Squidwolf Minecraft running prevented all other servers from starting up. This is especially ironic since the Alive patch worked on all the other clusters including Squidwolf Hosting, just not SM.

When it rains, it pours I guess. We should be through the roughest part now, though last night’s issue was due sizeone of our database drives running out of memory. That was pure management bork, it should have had enough room for an incident like that. So to fix it in more ways than one, we have ordered 32GB of DDR3 RAM to stick on Squidwolf Minecraft.

Oh, and this just in. The incident last night required some more work during yesterday’s downtime, which required us to fail-over our database cluster, which caused a lot of cache in our game servers to invalidate, which caused the startup problems today. Damn it. Then our network providers went haywire and the Bolton hosting center was disconnected. Someone give me something to bash Murphy with!

Anyways. We’re also hiring a lot of more Moderators to handle the petition flow. The response time is far from being satisfactory and we know it. So check that out if you are interested. For those of you wondering how you apply, you need to leave a comment below.

There are a lot of more, but this is getting really long. I’ll close this off with … The Inbox Monster

Some guy once said, “show me what you read and I’ll tell you who you are”. Or something. I think that if I show you some of what happens in my inbox, it might tell you a little bit of who we are. These are things I found in the oldest mails of the Game Design folder. Here goes. It’s DD/MM/YYYY.

15.4.2010 – Fibre Squirrel says Emealds gone haywire. NPC supply for them were incorrectly edited. Fixed.

15.4.2010 – Bookworm thinks we should build an insurance mechanic. Nice.

16.4.2010 – Squidwolf sees even more insane types on the market that go for good profits. Fixed. Wonder if he made a profit himself …

16.4.2010 – SoulSword says it’s way too easy to chain farm an NPC spawn since he was with a group of players doing it himself last night. Clever. Never forgave him for not bringing me along. We randomized the spawn areas and changed the logic a bit.

18.4.2010 – SoulSword doesn’t like all them Faction Mobs. I think he got griefed by some trying to gank some noobs. Wouldn’t surprise me judging by the fact that he managed to gank the dev that sat on the opposite side of the table at launch. Bastard.

19.4.2010 – Squidwolf on insurance again. Perhaps it shouldn’t cover Polaris offenses? The griefer in Polaris says the death penalty was high enough and that most people SAFEZONE’d got shot because of bugs or mistakes. (WOOP FOR DEVS!)

19.4.2010 – The highway situation. Blue Shadow (CosmicSword) says everybody was traveling through February now and we had to do something. We decentralized the highways. February died.

19.4.2010 – DemonSk8tr wants to create Spy drones which you attach to minecarts so you can trace them. We find too many drawback. Crap, nice feature to hunt down people.

19.4.2010 – ThatGuy had heard the tutorial was supposed to be revamped and wanted to help. He said it was so much crap it couldn’t be revamped and needed a totally new one. He was right.

19.4.2010 – Crimson completes build 1421 of Blue Light. Rejoice. Retort. Counter.

20.4.2010 – Entropy doesn’t like the exploitability of the static deployment of missions. More dynamic embarkation count implemented, more variety introduced.

20.4.2010 – GMs say that SafeZone and Sentries aren’t responding fast enough.

20.4.2010 – Entropy proposes subtle changes to the factions table since none of the grifers were really enemies and such. So we made them enemies.

21.4.2010 – Loot tables break and demonsk8tr gets lots of weird stuff, with premonitions of doom for the Squidwolf Minecraft economy if it wasn’t fixed since we can’t have too many Plasma physics skills out there. Turns out it was one NPC that was borked, in a mission demonsk8tr was doing. demonsk8tr’s briefer reputation was scarred for life.

22.4.2010 – SoulSword wants racial armor brackets. Nothing happened, but comes in when we do the overview much later on.

22.4.2010 – SoulSword suggest we give all NPCs a secondary attack, like a short range and long range attack. Ends up being far more complicated, with support mobs, explosives and such stuff. Comes in over a long period of time.

22.4.2010 – BlueShaodw asks if ore Processing is seeded with NPCs. It isn’t and I call him a carebear.

23.4.2010 – SoulSword doesn’t like people in the NPC factions and wants them all thrown out. Now, nobody ends up in a starter faction

23.4.2010 – ThatGuy analyzes Squidwolf Minecraft and finds out the client and the server were getting hammered with all the standing lookups. Needs a new design, talks about some programmer looking formulas like O( A * C1 * C2) and O( N^3 ). I still have no clue what he is talking about.

26.4.2010 – Bookworm sends yet another post about hosting slots, from hell.

26.4.2010 – ThatGuy points out that he’d like npc friendlies in Factions, so we can do “PaperClippy the Faction member”. We hate the Paperclip in a certain help feature. But friendlies was something we always wanted to do, so tabbed for a later look.

27.4.2010 – BlueShadow is worried that you don’t get a notification if your chests are being opened by someone.Internally dismissed as carebear whine, we quietly stopped opening her chests (yes we know that sounds kinky).

28.4.2010 – Bookworm suggests a gang proposal with gang abilities. We think it’s a really good proposal, at least one of the best so far. It evolves and a derivate of that has started coming into Squidwolf Minecraft today. Talk about a slow birth.

28.4.2010 – SoulSword wants to show damage done, seperated into damage type, with nice colors and stuff. It was one of those nice ‘to haves’ that’s still lingering around on a todo list.

29.4.2010 – SkyCaptain finishes the first “dungeon” in Squidwolf Minecraft and writes a ToDo. It’s all text commands and it takes weeks to create one small one. We decide to hire in people to do a real tool and create a lot of them. They end up as Complexes, yet they are still not in the game.

30.4.2010- Squidwolf has changes to the physics lined up. I read it and see some really complex math stuff (he still hates math). We survived.

2.5.2010 – I send out a huge word document. Nobody bothers to read it. Gets split into pieces with a mental note to not write so much. Ends with addition of NPC Officers and Commanders, Officer and Faction loot, Implant sets, Industrial Operations, Faction Minecarts and such. Didn’t get the NPCs smacktalking in tho. Escalating paths first mentioned. That one’s coming!

3.5.2010 – We decide to change arrows, since everybody is now equipping bows & arrows and completing disregarding swords.

3.5.2010 – Squidwolf introduces new market, market in regions, new restrictions. There was much rejoicing!

6.5.2010 – Squidwolf Minecraft is closed.

9.5.2010 – New ores powered by SpoutCraft? hmm…

10.5.2010 – SoulSword suggests we do better incorporating gangs in faction war flagging, that wars spill over and supportive actions also included. Lots of big badaboom after that.

12.5.2010 – GMs request the Message of The Day (MOTD) in the client. They got their wish. Now we just had to find someone to put updated information in there.

17.5.2010 – BFG points out the exploitability of mines. They are removed and will be introduced as structures laying mine-shafts. Uhm … right.

That’s enough for this round folks. I have lots more where that came from. Of course, please note that these have been considerably simplified since the discussion usually involved an e-mail war of ~20 emails and some spicely adjusted for humor.

Comments are closed.