Anson the Gnome

So... yeah. Very quiet here. First it was starting the new job at BioWare, then it was buying a house, then it was starting to take classes (I'm trying to go back for a Master's degree). Somewhere in there, my web server got misconfigured, resulting in approximately a 20% HTTP 500 (Internal Server Error) error rate. That was hard to track down, but in the meantime made it difficult to write new blog posts (or read the site, for that matter).

I decided to restart my blogging with a subject that I've been thinking about for a couple of years, but never quite had the gumption to blog about: security in MMOs. There are a few kinds of general security areas with online games, and not all of them are as interesting as others. After being involved in a few projects, I've noticed that trade systems frequently don't get the security consideration I would like¹. To that end, I'd like to highlight some of the general concerns that should be taken into account in building a trade system for an online game that doesn't immediately blow up in your face.

One common problem with MMOs is that they avoid a perfectly relational database model in the persistence layer. Here's an example of a situation where that hurts you²: the database itself can't enforce ACID, so the rest of the game software has to - and as it turns out, generally doesn't have the same level of scrutiny applied to it that database engines do. So we've evolved not-quite-relational ways of reducing the window of exposure. One example: when two players execute a trade, both of their characters (which may not be serializable at higher resolution than "the whole character") are saved to the database in a single transaction so that either the whole trade succeeds, or the whole trade fails. If you do that, by the way, make damn sure the error bubbles up all the way: otherwise the trade transaction could fail but leave open the possibility of future character saves (where both characters aren't saved in a single transaction) persisting an incorrect state. You also have to make sure precisely one authoritative in-memory copy of both characters exists across all servers, or else that perfectly valid trade transaction could be overwritten by another server (which could result in item duping, or completely erasing the items traded from the game, depending).

All of that is very interesting, but it's pretty much textbook examples of transactions (including the not-completely-ACID part - bank transactions are NOT that perfect :-). The more interesting part is preventing players from scamming each other. For example, when Dungeon Runners first launched, it had no built-in support for a trade system: if two players wanted to trade, they both dropped a pile of stuff on the ground, and picked up the other guy's pile. Scamming wasn't so much a problem as it was the day-to-day experience. On my most recent previous project, one iteration of our trade system³ involved - at the network protocol layer - one player's computer telling the server "this is the trade, and by the way the other guy already agreed to it." That wasn't actually a problem for our prototype, since none of us were hacking the client and anyway we were all in one room - if someone cheated, we could kick them in the shin.

So how do you do it right? First of all, treat the whole thing as a transaction. Store the whole trade on the server, and if there's a problem halfway through executing a trade, be prepared to roll it all back (also? test this code). More than that, though, assign every step of the trade process a globally unique identifier. When a player initiates a trade, that's a unique ID. When the other players responds with a counter-offer (or even an initial response offer, if it begins with "I'm offering X, what can you trade?"), that's a new unique ID. Each change and counter-offer... unique ID (this ID is also shared with both parties' game clients - it isn't a secret). The final step is when both players agree: both have to indicate their acceptance of the trade using the ID for that version of the trade.

The server, of course, tracks what the ID is, and more importantly what the most recent ID is - and the trade doesn't happen unless both players agree. This solves what seems to be the most commonly-seen trade scam across MMOs, and stops it dead: the network latency scam. Without these unique IDs, it is possible to change the offered trade right as the other player (the scammed player) accepts it - and then immediately accept it yourself. Offer a sweet mythical axe, and then swap it for a newbie dagger - and get whatever the other player thought the axe was worth.

Of course that scam doesn't work every time, but if it ever works, some players will try over and over until it does. There are also trickier variations that rely not on protocol problems, but on information presentation: for instance, swapping out the sweet mythical gear for trash loot that has the same icon (or, if you can trade many items as a stack, offer a stack of 99 uberpotions and then reduce it to 1 such potion before the trade actually happens). The solution, in these cases, is to enhance the information presentation - that is, the user interface - to highlight and emphasize trade changes. Make the axe's icon pulse until the user mouses over it. For that matter, don't let a user accept the trade until they've moused over the new items, so that they see what it is (assuming most details only show up on mouse-over, of course). Consider a waiting period after each change to the offer (1-2 seconds, nothing unpleasant for the user) before the user can accept - this also serves to highlight that the change occurred, and prevents them from clicking it as the offer changes (a more minor timing problem, to be sure).

Generally speaking, though, worry about the unsuccessful path. Worry about timing issues, information presentation, out-of-band attacks... there are people whose jobs it is to develop a taxonomy of security threats, and most of their results apply to MMOs.

---

1. I may be a bit of a grognard in this regard: my entire philosophy in managing the server that runs this blog, for example, can probably be summed as "paranoia." I run OpenBSD, I pay close attention to everything from partition layout to process management to minimize exposure from security holes. And yes, I use PGP. I don't mean to say this system is unhackable; merely that it is a tougher nut to crack.
2. Of course, there are also plenty of times where a nicely relational database model will bite you in the ass. There isn't a perfect solution.
3. Yes, by the time our office was closed we were on our third iteration of a trade system. That third system incorporated everything I mention here, plus it had a pretty polished UI; I was very proud of it. You only get that out of the third (or later) iteration; the first iteration was a hideous monster-child I felt obligated to drown like Oswald Cobblepot in Batman Returns.

James Gosling comments:



We'll see what happens, but it seems reasonable to assume the character of major Sun technologies like Java and MySQL will change. How exactly is hard to say, although I suspect in Java's case, Oracle is even less inclined to support standardization and open source than Sun was. MySQL is harder to predict, but some change seems inescapable.

I expect Oracle to exert relatively little influence on MySQL's ongoing development, because several core MySQL developers left Sun prior to the merger and have the community support, skills, and funding to seriously threaten a fork. I also don't expect Oracle to make much money on (new) MySQL support contracts, due to a combination of lack of trust and expertise that has (or will) leave the building. Hopefully PostgreSQL will see an uptick in usage as well.

I've had my vanity domain, idempot.net, for a pretty long time (I know plenty of people who have had vanity domains for longer, but 2002 was about when I realized I needed an email address that lasted longer than any single employer). I don't remember how exactly it started, except with the observation that you could make some puns with '.net' and words like 'omnipotent.' Of course omnipot.net was taken, and I didn't really want impot.net... but I was familiar with another 'potent' - 'idempotent' - from both the pure mathematical meaning, and the C programming idiom.

Since idempotency is a form of uniqueness, it inspired the further pun of notunique@mac.com - the only .Mac user who is notunique. Then Apple started charging for mac.com addresses, and I was unemployed around the time a renewal fee was due, so I gave up on that pun, and started using idempot.net consistently. I liked it better anyway: it is unique, it's punny, and it's geeky.

On the downside, t's hard to pronounce. Not as bad as some, but not as thoroughly in the zeitgeist, either. Personal business cards have helped, but it's still a pain some of the time: if I'm giving my email address over the phone, if I've run out of cards (as happens late in the night at conferences), or if someone simply transcribes it incorrectly.

So now, to fix those problems in the future, I've registered easytopronounce.com. I don't know what, exactly, I'm going to do with the domain, but in the short term it'll just point here.

Happy New Year, everyone. No prognostication from me at present, I'm still figuring out what kinds of things I can talk about since I work for a publicly traded company.

Last night marked the end of Dungeon Runners; it is gone. I haven't been involved in the project, or really the community, since I left NCsoft a year and a half ago, but I am still very sad to see it go. There was something really special there, and there isn't even a hollow shell left to remind us.

Clearly I haven't done a very good job of blogging lately. I'll chalk that up to putting a lot of effort into finding another project to work on.

Today, I started a new job at BioWare, as an Infrastructure Engineer. It's not a glamorous job - I'm working on the tools and systems that keep the more game-specific teams agile - but as I've said a few times, there's a lot to get right when you're making, launching, and operating a successful MMOG.

That's pretty much what I want to do right now, with Dungeon Runners about to close and my last project never seeing the light of day. I'm in this industry to make people happy, and I'm working on MMOGs in particular because they can reach such a broad audience for so long. I've had some trouble actually making that happen, but I think BioWare can do it - that's a horse I want to hitch my wagon to.