Thursday, January 3. 2008
no subject
I guess I'm going to keep referencing my old entry about Austin GDC '07, and on the same subject of databases. Apparently the folks from Cornell I mentioned published a paper on the subject shortly thereafter. I should be asleep, but I'm curious: I've been too busy to really pursue the subject of SIGMOD and database literature since the conference.
Thursday, December 6. 2007
solutions for databases in MMOs (and insomnia)
As has been mentioned before 1, database access in a real-time game with massive scale (relative to other games) continues to be a problem. Well, mostly. As someone pointed out to me recently, when you have a 1000-machine Oracle cluster with a 3:1 machine:DBA ratio, database performance tends to no longer be the bottleneck.2
One big part of the problem is that like all databases, there are a lot of different needs pulling the database design in different directions. It needs to be fast for storing and retrieving all of a character's information. It needs to support constant iterative development, as the information stored on each character (and on each account, and on each group, and so on) changes over time. In addition to all of these, there are going to be reports on the in-game data.
Reporting is the bit that I'm thinking about since I woke up an hour or so ago. There are a few kinds of reporting involved; one is deep game metrics. Because the whole point of deep game metrics is to find relationships and information you didn't anticipate, it boils down to this: be prepared to store a staggering amount of data, and be prepared to spend a lot of CPU cycles actually getting the information you want.
The other kind of reporting is quite a bit more lightweight: an overall look at the game as it stands right this moment. Theoretically, this doesn't require any different data than what you're already storing for the game; in practice, taking that approach to the database design means that your reporting tools have to know as much about the object-relational mapping as the game itself knows. Every time you add another bit of information to what's stored for characters, you break (or leave out-of-date) your reporting tools.
The solution for Dungeon Runners has been to mostly ignore the relational part of the database: almost everything about a character is stored in one binary blob in the database, that gets parsed by the game server on load. It's fast and easy to save or load, it was quick to develop, and the database design doesn't change unless we make drastic changes to the data representation (for instance, when Dungeon Runners went from one character per player to three, the database underwent significant changes). Load times are the most important metric, but with a small development team, "quick to develop" was a similarly important criterion.
However, it doesn't support any kind of reporting whatsoever. I'm not talking about deep game metrics (which have only weak relationships to in-game data like characters anyway), but it seems counter-intuitive for snapshot reporting. My solution - which also side-steps the problem of the reporting tools having to know a lot about how the game stores the data - is to store the data a second time, in a reporting-friendly format that is updated somewhat lazily. Right now we only do this to create a 'character sheet' in XML (essentially, a second blob that is easily pumped out to a web server).
In the future, however, this will extend easily to things other than blobs - some high-level data can be stored relationally as well, to support more web-accessible snapshot reporting. Stuff like a PVP leader board and other asynchronous competitions are simply infeasible to keep current unless it's extractable from the database with a single lightweight query. And frankly, "single lightweight queries" are not the forte of object-relational mappings, so we probably would have started duplicating data to accomplish this anyway.
1. I describe the problem there as "object serialization," but if you start with the assumption that you'll be using a relational database (Oracle or SQL Server or DB2 or PostgreSQL...) to store the data, then the issue is generally referred to as Object-Relational Impedance Mismatch.
2. If you're an MMO game studio with the budget to do this, let me know. You could probably score most of your programmers with that one promise, if you could convince them you weren't lying. Short of that, database performance will continue to be a bottleneck for games, even if it's a theoretically solved problem.
Now I go back to sleep.
Edits: The lazy updates happen each time the in-game data is saved, but scheduled asynchronously from anything the game client will notice. It can take as much as 30 seconds in my testing environment, and will probably creep up to a minute of lag in production. (Actually, in production ... it takes about 20ms to process each character.)
With the system I described above, lazy updates to the XML version of a character that is triggered by an update of the real version of the character - lag is 30 seconds to a minute. It takes another 30 seconds or a minute tops for that XML string to get replicated to another database that web servers can access, and then the next time a request hits the right web server for that character, it's there. So 1-2 minutes of lag between when a game server saves a character's progress, and its availability.
Compare this to World of Warcraft, where (I've been told) the Armory's character information is updated about once daily and ladders are updated once a week.
We want to keep the lag in the 1-2 minute range. It would be a pain to set up a scheduled process that could run that frequently, would only process freshly-updated characters, and could gracefully handle the situation where one process is started before the other completes.
Still, the point of my entire post was that we are using separate tables for reporting, and they are designed for the needs of the reporting system instead of the needs of the game system - which means "single lightweight queries" are all that is needed to get a report. The reports aren't super-flexible because we need to change what data is put in the reporting tables before many new reports can be used, but in general the system is sufficiently loosely-coupled from the game itself that these changes are fairly easy.
The Problems
One big part of the problem is that like all databases, there are a lot of different needs pulling the database design in different directions. It needs to be fast for storing and retrieving all of a character's information. It needs to support constant iterative development, as the information stored on each character (and on each account, and on each group, and so on) changes over time. In addition to all of these, there are going to be reports on the in-game data.
Reporting is the bit that I'm thinking about since I woke up an hour or so ago. There are a few kinds of reporting involved; one is deep game metrics. Because the whole point of deep game metrics is to find relationships and information you didn't anticipate, it boils down to this: be prepared to store a staggering amount of data, and be prepared to spend a lot of CPU cycles actually getting the information you want.
The other kind of reporting is quite a bit more lightweight: an overall look at the game as it stands right this moment. Theoretically, this doesn't require any different data than what you're already storing for the game; in practice, taking that approach to the database design means that your reporting tools have to know as much about the object-relational mapping as the game itself knows. Every time you add another bit of information to what's stored for characters, you break (or leave out-of-date) your reporting tools.
A Solution
The solution for Dungeon Runners has been to mostly ignore the relational part of the database: almost everything about a character is stored in one binary blob in the database, that gets parsed by the game server on load. It's fast and easy to save or load, it was quick to develop, and the database design doesn't change unless we make drastic changes to the data representation (for instance, when Dungeon Runners went from one character per player to three, the database underwent significant changes). Load times are the most important metric, but with a small development team, "quick to develop" was a similarly important criterion.
However, it doesn't support any kind of reporting whatsoever. I'm not talking about deep game metrics (which have only weak relationships to in-game data like characters anyway), but it seems counter-intuitive for snapshot reporting. My solution - which also side-steps the problem of the reporting tools having to know a lot about how the game stores the data - is to store the data a second time, in a reporting-friendly format that is updated somewhat lazily. Right now we only do this to create a 'character sheet' in XML (essentially, a second blob that is easily pumped out to a web server).
In the future, however, this will extend easily to things other than blobs - some high-level data can be stored relationally as well, to support more web-accessible snapshot reporting. Stuff like a PVP leader board and other asynchronous competitions are simply infeasible to keep current unless it's extractable from the database with a single lightweight query. And frankly, "single lightweight queries" are not the forte of object-relational mappings, so we probably would have started duplicating data to accomplish this anyway.
1. I describe the problem there as "object serialization," but if you start with the assumption that you'll be using a relational database (Oracle or SQL Server or DB2 or PostgreSQL...) to store the data, then the issue is generally referred to as Object-Relational Impedance Mismatch.
2. If you're an MMO game studio with the budget to do this, let me know. You could probably score most of your programmers with that one promise, if you could convince them you weren't lying. Short of that, database performance will continue to be a bottleneck for games, even if it's a theoretically solved problem.
Now I go back to sleep.
With the system I described above, lazy updates to the XML version of a character that is triggered by an update of the real version of the character - lag is 30 seconds to a minute. It takes another 30 seconds or a minute tops for that XML string to get replicated to another database that web servers can access, and then the next time a request hits the right web server for that character, it's there. So 1-2 minutes of lag between when a game server saves a character's progress, and its availability.
Compare this to World of Warcraft, where (I've been told) the Armory's character information is updated about once daily and ladders are updated once a week.
We want to keep the lag in the 1-2 minute range. It would be a pain to set up a scheduled process that could run that frequently, would only process freshly-updated characters, and could gracefully handle the situation where one process is started before the other completes.
Still, the point of my entire post was that we are using separate tables for reporting, and they are designed for the needs of the reporting system instead of the needs of the game system - which means "single lightweight queries" are all that is needed to get a report. The reports aren't super-flexible because we need to change what data is put in the reporting tables before many new reports can be used, but in general the system is sufficiently loosely-coupled from the game itself that these changes are fairly easy.
Wednesday, November 14. 2007
Some nice press
Ten Ton Hammer's Dungeon Runners Chunk #2 Preview
I keep going back and reading two parts:
and
Hopefully, they will reconsider that score when we blow Chunk #2 - it's going to do quite a bit to polish the game as it stands, add more content (this happens all the time, almost not worth mentioning) as well as provide the side benefits of "more of the game for free players" and "more money for us."
Edit: Hi codybye! :-)
I keep going back and reading two parts:
[Dungeon Runners] was the surprise hit of 2007.
and
While I originally gave Dungeon Runners a 3.5 / 5 Hammers, I’d probably have to seriously reconsider that score these days.
Hopefully, they will reconsider that score when we blow Chunk #2 - it's going to do quite a bit to polish the game as it stands, add more content (this happens all the time, almost not worth mentioning) as well as provide the side benefits of "more of the game for free players" and "more money for us."
Edit: Hi codybye! :-)
Saturday, September 8. 2007
my toys broke
Lum the Mad has a more complete write-up on Austin GDC — and the fact that it's not AGC — up on his blog. Naturally as a designer / producer he focused on different sessions than I did, but I agree with his overall assessment of the conference, and we did both go to Raph Koster's session.
Friday, September 7. 2007
Alive!
I survived Austin GDC (née AGC) pretty well. It's not nearly as hard as an attendee as for a volunteer (or someone staffing a booth). I very much enjoyed the technical sessions this year: I mostly attended talks on scaling the servers up to handle thousands of players, and gathering metric data.
Scaling continues to be a hot topic, with a surprising focus this year on databases. Latency seems like the big issue with them - and pretty much everyone seemed to have a custom cache layer between the database itself and the game servers, with some horror stories about games going live without them. The other big issue with databases is object serialization - how to store a character, all of the characters items, etc. in the database the "best" way. Nobody agrees on this one, naturally, because it's a fairly hard problem.
Interestingly, there was a professor (from Cornell?) and grad student who had very strong opinions on the subject of databases. Why can't game developers just read the literature? Why don't we realize object serialization is a solved problem? I'm interesting in doing more reading, but I suspect that latency is going to continue to be the bugbear here - what's considered acceptable performance in the wider database community is, I think, different.
Metrics (what are players doing? how often?) were also a big subject this year, up from the approximately one talk on the subject last year. There were a lot of interesting approaches discussed, and of course the big issue here is storage. One game, for instance (currently in beta) is generating 4.5 million database rows a week collecting data on game play.
Edit: The real tricky bit is storing objects in a database. Once you start using a database, suddenly people want to use the database like any other database: data mine it. The simple ways of serializing data (objects) into the database don't really let you also data mine, but setting it up for easy data mining tends to screw your performance.
Scaling continues to be a hot topic, with a surprising focus this year on databases. Latency seems like the big issue with them - and pretty much everyone seemed to have a custom cache layer between the database itself and the game servers, with some horror stories about games going live without them. The other big issue with databases is object serialization - how to store a character, all of the characters items, etc. in the database the "best" way. Nobody agrees on this one, naturally, because it's a fairly hard problem.
Interestingly, there was a professor (from Cornell?) and grad student who had very strong opinions on the subject of databases. Why can't game developers just read the literature? Why don't we realize object serialization is a solved problem? I'm interesting in doing more reading, but I suspect that latency is going to continue to be the bugbear here - what's considered acceptable performance in the wider database community is, I think, different.
Metrics (what are players doing? how often?) were also a big subject this year, up from the approximately one talk on the subject last year. There were a lot of interesting approaches discussed, and of course the big issue here is storage. One game, for instance (currently in beta) is generating 4.5 million database rows a week collecting data on game play.
Edit: The real tricky bit is storing objects in a database. Once you start using a database, suddenly people want to use the database like any other database: data mine it. The simple ways of serializing data (objects) into the database don't really let you also data mine, but setting it up for easy data mining tends to screw your performance.
previous page
