Sunday, November 2. 2008
Nobody else did it
...so I went ahead and submitted the Dungeon Runners in-game credits to MobyGames.com. Getting the credits put in was a tad rushed, so we weren't able to credit everyone as well as we should. My apologies, it's hard enough to do due diligence and fact-checking on credits when you aren't rushed by deadlines. :-(
Sunday, October 19. 2008
The Art of SQL
I made a brief mention of The Art of SQL, but in light of my last fairly 'meh' book review, I wanted to talk about good books a bit more. I've been working with databases for a few years now, and I had some really great mentors on the subject early on... so I have felt pretty comfortable with my level of expertise in the area. Still, Faroult and Robson completely blew me away with this book, and reading it helped me understand and solve a whole new range of problems in databases.
The attempt at mapping chapters and subjects onto The Art of War is a bit of a waste of time, but it seems to mostly involve goofy chapter titles and preface. More frustrating for me personally was the discussion of (yes, here we go again) trees and hierarchies in SQL. It's a nice discussion of accessing data in hierarchical formats, but it avoids handling some very important areas: nodes with multiple parents (consider MMO crafting recipes as an example: ingredients can be used in multiple recipes, and some recipes yield things that can be used as ingredients in other recipes), and on-the-fly inserts into trees.
The authors' best advice regarding trees, incidentally, is "SQL still lacks a truly adequate, scaleable[sic] processing of tree structures." It's hard to disagree with that conclusion, and it's hard to argue the authors' decision - in light of that conclusion - to not fully address tree structures in SQL. Out of curiosity I'll probably take a look at Joe Celko's Trees and Hierarchies in SQL for Smarties soon, but to be honest I think professionally my solution is going to be "stay the hell away."
However, in general the book covers a lot of ground in much better depth than most books - offering interesting examples, SQL demonstrations of principles, and performance comparisons of different approaches that is well worth reading, and keeping handy as a reference. It is, frankly, the only database book I've read thoroughly and still been satisfied enough with to recommend.
The authors' best advice regarding trees, incidentally, is "SQL still lacks a truly adequate, scaleable[sic] processing of tree structures." It's hard to disagree with that conclusion, and it's hard to argue the authors' decision - in light of that conclusion - to not fully address tree structures in SQL. Out of curiosity I'll probably take a look at Joe Celko's Trees and Hierarchies in SQL for Smarties soon, but to be honest I think professionally my solution is going to be "stay the hell away."
However, in general the book covers a lot of ground in much better depth than most books - offering interesting examples, SQL demonstrations of principles, and performance comparisons of different approaches that is well worth reading, and keeping handy as a reference. It is, frankly, the only database book I've read thoroughly and still been satisfied enough with to recommend.
Wednesday, October 1. 2008
Agile Database Techniques: Effective Strategies for the Agile Software Developer
Agile Database Techniques was a bit of a disappointment. In particular, it was targeted at getting DBAs to be more 'agile' and open to an iterative, evolving database design... which I can agree with, but I didn't need a book for that. Past that, the book spends a lot of time introducing the reader - not "agile software developers" as the title implies, but mostly DBAs - to agile strategies, database design, and finally (finally!) object/relational problems and solutions.
To some extent, object/relational mappings are a solved problem. Object Relational Brokers can do all the heavy lifting for you if you're using a language like Java or C# (I'm not), and if you're not worried about the number of database round trips required to gather your data (or update it). Add in a hierarchy of objects containing other objects (in addition to inheritance), and you're basically done with ORBs and/or performance.
That, really, is the bugbear I should have been talking about a couple of years ago. It's not some vague "databases aren't fast enough" problem, it's "collecting a tree of objects, each of which is also in an inheritance tree, is slow." Individual objects are no big deal... definitely a solved problem. A collection of objects in a language with limited run-time reflection, on the other hand...
I had hoped that Agile Database Techniques would address these larger-scale object/relational mapping problems, but unfortunately it has not. It appears very much targeted at the enterprise software market (one where, in my opinion, "agile" is relative), and avoids (predates?) the web-oriented agile development community (where I have also been looking, to some extent).
So, it is a decent read, and puts in one place a lot of information on iterative database development in enough depth that you can connect the dots and do it yourself (e.g., test-driven development is a perfectly valid strategy for database schema design too). Not so awesome if you're not a DBA, not working in enterprise software, or dealing with the particular problems of games, but still a decent read. At $26 from Amazon, it may be worth putting on your bookshelf... as long as your expectations are reasonable. :)
That, really, is the bugbear I should have been talking about a couple of years ago. It's not some vague "databases aren't fast enough" problem, it's "collecting a tree of objects, each of which is also in an inheritance tree, is slow." Individual objects are no big deal... definitely a solved problem. A collection of objects in a language with limited run-time reflection, on the other hand...
I had hoped that Agile Database Techniques would address these larger-scale object/relational mapping problems, but unfortunately it has not. It appears very much targeted at the enterprise software market (one where, in my opinion, "agile" is relative), and avoids (predates?) the web-oriented agile development community (where I have also been looking, to some extent).
So, it is a decent read, and puts in one place a lot of information on iterative database development in enough depth that you can connect the dots and do it yourself (e.g., test-driven development is a perfectly valid strategy for database schema design too). Not so awesome if you're not a DBA, not working in enterprise software, or dealing with the particular problems of games, but still a decent read. At $26 from Amazon, it may be worth putting on your bookshelf... as long as your expectations are reasonable. :)
Tuesday, September 30. 2008
Rest in Peace, Jeff
Scott Jennings has more details. I'm not sure if I can express this properly, but I was immensely grateful to Jeff for our online interactions. I started reading his blog when I first joined the Dungeon Runners team, mostly because he was (at the time) working for another studio in town being published by NCsoft. The more I interacted with him, the more I came to believe he was a genuinely wonderful person, as well as insightful and funny... and then he went and wildly praised one of my first big user-visible contributions.
The discussions that ensued, and his constant encouragement, had a huge impact on me and the direction I've gone since. I owe him a lot, yet all of our interaction constituted relatively little of his time; I can only imagine how many people he had that kind of effect on. My prayers go out to his family in this difficult time.
The discussions that ensued, and his constant encouragement, had a huge impact on me and the direction I've gone since. I owe him a lot, yet all of our interaction constituted relatively little of his time; I can only imagine how many people he had that kind of effect on. My prayers go out to his family in this difficult time.
Friday, September 26. 2008
Object/Relational Impedance Mismatches
Bleh, what a boring subject. Sorry. I'm spending a lot of time looking at technology to decide where to go for the new project, very similar to Joe Ludwig's current focus. I'm also reading a lot of books, trying to get a feel for what I should be doing now that I didn't want to or couldn't do when I joined the last project.
After talking it over with the CTO, I think we're headed toward a Linux server: possibly cross-platform so that each dev can choose either Linux or Windows, possibly with a managed VMWare image, and possibly on a big shared dev server. It's crucial, in my opinion, to have at least some of us running Linux if that's our production platform, and the consensus seems to be that Linux should be our production platform.
Now I'm looking at "what database system to use?" The biggest problem I've experienced, in dealing with ORM, is: how do you get an entire object hierarchy with a single query? Examples are "an area, all of the map data for this area, all of the NPCs on the map for this area, all of the items the NPCs on the map for this area are holding" or "a character, all of the stuff the character is holding, all of the stuff contained in the stuff the character is holding." If you store the top-level objects as binary blobs and unpack in your game code, this problem is solved.
Otherwise, you're in the classic problem of recursively descending a tree in a query (this is discussed in depth in The Art of SQL, and Vadim Tropashko has also written a broad view of the subject with his own proposal). The latest versions of the SQL standard offer a way to do this ("recursive with"), and some RDBMS's implement it, but it's never very fast and it always looks a little ugly1.
Another problem frequently faced with common object/relational maps is figuring out which tables the data is in - this is particularly a problem when your object model uses inheritance, so you've got classes that share some data, but don't share other data. Again, this problem is largely solved when storing objects as binary blobs. You can a) put all the information for all possible objects in a particular inheritance hierarchy in a single table, or b) put all the information for each class in its own table (often repeating columns across tables that share a common ancestor), or c) split it up so that each class's table has just its new properties, plus a pointer to its parent class's table. All of these approaches have problems.
Picking the right database system involves picking what solutions to use for these problems.
SQL Server is a possibility, we have fairly recent experience with it and it can definitely work. It also supports recursive with syntax, so that you can recursively descend a tree in a single query. SQL Server also has the advantage that there are a lot of very good pre-existing tools to remotely manage a SQL Server database above and beyond what Windows itself offers for remote management. So, managing SQL Server machines is not as bad as other kinds of Windows-based servers, but there are still gaps.
Versant is interesting because of how completely it solves object/relational mappings and recursive tree traversal, but on the other hand ad-hoc queries against a Versant database are... less than optimal. It involves writing quite a bit more code, and quite a bit less work is done in the database kernel - so it has to be done after retrieving potentially enormous result sets. In a lot of ways, Versant has many of the same pros and cons as storing binary blobs in the database, and in my mind represents the state of the art in binary blobs.
PostgreSQL is interesting because it provides table inheritance out of the box; I'm still playing around with ways of turning that into a solid implementation of approach c), but I'm not there yet. Recursive with is currently only proposed for PostgreSQL 8.4, although there are community-supplied stored procedures to implement (opaque to the planner/optimizer) Oracle-style "connect by."
I haven't looked past those three yet. Oracle is used by some in the game industry, but I'm a little wary of their cost and baseline performance. MySQL... I'll be frank, I haven't seriously considerid MySQL for any task in quite some time.
1. There's also the materialized path and nested set models of handling tree structures. Nested sets are horrendously expensive for any data change (many, many rows have to be updated when a leaf node is added to or removed from the tree). This is simply not acceptable for a tree structure such as a character with a frequently-changing inventory.
Materialized paths are less expensive than nested sets on updates (you only need to deal with sibling nodes), but still significantly more expensive than the adjacency model afforded by recursive with or connect by. I suspect that with a large enough number of separate trees (characters, maps, etc.) a materialized path will also start choking.
Now I'm looking at "what database system to use?" The biggest problem I've experienced, in dealing with ORM, is: how do you get an entire object hierarchy with a single query? Examples are "an area, all of the map data for this area, all of the NPCs on the map for this area, all of the items the NPCs on the map for this area are holding" or "a character, all of the stuff the character is holding, all of the stuff contained in the stuff the character is holding." If you store the top-level objects as binary blobs and unpack in your game code, this problem is solved.
Otherwise, you're in the classic problem of recursively descending a tree in a query (this is discussed in depth in The Art of SQL, and Vadim Tropashko has also written a broad view of the subject with his own proposal). The latest versions of the SQL standard offer a way to do this ("recursive with"), and some RDBMS's implement it, but it's never very fast and it always looks a little ugly1.
Another problem frequently faced with common object/relational maps is figuring out which tables the data is in - this is particularly a problem when your object model uses inheritance, so you've got classes that share some data, but don't share other data. Again, this problem is largely solved when storing objects as binary blobs. You can a) put all the information for all possible objects in a particular inheritance hierarchy in a single table, or b) put all the information for each class in its own table (often repeating columns across tables that share a common ancestor), or c) split it up so that each class's table has just its new properties, plus a pointer to its parent class's table. All of these approaches have problems.
Picking the right database system involves picking what solutions to use for these problems.
SQL Server is a possibility, we have fairly recent experience with it and it can definitely work. It also supports recursive with syntax, so that you can recursively descend a tree in a single query. SQL Server also has the advantage that there are a lot of very good pre-existing tools to remotely manage a SQL Server database above and beyond what Windows itself offers for remote management. So, managing SQL Server machines is not as bad as other kinds of Windows-based servers, but there are still gaps.
Versant is interesting because of how completely it solves object/relational mappings and recursive tree traversal, but on the other hand ad-hoc queries against a Versant database are... less than optimal. It involves writing quite a bit more code, and quite a bit less work is done in the database kernel - so it has to be done after retrieving potentially enormous result sets. In a lot of ways, Versant has many of the same pros and cons as storing binary blobs in the database, and in my mind represents the state of the art in binary blobs.
PostgreSQL is interesting because it provides table inheritance out of the box; I'm still playing around with ways of turning that into a solid implementation of approach c), but I'm not there yet. Recursive with is currently only proposed for PostgreSQL 8.4, although there are community-supplied stored procedures to implement (opaque to the planner/optimizer) Oracle-style "connect by."
I haven't looked past those three yet. Oracle is used by some in the game industry, but I'm a little wary of their cost and baseline performance. MySQL... I'll be frank, I haven't seriously considerid MySQL for any task in quite some time.
1. There's also the materialized path and nested set models of handling tree structures. Nested sets are horrendously expensive for any data change (many, many rows have to be updated when a leaf node is added to or removed from the tree). This is simply not acceptable for a tree structure such as a character with a frequently-changing inventory.
Materialized paths are less expensive than nested sets on updates (you only need to deal with sibling nodes), but still significantly more expensive than the adjacency model afforded by recursive with or connect by. I suspect that with a large enough number of separate trees (characters, maps, etc.) a materialized path will also start choking.
previous page
