Making AAA games is hard

In June 2010, as the world waited for news on the release of Portal2, Valve announced that making games is hard.  Everyone inside the industry knows this, but to gamers and journalists alike this seemed to come as a surprise. Partly because of disappointment that a game they were looking forward to wasn’t coming out for a bit longer, and partly because Valve is an industry leader. For them to put their hands up and say “hey, this ain’t so easy” made people think: “if they find it hard, what about everybody else?”.

Well, yup, everybody else finds it hard as well. But why? Surely we just take the last game we made and stick some new levels and graphics in it? Or just sit around all day playing the game until one day we decide to release it?

Unfortunately, it’s not like that at all. But why not?

Everything you see has been made by me.

The human race has evolved so much because of two things: opposable thumbs and imagination. We started by imagining tools, and since then developed houses, cars and gucci handbags.  And, more relevantly to this post, computers.  Upon which we can play video games.

Computers don’t have imagination.  All they know is 1′s and 0′s, and they only do what they’re told to do.  So, as the creators of a video game, we have to not only imagine the game we want to make, but imagine it being played, and all the possible ways it could be played, and then make that.

This isn’t a new skill by stretch: it’s how everything in we’ve made around us takes shape, from houses to films.  I’ve only worked in properly in the games industry (unless you count three years bar work at University), and the biggest initial hurdle is discovering that absolutely everything in a game requires so much detail and work.  Take, for instance, putting a character on screen.  Not only does a character need modelling, texturing and animating, but a script must be written and dialogue lines recorded.  Code has to be written to get all those elements working, and then there’s all the extra layers on top: getting eyes to work in a realistic fashion can take days, making hair look like hair weeks and facial expressions to look realistic months.

A lot of the challenges faced making video games are no doubt found in pretty much all industries, and there is a lot of cross over with any software design.  In addition to “having to make everything”  there are four things that really stand out as problematic when making a piece of software: its scope, the systems it is on, the tools it is made with and, possibly trickiest of all, how fragile the finished product usually is.

Scope

Scope is the broad size of a project, no matter what that project might be. If you want to build a house you get a plot of land and work out how big a house you want to build on it. You decide how many bedrooms, bathrooms and how big the kitchen should be. There’s lots of things to be ironed out and things will change as it’s built, but the scope will remain roughly the same as when the project started.

Video games are very difficult to scope because we have an infinite virtual space in which to build our world.  And it’s surprisingly difficult to build something that’s finite in size when you’ve got infinity to play with, and so a great deal of time is spent working out where the boundaries are.

The biggest problem we then face regarding scope is that it is always changing, and often far later in the day than most people on a team would like. This is true of most things I’m sure, but the scale at which it happens with video games is alarming.

Sticking with the building a house analogy: imagine the one you’ve had built is nearly finished.  At the start you decided on a single garage enclosed within the main shape of the house, with a loose gravel driveway.  All that’s left to do is paint the front door and install the phone line and then you can move in.

At this point your mother turns up and tells you that where the garage is there should be a bathroom, and you need a double garage stuck on the side of the house with a tarmac double driveway up to it. You don’t agree but she bulldozes through your procrastinations and tells you she still expects you to move in on the same day as planned (she’s fed up with you living at home). Somehow the builders agree to work through the nights and the day before you get the keys it’s all done.

Your mother comes up to the house with you, takes one look and says “actually, you were right. Put it back to how it was.”

This happens in games all the time. Partly it’s due to the nature of the beast, as making a game has to be an iterative process: you should rapidly build, play, test, refine and repeat. But it’s also to do with a lack of correct hierarchy and procedure. On a building site the project manager has the ultimate say, on a film it’s the director, in a kitchen the head chef.

Having said that, a single person with ultimate power rarely seems to work on large scale games and so we don’t really have a solution for that yet, but as understanding of making games improves I’m sure we will in time.  This may end up just being an acceptance of the fact that things do change, and have to change, to make a top quality game and making sure that schedules reflect that.

Hardware

One of the most amazing things about computers is how quickly they evolve. This is a double edged sword, because it keeps opening up new opportunities for us to strive towards realism: better graphics, sound, physic and so on.  I’ll leave off discussing if any of these things actually make games better for now, and simply highlight that the flip side is that the ground beneath our feet is constantly changing. For instance, the PlayStation3 is a completely different to the PlayStation2 to code for.  Which in turn was very different from the original PlayStation.  Sure, the PS3 will run some of the code that got written for PS2. But it won’t be optimised for the PS3 and so it needs to be rewritten.

The life cycle of a console has historically been about 5 years, though the current generation is looking more like lasting between 8 to 10.  The PC is constantly evolving and keeping up with the latest graphics cards takes a lot of gambling as to what will be common, yet impressive, when your game is released.  As a project matures, and code bases stabilise, programmers learn more about the hardware they’re working on and optimise or rewrite code to suit.  The limitations are better understood and the art team can start experimenting with clever ways of hiding them so, by the end of a console’s life cycle, the games will look and run better than they did when it first came out. And then a new console comes out and we have to learn it all again.

The landscape of gaming is changing a lot right now, and games on platforms like facebook, iOS (Apple) and Android have opened up a new demographic of gamers, and also games. While specific code is still required for these platforms it’s not always required to squeeze every last bit of juice out of them, making the focus more on the game itself rather than the candy surrounding it.  Maybe one day we’ll get to the same point on the big consoles as well…

Tools of the trade

The biggest knock on effect of the hardware changing so quickly is that as well as striving to catch up with the system we’re developing the game for, we’re also writing the tools that we make the game with.  As games are developed there’s always a lot of rewriting of code, and in fact it’s worrying common for a completely new engine to be started.  Starting from scratch always seems like an exciting prospect: the chance to begin afresh, a clean slate upon which to get everything right from the off.

This is common across all software development, and in games it causes huge problems because without an engine to run the game on, you have no game.  Writing a new engine for a console these days takes a very long time: I’d say a minimum of 10 man years just to get something usable up and running, and a lot of the time all that happens is a re-inventing of the wheel.

Once the engine is up and running it’s then time to develop the tools properly, but by this stage the game is usually behind so it’s all hands on deck to just cram things in as fast as possible and we end up effectively trying to dig a big hole with a spoon.  By the end of development everyone acknowledges what a mess it all was, and so the decision is made to start again from scratch.  This time we’ll learn!

Alongside the fun with the internal toolset are all the pieces of external software used as well. Code tools for writing C++ are pretty robust these days, and the fundamentals between Max, Maya and Photoshop haven’t changed a great deal recently either.

But for making high end assets for games the software is still in a great deal of flux, and artists have to constantly keep their knowledge up-to-date of the latest, and greatest, piece of software. A few years ago Silo was cutting edge, now that’s old hat. Z-Brush is great but confusing, Mudbox funky but buggy.  Animators and sound guys have a similar minefield to navigate. There’s a lot to learn and keep learning, and not much time to do it.

So how to fix these problems?  Early on, ensure that your team has sufficient time allocated for keeping up to date with the latest software available for asset creation.  About half way through the project, the external software that is used should be locked down so you don’t have to spend lots of time updating in-house file formats and exporters.  For the internal engine used, make sure that there are a couple of coders fully dedicated to the tools side, and that there’s a technical designer who is dedicated to designing the tools to meet the team’s demands.  These steps won’t solve all the issues by any stretch, but they should go some way to helping.

House of cards

Nearly there!

One of the biggest surprises when I started in the industry was how fragile a game actually is.  I think the public is starting to realise this as it’s now very common now for a big release to have one or two mandatory patches applied to fix issues that got missed towards the end of development.

The thing that isn’t understood is why this happens.

Well, like a lot of things, games are complicated.  Modern day engines have millions of lines of code whizzing away in the background, churning through gigabytes of data to present you, the player, with one hell of an experience.  But there’s a lot of proprietary software running behind the scenes as well.  Consoles have always had an operating system of sorts, but now you have a fully blown system running away all the time.

Games have to interface with these systems, especially when you’re doing anything online (which is pretty much mandatory for a big release today).  While it’s possible to test the game on a small scale, as soon as you get thousands of people all trying it at once bugs in your code, bugs in the system code, bugs in the interfacing between code, bugs in the telephone system… it all adds up and needs to be found and ironed out.

During development there’s a constant, silent, tug-of-war going on in the background.  Everything is so intertwined that something that worked yesterday might suddenly not today, even if it hasn’t been directly touched.  One bad move from someone and, completely by accident, an entire game can stop working.  Delays take place while that’s fixed, and then you’re onto the next day.

In an ideal world we’d ship games without any bugs and everyone would be happy.  But we don’t currently have the systems in place to allow for that to happen: be in testing in large scale or robust enough code bases.  It might not even be possible, because the number of permutations makes it physically impossible to test ahead of time.  Scheduling, sensible scoping, good tools and an experienced team can help a lot with keeping the game on track, but even if you do release a bug free game: will it be any fun?

Conclusion

I don’t want this to come across as a big negative rant, and I hope it hasn’t.  Yes, making games is hard.  Just as lots of things in life are hard.  We have areas where we can improve, where we can learn from the past and try not to make the same mistakes.

The great thing about the industry is how rapidly it changes, how every day there’s new challenges to overcome.  The thrill of getting it right, of seeing someone happy because of something you made.

Escher render from here (made by Daniel Martinez Lara and Kike Oliva).
Turing Bombe picture from here
House of cards from here.