Game Testers – they’re cheap, get more!

There’s an elephant in the room in the games industry, and it’s QA.

QA isn’t the problem; it’s the symptom and a solid indicator that the games industry has a lot of growing up to do. Why? Our development process is broken. Badly.

QA is the bottom rung of the game development ladder, and is seen as an entry point for the hordes of avid gamers that want to get a foot in the door of game development. As such, it’s at the bottom of the pay scale, and QA is often treated as a necessary side trip on the road to producing a triple-A title. QA teams in large studios are often kept at arms length from the development team, and typically not brought in until pre-Alpha planning and submission deadlines are looming.

In my first producer role in the games industry 11 years ago, I received pushback when I pointed out that we hadn’t hired any QA in pre-production. I was summarily informed that QA is only needed in this industry as we approached Alpha.

Coming from a business software development background, this was surprising. Overriding my recommendation to hire professional software-dev QA, management opted for a tried-and-true formula of hourly wage gameplay testers. Did it work? Yes. Did it work at optimal efficiency? No. Why? People were working overtime to find and report bugs, then get them fixed.

The same view was the de rigeur standard in larger studios. My next gig was managing several game aspects of a development team for a triple-A title in a major studio. On the larger teams, we had up to five development directors, each responsible for the delivery of multiple game areas. I volunteered to work with QA, as I wanted to get an idea of what was going on in Testville.

What I found was that QA was indeed the very, very bottom rung of the studio ladder. As I was doing my due diligence on equipment, the QA team leads indicated that they didn’t get adequate numbers of console test kits or have their basic desktop systems repaired in a timely fashion by IT, because, well, they’re QA. They were also discouraged from spending time on the dev floor, and kept in a building across town.

I came from a Windows application and database development background, where QA is on par with all aspects of software engineering, including pay, for the sole reason of its importance in the development cycle.

Why isn’t this the case in game development? I can only surmise it’s due to old-school thinking from the wild west game development days: “We don’t need a process; we’re making money hand over fist so we’re doing things just fine, thank you.” Actually, you’re losing more money than you know – more about that in a bit.

Our industry development processes are where the software development industry was 30 years ago, before the non-profit Software Engineering Institute came up with the Capability Maturity Model. Game development is, for the most part, at CMM Level 1, known as Initial: Chaotic, ad hoc, and projects are completed through individual heroics. If you think that’s sustainable, look at how few triple-A games are out there compared to the old days.

I’ve heard the repeated mantra for justifying lack of process and questionable management approaches to QA: “We don’t make software, we’re more like making movies.” There is only one answer to that: No.

We make entertainment software, which is infinitely *more* complex than traditional software development due to the player changing the outcome through gameplay, not to mention online interaction.

What we need is an honest look at what is broken in the development processes, or lack thereof, in our industry, and a commitment to bring up the level of professionalism. There is no excuse for the 80-hour work week. The only reason for overtime is simple: bad planning.

Scrutiny of development processes also requires a re-vamp not only of how QA is staffed and managed, by hiring QA engineers right out of the software development field – engineers on par with any other engineer on the team – and integrating them with the development team from the first day of pre-production. If we did this, many of the testing requirements for our titles could be automated, and QA team sizes significantly smaller. This is perfect-world thinking, of course; feel free to disagree with me on this.

One of the well-established practises in traditional software development is to have QA test the requirements and initial design documents of your system, before a single line of code is written. Why? The 80:1 return on investment. When I was working as a consultant to optimize the development processes of software companies, the cost differential between fixing a bug in design (i.e. on paper) versus fixing it in production software was 80:1. To put that in games industry terms, QA finding and fixing a bug in your GDD will cost you a buck. Fixing this at Beta will cost you 80 dollars. This is a 25-year old paradigm; more recently I’ve seen project managers put that at 120:1.

The big picture is that we need to rethink how we manage QA in game development. QA is at the tail end of our dev cycles, when it should be at the forefront of the next triple-A kickoff meeting. I don’t see the logic of trusting the last stop before a $20 million title hits the customer to entry-level employees. Empower them, train them, and treat them like software development professionals. They’re a highly motivated and ambitious employee group.