Ultima Game Developer: Testing Requirements for Game Development

Created Worlds

Hardware and Software Requirements for Game Development

Part Two

I’m not a professional developer, but this is what I’ve found will work for independent game studios.

We can’t say, “I want this to run on anything made in the last year”, because that is far too encompassing: from all-in-ones from Walmart to custom Alienware XPS. As a developer, you need to pick a type of platform and range: “mid-range Windows-based gaming computers made in the last year”. That will provide a minimum range of memory, processor, video and audio profiles which you can compare against. Note that you do NOT need to have all of these available to you for testing, but you should be sure that you get hardware profiles from alpha and beta testers so that you can say you work on this range of platforms!

What do you need?

Target hardware within that range, obviously. It would be useful to have one or more at the “low end” of what you want to support. These will help test to make sure you don’t encounter game-ending problems due to hardware. Additionally, one “middle of the road” system, and one “target” system are ideal. I generally aim my target system as something that has been released in the last 6 months, and compares favorably with what is coming out today. (If nothing else, it let’s me update my computer regularly.)

Think about non-game things before beginning tests: if you are distributing online only, you can assume that the user is online while playing. If you are not creating an online-only play mode (social, MMO, etc) however, that is not a valid assumption. Additionally, you cannot assume that each user is running the game with no other applications running. In some cases, I will start a large, streaming movie in a browser and run it behind the game, to eat up memory, processing power, and network bandwidth while testing. If you are creating an online-only game, don’t install the entire game to a test unit; install only the “end-user” files and stream the content from another computer, to emulate the client-server dynamic.

This works with non-PCs as well; if you are developing an Android or iPad app, you’ll need devices you want to support. You’ll need to set up a server to pull the data from, and use a network connection to emulate the data transfer. Emulate load on your network by transferring a bunch of files or streaming items across the same network.

Should you program on your spec computer?

No, your testing computer is for testing. Your development box should be separate, on a different network, and with lots of redundancy. If you are serious about development, everything should be in source control, and your development platform should be backed up regularly. As in, nightly at a minimum. I check everything into source control after each session, which can be 2-4 hours worth of work, normally just before I build.

For testing, I like to keep a DVD image of the computer from just before installing the program, so I can wipe the system and restore it to a “live” state: this consists of a lot of extra programs installed that will interfere with a clean test, like WeatherBug, uninstalled Windows Updates, huge numbers of files in the Recycle bin, known Spyware, etc. Since these computers will never see the light of day, and are not connected to the Internet, and will be wiped tomorrow, thats fine. I also have a server, which is just a glorified PC that runs a server OS, and emulates the Internet by running a DNS server that forwards everything except calls for my game to a “chum bucket” where it logs and returns fail messages after eating bandwidth. This makes it easy to see if something gets called incorrectly, since I can check the chum log.

How should you test?

As a developer, you are not testing to see if your game can be played. You are first testing to see if your game can be launched. You do load testing. You make sure your game systems function. Your testers do everything else, while you develop more content and fix the issues they find. The more you perform testing, the longer development takes.

As an alpha tester, you are in charge of finding everything that doesn’t work in the game, works but not as expected, works but could be better, and things that weren’t ever expected to work but you feel should. That’s a tall order. Fortunately, you will be focused on small groups of tests at once. When I assign testing, its generally fairly specific: in map “X”, interact with NPC “Y” in every possible way and see if what you can break. You will submit everything from conversation lines with misspelled words, to the fact that when you kill NPC “Y” with weapon “Z”, his death animation isn’t correct. Once they are ready, you will test entire branches of quests, from a pre-loaded point. Alpha testing is among the most difficult and rewarding experiences during game development.

Beta testers, on the other hand, are generally much more self-focused. While you can (and should!) request beta testers provide feedback about specific items, it is much more common to simply let them take a crack at whatever they get their hands on. By the time beta starts, the systems and specifics should be ironed out and there should be no game-stopping problems. If there are, note those as known issues and not to report them until they are marked resolved. As a beta tester, you should try to do as many things as you can think of. Follow quests, and then do them again in different ways, different orders, try to break the quest by killing a required NPC, try to use the wrong item, etc. Anything you submit should be as specific as possible.

One thing that beta testers frequently misunderstand, especially if a public beta is allowed, is that your feedback is NOT to change the game. You are in charge of testing the nearly-complete product. Large changes are NOT likely at this point. While it is not unheard of for a development team to pull something from beta and revamp it before release, that is rare. For those taking feedback from beta testers, you will quickly get used to reviewing feedback “with a grain of salt”.

If there are changes after beta, but before release, almost all testing will be done by the production team itself. This can either be the developers, or be returned to the alpha test team, depending on availability. During this time period, programmers, scripters and developers are likely to be extremely busy, so retaining the alpha testers until this point can be highly advantageous.

Of course, after release you will likely receive feedback from your users. Depending on your audience, this can range from good-natured e-mails, to scathing posts on your forums, to death threats. Some audiences are fickle. But reasonable requests can lead to patches, or versioning, of the game. Patches should NOT be used to change content, which is what releases are for, but can be used to “tweak” things which made it through testing, such as a particularly difficult fight, or a place where the player can get stuck. These are good patches.

If you are planning to add new content after release, then you are planning an Expansion (even if it is distributed as a patch). In that case, the expansion needs to go through the test process again, or you are liable to end up with serious problems that then require more patches, along with a loss of players.

So now you are ready to start testing your game. How’s it going?