“What we’ve seen, heard, felt… anger, joy, and sorrow… these are the things I will pass on. That is what I live for.” – Solid Snake

Welcome back to QA Basics in game development!

This week we’re looking at Impact and Severity and this one could get controversial! I’ve seen these approached in different ways in different places, I’ve seen places use only either Impact or Severity plus Priority. I feel like it’s bug classification chaos out there! Before we get into Impact and Severity, I’m making this clear first, Priority is for Producers! Yes, there may be certain bugs that trigger a specific Priority but in general, it’s Prod’s territory.

Now that’s out of the way I’ll set the scene, I’ll point out the key difference between my Impact and Severity:

Impact – A specific event defines the level of impact.
Severity – An assessment of various factors define the level of severity.

Keeping that in mind, lets see how these two key attributes, in conjunction with the Summary, help save a Producer or Lead a big chunk of time when assigning a bug to the right team member.

---

Impact

Expanding on “A specific event defines the level of impact”, what I mean by that is a specific defect will always have the same Impact rank. I can just hear experienced QA peeps sucking air through their teeth now but hear me out. I come from a compliance background and pretty much any issue we cared about had a predetermined class, there was very little leeway. This approach gives you a very solid foundation on which to document defects in the non-compliance game development world. If you’ve got an army of testers looking for a reference with which to rank a bug against, you need a document they can refer to that makes this quick and easy. No, “hmm it’s not that bad” or “other people may not mind but I want this changed!”, Impact is “Did A happen? Then X category. Did B happen? Then Y category.”

I can hear them again, those QA peeps, “You can’t categorise everything!”. Oh, but you can. You can start off small with some simple rules and build from there. The data you can collect in this manner will retain a QA team’s knowledge no matter who comes and goes and it could even help influence design down the road. If you’re not convinced yet, let me introduce my categories to help me fight this fight!

*In the voice of Michael Buffer*

“In the Impact corner we have Showstopper, Critical, Major, Minor and Trivial!!”

(I’m way too hype about writing about this.)

Hopefully, those categories look very recognisable to most QA peeps and even if you don’t recognise them, they should make sense. To help illustrate what I mean by an event defines the Impact, I will give some examples for each category:

SHowstopper

🔹Mission Critical Engine Failure (Crash/Hang)
🔹Save data loss
🔹User cannot progress main objective under any circumstance
🔹50% drop in FPS for 3 seconds.

Critical

🔹Missing operational instruction
🔹Non-Mission Critical Engine Failure (Crash/Hang)
🔹User cannot progress main objective unless x, y, z.
🔹40% drop in FPS for 3 seconds.

Major

🔹Misplaced intrusive asset that hinders gameplay.
🔹Mission Critical materials issue.
🔹Mission Critical SFX issue.
🔹20% drop in FPS for 3 seconds.

Minor

🔹Spelling or grammar issues within operational instructions.
🔹Misplaced, non-intrusive, asset.
🔹Non-mission Critical materials issue.
🔹Non-mission Critical SFX issue.

Trivial

🔹Spelling or grammar issues not within operational instructions.
🔹Non-Mission Critical clipping.

As you can see, they’re specific and broad, like a set of requirements. What is “Mission Critical” is determined in the GDD and that means that these categories will be the same across any and all projects QA work on so they know exactly how to rank the Impact of an issue. Just keep in mind that if you adopt this approach, as your categories expand, try your best ensure that they do not contradict or overlap and if you manage to pull that off, you should get a job writing requirements!!

Did I win anyone over? If you still have your doubts, continue to hear me out as up next, we have Severity and the fight will really begin.

---

Severity

OK, take a breather, that was a good chunk of info to ingest and hopefully Severity will start to tie up any loose ends for those that had them.

For me, Severity is an assessment of various factors and I use it to compliment Impact with the goal we have had in mind since the beginning, helping a Producer or Lead save time when assigning a bug.

This group of factors are things like:
🔹Frequency
🔹Quantity of Reproduction Steps
🔹Timing of Reproduction Steps
🔹Does it contravene the GDD?
🔹Will it cause a platform certification failure?
🔹Does it block development?
🔹Does it make the game less fun?
🔹How difficult will it be to fix?

Some of those are easy to assess, some may require discussion with other departments, some are subjective but all are super valuable information. As mentioned earlier, Impact serves as a foundation for knowledge retention. The thought that goes into assessing Severity, how you came to your conclusion during the process, is really helpful information and you can store that decision in a document organised by Impact. To help explain that a bit more let me introduce my Severity categories: Must Fix (MF), A class, B class, C class and U class.

MF to U is a scale of “this Must be Fixed” to “nice to fix” and I’m sure you could rank the Severity of some issues without a second thought but here’s some examples:

Must Fix

🔹Certification failure
🔹Blocks development.
🔹Non-Mission Critical Engine Failure (Crash/Hang) occurs 100% by restarting mission twice.

A Class

🔹Mission Critical Engine Failure (Crash/Hang) occurs <20% of the time.
🔹Mission Critical NPC has incorrect materials, making them difficult to recognise.

B Class

🔹Spelling issue within operational instructions which results in user heading to wrong location.
🔹Save data can be lost if the user deletes their save just before an auto-save event is triggered.

C Class

🔹Mission Critical SFX missing in first instance but plays for all other instances.
🔹Misplaced intrusive asset that hinders intended gameplay but is actually kind of fun to try and avoid.

U Class

🔹50% drop in FPS for 3 seconds in a very hard to get to location at the far end of the map.

It’s those areas below, A through to U, that require the most thought and they contain the most valuable information.

Picture this in your QA documentation under the “Critical – User cannot progress main objective unless x, y, z.” section, “Example bug was ranked C class because although the user has to talk to a specific NPC at dusk to trigger the next objective. Although the GDD stated that the objective should have continued without the interaction, the QA team believe this interaction gave life to the area and made the scene more fun!”

That right there is the gold you’re panning for in all this information gathering that is known as testing. Don’t lose it. Categorise it and use it in future projects. It’s one of the reasons why use this dual ranking of Impact plus Severity, the other being that all important support for the Production team. They can read the Summary, Impact and Severity, set a Priority and assign out to the team in a matter of seconds. Those three key attributes really boil down the most important aspects of a bug into actionable information which is important because you never know when it’s going to get busy so have a bug format that is ready for it when it comes.

---

Final Thoughts

As I said at the beginning, I’ve seen so many ways of ranking and categorising bugs and I think that has put me in the fortunate position of being able to chose and create the systems I think will be the simplest and fastest to use while also retaining juicy data.

Next week, we’re going to get stuck into a bug’s Description section.

Again, please feel free to message me or comment about your thoughts. I’d love to hear how this one went down with the QA peeps!