Welcome Back To Our Unit Trust Discussion!

Last week, we started getting into the nitty gritty of my Unit Trust (UCRT) system and how converting certain aspects of a bug into a numerical system will allow for better application health visualisation. We explored bug Scores (based on an exponentially weighted Impact), Factors (like Severity, Status and Time in Status) and touched on Warm-up/Cooldown periods.

This week, we’re getting into the UCRT Ranks!

The Trust Climb: From Untrusted to Trusted (One Bug at a Time)

Remember the original goal of this system? Convert all the juice data into a visual that’s easy to interpret. This is where the Ranks come in – Untrusted, Credible, Reliable and Trusted, the namesake of the UCRT system. At this point in our process, we have our bugs converted to numerical values and now we’ve got our Ranking system to plug them in to.

But after a bit of prototyping this, with the bug scores plugged directly into the Ranks, I found it to be a bit erratic. Ranks would change quite dramatically with influxes of bugs. On top of that, we don’t have all of the data we set out to include, such as “How often a unit has been tested?”.

Adding in that information helps to provide a “buffer” and we can increase the efficacy of the buffer by adding intermediary levels between the Ranks.

If You Add Buffers Between The Ranks, Are The Ranks Now Buffy?

After a bit of trial and error, I ended up with a system where Untrusted, Credible, Reliable and Trusted is divided into 12 Levels. (Remember this, we now have UCRT Ranks and UCRT Levels). This means that as a minimum, in order for a unit/feature to go from Untrusted to Credible, it would need to be tested 3 times and have no bugs found in it.

And you guessed it, for a unit to go from Untrusted to Trusted, the fastest way to get there is for it to be tested 12 times without issue. Each level is separated by a points value that is inline with your Bug Scores. I found that having 1 level equal to 10 Trivial bugs works quite well.

Need a little visual? Here’s our example UCRT Level thresholds:

🔹Untrusted Lvl 3 = 111 Points
🔹Untrusted Lvl 2 = 101 Points
🔹Untrusted = 91 Points

🔹Credible Lvl 3 = 81 Points
🔹Credible Lvl 2 = 71 Points
🔹Credible = 61 Points

🔹Reliable Lvl 3 = 51 Points
🔹Reliable Lvl 2 = 41 Points
🔹Reliable = 31 Points

🔹Trusted Lvl 3 = 21 Points
🔹Trusted Lvl 2 = 11 Points
🔹Trusted = 1 Points

Imagine we get a new feature in the build, it starts off Untrusted and at 120 Points (Everything starts off Untrusted!).
It passes two rounds of testing, UCRT Level reduced by 20 points! We’re now at 100, still in Untrusted Territory.
In the third round of testing, a handful of B Class Trivial bugs are discovered.

Now, because bugs were found, our UCRT Level will stay at 100 and those Trivial bugs add 5 points to the total (105). So the feature stays where it is in Untrusted with a little “drag” on it. It wont be moving up the Ranks so quickly now.

Fourth round of testing, no new bugs discovered! We reduce our UCRT level by 10 points again and the feature could have been upgraded to Credible but because of that handful of bugs, even if they’re all fixed in the next round of testing, we stay just about in Untrusted at a score of 92.5.

That Bug Score going from 5 to 2.5 is where you can see the effect of the cooldown period I first mentioned last week. In this example, the cooldown factor is a exponential decay of 50%.

Wrapping Up

Hopefully, that scenario helps to explain how the Levels add a buffer to the Ranks and you can now visualise how finding 3 or 4 Showstopper bugs early on in development can really put a drag on a Units ability to develop Trust early on. You may also have wondered about negative numbers, well, I have this to say; Trusted stops at Zero! That is to say, you can’t earn a bank of Trust that a few Major bugs can’t erode but even at Trusted, one Showstopper and a couple Majors won’t knock the Unit down lower than Reliable. That’s the buffer we wanted.

You may also have realised that there’s no upper limit to that total Bug Score + UCRT Level. Lots of serious bugs can make deep holes that are hard to climb out of and the only out of it is more bug free rounds of testing!

---

It’s Slowly Coming Together

Right, we’ve got most of the data we first mentioned in Pt. 1 factored into our analysis. We’ve got a good overview of how the system processes the data and a target output.

Now, I don’t know if you noticed but there’s a fair bit of calculating going on here! Every Test Round, a calculation of every bug in every feature is performed to calculate the Bug Score and UCRT Level which are then combined to give us our UCRT Rank.

The end result is nice and simple, par exemple: “Yo, that new plushie collectable we added, it’s still Untrusted 6 tests in” but who’s doing all the maths?!

AUTOMATIONNN!

And that’s what we’ll start looking at next week.

What’s the point in a beautiful system if it takes more effort to maintain than the information is worth?