How to choose the right software quality metric

Metrics run the world. Businesses need effective benchmarks to measure success, to see where changes need to be made, and to determine which projects have had the greatest impact. Within individual teams, too, metrics play a vital role in tracking success. 

But all too often, the metrics that teams think are important do not align with the wider business. There is a tendency to view our day-to-day work through the lens that is in front of us.

This means that one team might measure high activity as a success, but if this activity doesn't influence the wider goals of the business, it's not likely that its impact will be understood. "You don’t grow a business on increased activity. You don’t generate revenue by selling story points," said Melanie Ziegler, founder of the VPE Forum, a peer network for development execs. "You grow a business on outcomes."

With QA, this is no different. How you choose the metrics you measure in QA testing and performance engineering is extremely important. In fact, it is a crucial part of influencing how the business views the QA function, and it is key to developing a narrative that communicates the true value of quality.

The quality journey at HelloFresh

When writing my book, Leading Quality: How Great Leaders Deliver High-Quality Software and Accelerate Growth, I spoke to Ilya Sakharov, former director of quality assurance at HelloFresh, who told me a story that may resonate.

When he first joined the company, Sakharov noticed a disconnect between the quality team and the rest of the business. The quality team was developing test strategies that it believed to be most important but that did not align with the company's key metric: the number of recipe-box subscribers.

Recognizing that this disconnect was causing misalignment between the QA function and the wider business, Sakharov set to work communicating the importance of this key metric.

This renewed focus meant that the quality team started speaking in terms of the impact that poor quality could have on the number of subscribers. When it performed risk-based analysis on where to test, this metric was front of mind.

ITs key testing focus became areas that would have the biggest impact on this all-important metric. Knowing how it fit in with wider business goals and contributed to the success of HelloFresh overall gave the QA team a rush of motivation.

With the new alignment, cross-functional department relationships strengthened. What's more, there was a reduction in critical bugs making it to production.

Lessons from HelloFresh 

This example shows the relationship between metrics and quality.

Your QA team may believe that the number of bugs fixed is the key metric, with a high number of fixes being the goal. But that is a metric driven by an activity; it's not driven by an outcome. In fact, you might go as far as to say it’s a vanity metric—one that sounds great but doesn't mean much in the bigger picture.

Businesses need quantifiable results. They need to see quality improve and ROI increase as a result. They need to see lower customer churn and a higher number of downloads. Your activity could be sky high, but the wider company won't take notice unless there are tangible results.

The question that needs to be put to yourself and your team is: "What does it show?"

Which metrics will have the biggest impact on your company?

First, decide your metric. If you aren't lucky enough to have your business's key metric laid out for you, as Hello Fresh did, you'll need to define what that metric is.

If your company is in gaming or entertainment, along the lines of Fortnite or Netflix, your metric will most likely be the time users spend on the platform or app. These are attention-based metrics. If you are an e-commerce or subscription platform, transaction-based metrics will rule. This is all about reducing the friction of purchase and creating a smooth customer journey.

If you are a B2B platform, productivity metrics are going to be your focus. Focusing on user activity will enable you to see whether your customers are getting the most out of your platform and, in turn, whether they are likely to stick with it.

This silver-bullet metric is your growth metric. Whatever the business, identifying this key metric is the first step to understanding how QA can move the needle. You get to this by having conversations across departments and in-depth discussions with your team, or by simply asking senior leadership.

How QA teams can calculate their value

Communicating value is all about shifting the way you approach the work you do. Once your QA team members are all aligned on the growth metric, you can assess your current software testing process.

Ask yourself why you run certain tests, why certain bug fixes are prioritized, and why you use certain testing techniques. 

At Wizz Air, for example, the growth metric is the number of flights booked. When prioritizing bug fixes, then, the QA team at Wizz Air ranks potential fixes in terms of how they impact a customer's ability to book flights. If there is a critical bug at checkout, it goes straight to the top of the list.

A gaming company with a focus on hours spent playing would prioritize fixing a bug that stopped play after an hour, and a messaging app would prioritize fixing an app that impacted the number of messages sent. By refocusing the efforts of your QA team, you can ensure that the hard work you do impacts the bottom line.

Communicate the true value of quality

QA is a crucial part of the success of a business, and the QA team knows that. The hard part is communicating that value. Incorporating the right metrics into your testing strategy, and making them a vital part of everything you do, are key steps.

By realigning your team's focus around driving growth, you can deliver a quality product and a well-oiled software development function, too. Great cross-functional relationships lead to great products and a renewed sense of purpose.

In my keynote talk at this year's online EuroSTAR conference, I will be discussing the other steps you need to take to develop a quality narrative, drive growth within your business, and ensure your success in QA. The conference runs November 17-19, 2020.