Top 5 software quality metrics that matter now

How often do we hear development and testing organizations and even managers refer to lines of code written, scripts passed and executed, defects discovered, and test use cases as a measure of their commitment to software quality?

At the end of the day, these measurements are useless when it comes to delivering results that matter to your end users that keep them coming back for more of your products and services. Think about it. Who cares how many defects you’ve found in pre-production. What does that measure?

I want to make a fairly bold statement: These old stand-alone test metrics don’t matter anymore.

When it comes to quality in development, testing, and overall operations, these are the questions you should be asking yourself:

• How many stories have we committed to?
• How many of these were delivered with high quality to the end user?
• How much time did it take to deliver from business or customer concept to production?

Finally, ask yourself this: “Are our end users consuming the capabilities they asked for?”

Activities vs. results

The difference between this sort of focus and a purely technical focus is the difference between activities and results. If our team commits to 80 story points at the beginning of a four-week sprint but we deliver only 60, then we’re not meeting our commitment to ourselves, the business, or the customer. It also means our team is preventing another team from delivering on their commitments. The release will instead be put on hold and pushed to our next release. Ultimately, the business results are going to be less than what we promised.

Over the last several years, improvements in development and testing have provided an opportunity for organizations to apply new metrics that can lead to genuine transformation. The most common of these proven concepts is agile development practices. When executed well, agile methods can enable a team to quickly deliver high-quality software with a focus on the highest priority for the business and end user. As teams transform, having a few key measurements and producing results helps the organization evolve in an informed manner with continuous feedback from the investments they’re making.

Without these types of metrics, organizations will simply attempt their transformation blindly, with limited capacity to show results, including the business outcomes demanded of today’s technology organizations.

Top 5 software quality metrics

Here are the top five quality metrics that really matter:

1. Committed stories vs. delivered results meeting “doneness” criteria

Remember the last time someone committed to do something for you and either failed to deliver or didn’t meet your standards? It caused delays and rework, along with a lot of frustration. In software development, “stories” are the pieces of work that are committed to and, ideally, delivered on time and to a certain spec.

As you may know, stories represent the simple, high-level descriptions that form a use case, such as a user inserting a credit card into an airline kiosk. Each story needs to be delivered at a specific level of quality or “doneness” criteria. As teams continuously plan, elaborate, plan again, commit, and deliver, the ultimate goal should be to deliver these results in alignment with the broader team’s doneness criteria. When that can be measured, the team can showcase its abilities to meet its commitments on schedule and with the highest standards.

2. Quality across the life cycle

The demand for software delivery speed continues to increase along with the demand for reduction in costs. But how can you achieve these goals when you don’t have the time and resources to manually test every build? When you can’t afford to wait and find those defects in your late-stage releases? The answer is to follow the build life cycle from story to code on a developer desktop. Next, you should check, build, and unit test. Continue by using automation through the rest of the process, including automated, functional, performance, security, and other modes of testing. This gives teams the ability to show the quality of a build throughout the life cycle with quality metrics and automated pass/fail gates.

Given the frequency of these builds and automated tests, build-life results can be created and measured in seconds, minutes, and hours. Now, your most frequent tests are fully automated, and you’re only doing manual tests on the highest quality releases that make it through the automated life cycle. This results in automated build-life quality metrics that cover the full life cycle, enabling your team to deliver with speed and quality, while reducing costs through higher efficiency.

3. Production incidents over time and recurrence

Just as it’s important to show the quality of the release over time, it’s also important to minimize production incidents and their recurrence over subsequent releases. The table below illustrates a technique I’ve used to compare team performance over time. Imagine you are working with five teams over three completed releases.

The target for this typical (though imaginary) organization is 95 percent of committed stories delivered and zero production incidents. Teams that didn’t meet these goals are highlighted in bold red. Often, production incident numbers are found within an incident management process. Defining the root cause and implementing corrective measures enables continuous improvement and prevents recurrence of same issue in subsequent releases. With these quality metrics in place, you can learn which teams meet or don’t meet specific goals. Finally, you can look across teams and discover why proven concepts work.

4. User sentiment

Get to know your end users by measuring how they feel when interacting with an application or system. By capturing and dissecting the feedback they provide regarding new or improved capabilities, you can incorporate their needs into an upcoming sprint. At the very least, you can develop a plan to deliver something in response to those needs. On a larger scale, your analysis and incorporation of user sentiment can expand to a more general market sentiment, which can broaden your impact and market presence. Several components of quality can be covered via this metric, including simplicity, stability, usability, and brand value.

5. Continuous improvement

Following retrospectives, you should allow time and effort to implement prioritized continuous improvement stories. This enables the team to self-organize and be accountable for improving the quality of their process. By allocating this time and making it visible to all, the team and stakeholders can show their immediate impact. They can demonstrate how one team, compared to others, has delivered results at increased speed, with higher quality and value to the end user. This allows team leads to ask and possibly answer these questions: Are there certain practices that need to be shared? How do teams perform over time with certain changes injected? The continuous improvement metric can also justify recent or proposed investments in the team.

What really matters

I’m amazed and often taken aback when I hear that teams are still working the old fashioned way. In fact, I sympathize with them. I hear the same stories I heard 20+ years ago. Here’s one from last week: “I have 3,896 test cases, and I’m 30 percent complete on test execution.” I want ask, “So, what does that mean for time/quality/cost, along with on-time delivery to your end user?” It’s genuinely shocking when I ask a VP about their mobile testing process, only to learn that his or her mobile strategy is a “mobile guy” who does manual testing by putting the application on his phone and playing with it.

Let’s start focusing on metrics that really matter. We need to focus on results that are centered on the value we deliver to our end users and the quality results we can deliver to them. In the process, let’s not forget how to deliver. We need teams to contribute creatively and improve the practices they have, while measuring quality via metrics they can use to evaluate, modify, and improve processes over time.

What happens when we insist on the old style of quality metrics?

Well, for one thing, it helps explain why so many CIOs hold their positions for less than two years or why a third of them lose their jobs after a failed project. We’ve seen this before: A new CIO or senior leader comes in, fires a few mid-level managers, reorganizes a couple things, brings in a new partner, and suddenly they’re trying to measure results. Unfortunately, they don’t have the right metrics in place to show how the team is delivering. Command and control fails again. Sadly, this fails the business, shareholders, passionate individuals, and ultimately the end user: the customer.

You do not want to fail your customer.

Keep learning