Key software quality metrics and how to measure them (even when it seems a daunting process)

Download the whole article here

When it comes to measuring software quality metrics, the main challenge for engineering teams is that there is no correct, one size-fits-all approach to doing things. As experience has shown us, it is often impossible for industry professionals to agree on the metrics they use, so identifying the key performance indicators may easily turn into a daunting task which many avoid.

Still, there comes a point when you just need to take things to the next level and make software quality measuring a priority. This usually happens when the projects’ scopes, as well as their issues increase in complexity. In such a context, if you want to be a successful manager, you have to identify the relevant metrics, assess them constantly, compare the results and, of course, set qualitative and quantitative objectives accordingly. It is only by mastering the data you have that you may increase your team’s chances of being efficient and delivering results. 

In fact, this is a strong premise, at Waydev. What we do is that we provide information on each team and project, so that you make operational decisions based on evidence, facts and figures, instead of gut feelings. Such an approach puts significantly less pressure on managers and enables them to have a clear view of what is happening and what comes next. No matter what kind of software quality measuring methodology you choose, Waydev offers you access to the data behind, enabling you to increase your chances of delivering outstanding work. 

To offer you in-depth information on how to use software quality metrics for operational optimization, without disrupting your teams’ flows, we will walk you through the software quality model developed by Consortium for Information and Software Quality (CISQ), which includes elements like reliability, performance, security, maintainability and code quality, as well as through other important indicators, such as usability, performance, correctness. 

Why software development quality matters

Software quality metrics impact the company as a whole, but, most importantly, engineering departments, which are held accountable for the quality of the software they deliver. This is why, as a successful manager, you need to constantly pay attention to KPIs, as well as use them to understand software performance and plan work. 

A 2017 study quoted by Deloitte research points out that over half of software development projects are delivered late and require additional budgets, while 20% end up canceled. Moreover, CISQ showcases that, in 2020, in the US alone, companies lost over $2 trillion, because of poor quality software. 

Maybe one of the most popular examples in this concern is British Airways’ technical issues, from the summer of 2019. At that time, the UK’s airline company experienced an IT system problem that led to hundreds of flights being delayed and dozens getting canceled. This impacted everyone, from British Airways, to the airports that had to deal with the congestion and overall operational failures, as well as to the thousands of unsatisfied passengers which lost trust in the brand. 

In this context, software development optimization is translated into strong teams that function based on standardized operational structures, which enable engineering managers to correctly assess the quality of their software, as well as identify improvement opportunities that impact customer satisfaction and efficiency. 

What you get when you monitor software quality

As an engineering manager, you want your team to deliver great work that benefits from high client satisfaction, is cost-effective and contributes to the company’s bottom line. To turn these goals into everyday realities, you need to monitor the quality of the software you deliver. The better you do it, the better the results. 

Here’s a list of things you get when you monitor software quality the right way: 

• It’s significantly more efficient to manage and assess each project, identify the areas that need improvement and prioritize problems. When troubleshooting happens at an early stage of the project, issues are easier to solve and cost less. 
• You increase performance and efficiency, by reducing employee working hours, overtime and the associated resources.
• You enhance communications and collaboration between team members, offering them KPIs for their workflows. 
• You optimize return on investment (ROI), by maximizing the results of each project and lowering its costs. 
• You have objective methods of assessing team performance, thus encouraging fair, transparent evaluation criteria which keep professionals satisfied. 

Elements of software quality metrics

According to the software quality model developed by CISQ, software quality may be broken down into the following elements:

1. Reliability 

which emphasizes the software’s stability and risk of failure, in certain conditions. This is measured in Lead Time to Changes (LTTC), Deployment Frequency (DF), Mean Time To Recovery (MTTR), Change Failure Rate (CFR). 

2. Performance 

which showcases to what extent the software delivers its specific purpose, by taking into account the code’s efficiency level, the infrastructure’s optimization, the loading speed of different features and pages, as well as whether the system is capable of scaling. These are measured through load testing, stress testing, or Application Performance Monitoring (APM). 

3. Security 

which assesses how likely it is for attackers to breach systems, interrupt the activity or get access to sensitive information, such as customer data. In today’s complex, unsafe world, governed by cybersecurity concerns, this is particularly important, as it translates into the number of vulnerabilities, number of security incidents, as well as their severity, how long it takes to solve them etc. 

4. Maintainability 

which focuses on how easy it is for systems to debug and troubleshoot, as well as maintain functionalities and upgrade to new ones. The metrics that enable engineering managers to evaluate these are Cycle Time, performance in changing environments. 

5. Code quality  

which assesses the quality of the code, focusing on whether it is free of bugs and semantically correct. This includes quantitative metrics, such as code complexity and lines of code (LoC), and qualitative ones – code clarity, efficiency etc. 

Moreover, there are 3 other areas that you should take into account, as an engineering manager:

6. Usability

which focuses on the UX, how intuitive the software is and how pleased end-users are with it. 

7. Correctness 

which assesses how the system functions, if it delivers the expected results and whether it satisfies the users. 

8. Integrity 

which identifies whether the software may be integrated safely, with other platforms and softwares, to develop its functionalities.

What metrics to measure to improve software quality 

At this point, you surely agree that measuring software quality is vital for improving a project’s chances for success and client satisfaction levels, so the question that arises now is what metrics matter most for your work. 

At Waydev, we understand the struggle. Measuring a team’s performance is extremely difficult, as you might feel like you are reducing its work to key stats and observations. Especially when professionals are working on complex aspects, some indicators like lines of code or bug rates don’t do justice to the whole picture and are certainly not relevant for how well or poorly your team is doing. 

Some engineering managers would go as far as to say that actually measuring software quality is impossible. Indeed, while that is quite fair, you can, however, monitor the aspects that impact performance. There is much value in this practice, as it is what turns projects into best practices examples. 

The important aspect here, though, is to take into account the particularities of each team and project, and filter them, to create the best approach for you. Starting from these variables, you may compile a list of software quality metrics that are specifically tailored for your work and which will help you take it to the next level, without creating spikes in resource allocation and costs. 

Types of software quality metrics

Agile metrics

The Agile framework looks at software development quality metrics as part of a bigger picture, of a strategy that takes into account 3 key principles which revolve around the team and its goals. 

Firstly, they are not made to monitor, but to contribute to the team’s success. They are created so that the people who use them benefit from help in assessing their work. This means that they have to be self-enforced, or be the result of a conversation, where professionals share information about the actual process and what they are facing and decide accordingly what to measure. 

Moreover, when using software development quality metrics, Agile teams corroborate them to a certain premise and leads, so that they may investigate potential issues, instead of measuring for the sake of measuring. 

Agile software development quality metrics focus more on tangible results like the number of completed story points and customer satisfaction than on ambiguous criteria. 

Waydev is the new Agile Data-Driven method of tracking the output of engineering teams directly from your engineering toolstack, without requiring manual input. Extremely flexible, this approach benefits from real-time insights which ensure early issue detection, thus saving time and resources. 

By using our Agile Data-Driven methodology, you receive insights on how your teams work together, as well as automatic reports. This way, engineers may focus on their actual work, instead of doing administrative chores. 

Velocity

Velocity is, maybe, the best known and universally-accepted Agile metric. Its purpose is to show how much work a team may complete in a certain period of time. 

Velocity is calculated by adding the story points that have been marked as Done, in a certain sprint – no matter when they were initiated. 

What is worth noting is that velocity is not a measure of performance or competency, but one which focuses on the volume of work which should be attributed to a team. It is more of a quantitative metric than a qualitative one, as it doesn’t reflect the value of the work delivered.

The best way to use velocity, according to many of the engineering managers with whom we have discussed, is to analyze a 3 sprint series and calculate the average velocity. 

By understanding insights in real time, Waydev enables you to achieve more than 2X Accelerated Velocity. 

Lead time

Lead time is the time that passes from the moment the engineers start conceiving the software solution, until its deployment. This includes coming up with ideas, brainstorming, creative directions and design, as well as the actual launch on the market. 

Waydev accelerates lead time by 22%, thus contributing to streamlining operations, improving productivity, increasing output, and revenue.

Cycle time

Cycle time is similar to lead time, this is why some professionals confuse the two. The term refers to the time spent since the engineers started developing the app, up until the actual software project has been completed. 

With Waydev, engineering managers benefit from cycle times which are faster by 28%. 

Recidivism

In Agile software development quality metrics, recidivism refers to the number of user stories that have been marked as completed, but which have been returned to the development team. This is often a result of them failing QA tests, but it may also be a sign that requirements changed, for example. 

As a percentage, recidivism is calculated as the number of stories in a sprint that entered development twice, split by the number of completed stories. The smaller the ratio, the more successful the project. 

Generally, when recidivism is of about 10-20% it may indicate that there are issues in the actual chain, such as in the quality of requirements, test quality etc. 

Story point completion ratio

Story point completion ratio is the number of stories completed in a sprint compared to the number of stories that were estimated. 

This software development quality metric helps development units understand their forecasting capacity and ensure they estimate correctly their work, the size of the stories and what they deliver. 

The smaller the story point completion ratio, the more chances there are that teams are either taking on more than they can deliver or that the project is running into impediments. Before sharing this metric with the company, ensure that the overall situation has been assessed and that the reasons for under-delivering have been identified. 

Companies that use Waydev notice a 24% increase in features delivered. 

Production metrics

Production metrics are used to measure the level of work which has been performed, to determine how efficient and fast the team is. 

Active days

Active days represent the time spent on the actual coding and it excludes other activities such as planning or administrative tasks. By measuring this, managers may assess their cost sources and create healthy operational models. 

Waydev helps increase average active coding days by 15% per week.

Failure and repair time

Fails and bugs are inevitable when developing software, especially when it comes to projects that are done from scratch. Managers measure failure and repair time to understand how much time it takes for engineers to solve the issues that arise. 

Task scopes

Task scopes refer to how much code a developer may produce every 12 months. This metric focuses completely on quantity and it is used to assess how many developers are needed for a project. 

Code churn

Code churn focuses on the number of lines of code that have been modified, added or deleted in a 21-days time frame after being initially written. 

Managers that use Waydev have reported a 28% decrease in code churn rates. 

Efficiency 

While this is quite difficult to measure, it refers to the amount of productive code that software teams deliver. The lower a teams’ churn, the higher the efficiency. 

Impact

Impact is a metric that showcases to what extent added code affects the overall software development project. 

Security response metrics

Nowadays, each form of software you develop needs to be protected from maleficent actors and attacks. This is why it is crucial to ensure that the app, the website or the platform your team delivers recognizes and eliminates threats. Security response metrics do just that, they assess how secure your product is. When monitored constantly, these are extremely valuable. 

In order to understand security, engineering managers keep track of metrics like:

Endpoint incidents

which refers to the number of devices that have been infected by a virus, in a certain amount of time.

Mean Time to Repair (MTTR)

which, in this context, focuses on how long it takes to solve a thread, calculating the time from its discovery, to the moment when a remedy has been deployed. 

The age of dependencies

Managers often measure the age and status of dependencies, to ensure they work properly and that they don’t need to be updated. When dependencies are extremely old, they may compromise projects. 

Size-oriented metrics

Size-oriented metrics focus on the size of the software and are usually expressed per line of code. Here are some examples: 

• Bugs per KLOC;
• Errors per KLOC;
• Costs per KLOC. 

Function-oriented metrics

Function-oriented metrics are used to showcase the functionality offered by each software. Since the actual functionality is not measurable, per se, software managers focus on function points (FP). These are defined as units that quantify the business functionality of each product. An important benefit of function points is that they may be used to compare projects written in different languages.

Defect metrics

Maybe the most intuitive way of assessing software quality is by identifying the number of defects and focusing on: 

The stages in which they are identified (for example, before or after deploying the project);

Their density

the number of defects per LOC;

Their slippage ratio

the percentage of flaws that manual tests haven’t identified, before deploying the project;

Defect removal efficiency

Calculated through the following formula: DRE = E / (E+D), where

DRE = Defect Removal Efficiency

E = errors found before product delivery

and D = defects found by the end-user, after product delivery, 

the goal is to have DRE as close to 1 as possible. 

Quality assurance metrics

As software projects become more and more complex, the need for quality assurance testing increases. Nowadays, QA engineers are indispensable, as they are the ones that write and execute test cases, thus contributing to understanding the project’s progress. 

Examples of QA metrics include the following examples: total number of test cases, number of passed test cases, number of failed ones, number of identified bugs, number of test hours required etc. 

Pull request metrics

Measuring pull requests helps you better understand your projects and the way your team interacts. Our recommendation is to focus on the following indicators:

• The number of pull requests that did not pass tests;
• The number of pull requests that broke the build;
• The number of pull requests that have been rejected or merged;
• The number of pull requests that generated comments. 

When assessing them, bear in mind that complex software programs will have high metrics.

Waydev’s PR Workflow analytics bundles up all the features you need to identify bottlenecks, long-running or unreviewed pull-requests, so you have a better understanding of PRs and improve your engineering teams’ performance.

Customer satisfaction 

Customer satisfaction is one of the most valuable aspects worth measuring, since, at the end of the day, happy customers are what keeps the industry going, as they generate business through their own projects, as well as through recommendations. 

Customer Satisfaction Score (CSAT) 

requires clients to rate the quality of the service. 

Customer Effort Score (CES)

enables companies to understand the overall client experience, as it asks questions like “how easy was it to solve your problem with the help of our team?”. 

Net Promoter Score (NPS)

ask clients to rate, on a scale from 1 to 10, how likely it is that they would recommend you. The ones that answer with 9 and 10 are considered potential brand promoters. 

Waydev enables you to get full visibility of what the team is doing

Problem: In order to identify the level of progress made by developers, you need to firstly assess the performance of the whole team, as well as that of individual members. It is only so that you may have a clear image of what is happening. 

Solution: By tracking the output from your engineering toolstack, Waydev enables you to get complete visibility of what team members are doing. 

This means that you may:

• View all commits and pull requests that engineers create;
• Receive daily updates and analyze sprint over sprint velocity;
• Use the Activity Heatmap to identify the best moments to schedule meetings, thus ensuring you are not disrupting the engineering flow;
• Stay updated on sprint progress and use the forecast model to mitigate risks. 

Moreover, you may organize data according to your needs, sort it, filter it, search for certain projects or team members and download whatever is necessary. Waydev offers you complete control over the information you receive, empowering you to take the best measures, to benefit from excellent results: 

+22% Accelerated Time to Market

  2.2X Accelerated Velocity

+28% Faster Cycle Time

 -21% Less Unplanned Work

+15% Coding Days per week

 -28% Decrease in Code Churn

 +32% Faster Code Deployed

 +24% Increase in Features Delivered. 

If you want to have a clear view of your software quality metrics, contact us and schedule a demo today!