Quality KPIs and Scorecard – Full Guide with Examples

How to measure quality and anticipate product defects? In this guide, we will discuss specific quality metrics and show how they contribute to the overall strategy.

Part 1. Quality KPIs Analysis

The article starts with a list of quality metrics sorted out into groups. We’ll discuss some popular quality KPIs and the ways to use them. Learn more >

Part 2. Quality Balanced Scorecard

In the second part, we’ll talk about the business context behind quality and build an example of the Quality Balanced Scorecard. Learn more >

Part 3. Quality In Practice

In the third part, we’ll discuss ten ideas that one can use to convert a scorecard into an effective tool for ongoing quality control. Learn more >

Part 1. Quality KPIs

The hierarchy of quality-related goals and performance indicators.
Source: Quality Scorecard.

Before discussing specific KPIs, let’s answer two questions:

If you prefer to go ahead with the KPIs, here are the topics that we’ll discuss below:

Stakeholders of Quality

Quality is not an absolute value. When talking about quality, we always talk about the perception of quality. Who perceives the quality of the product or service?

We can start with:

• End customers
• Internal customers (our team and partners)
• Regulators

Thinking about end customers, we can be even more specific and define, for example:

• Prospective clients
• Paying customers
• Returning customers
• High-end customers

Definition of Quality

What is quality?

Oxford dictionary defines quality as:

The standard of something as measured against other things of a similar kind; the degree of excellence of something.

Different stakeholders perceive quality in different ways. For example:

• Prospective clients might be fascinated by your marketing and buy on a very basic perception of quality (quality of package and marketing materials).
• Customers who used your product for a longer time will have a much better idea about its quality (as applied to their needs of course); those who will find the match will become returning customers; those who won´t – will switch to the competitors.
• High-end customers will most likely have different needs and different ideas about the product’s quality.
• Your team will deal with the internal mechanics of your business, and their perception of quality will be more related to the product architecture and complexity.

We also know that the idea of quality is domain-specific. Still, we can observe that the approach to maintaining high quality is very similar across different industries.

For example, the idea behind quality metrics for the pharmaceutical industry looks similar to the approach used in the steel industry software engineering or aviation.

Quality KPIs for Stakeholders

Below, we will discuss some popular KPIs for quality. For the best implementation results, make sure that these metrics fit your business context perfectly. We discuss how to achieve this in the second part of the article.

Metrics that we review below:

NPS (Net Promoter Score) as a Quality Indicator

In its classical form (a question “Would you recommend us?”), this metric indicates a general perception of the product or service.

• The low NPS might be a result of poor product or service quality
• Ask some follow-up questions to find out what is actually broken

A metric, in this case, is as simple as:

• NPS, %

Number of Complaints per Period

Any customer feedback is important. In some cases, we only know about the problem after getting an email from an angry client.

Use the ‘Number of complaints per period of time’ indicator to understand and fix problems in the business systems.
Source: Quality Scorecard.

Even if you do your best to anticipate all possible problems, such complaints will still exist. Actually, they show that:

• Your customers still care (otherwise, they wouldn’t spend their time giving you feedback).
• There is some problem that needs to be fixed.
• There is something that you didn’t know about your product before, and you need to improve your quality control system.

Indicator to use:

• Number of complaints per period of time

A general approach to complaint solving is to understand and fix problems in the business systems rather than blame specific people.

Social Proof of Quality

In many cases, prospective clients cannot really evaluate the product before buying. Still, they can make a good choice based on social proof metrics.

• Average rating score. The aggregated value of the clients’ rating of the product and their comments form social proof metrics.

Think about booking a hotel or buying something on Amazon. You can get a rough idea about the product by reviewing the photos and the description, but too many products look the same:

In this case, we look at the hotel rating or starts on Amazon and form our expectation of product quality accordingly.

This quality metric is similar to NPS, but in this case, buyers are more likely to share more detailed unbiased options.

Complexity Metrics

The more effort your team invests in the quality control and assurance, the less problems you will see reported daily.

Complexity index formed by the indicators with different relevant weight.
Source: Quality Scorecard.

But it doesn’t mean that your product is ideal; on the contrary, you’ll face an even more challenging situation now:

Instead of fixing obvious the problem, now you need to read between the lines and understand what issues (less obvious ones) prevent users from unlocking the true potential of your product.

I’m talking about complexity.

Before, we discussed how to measure complexity, and here are some key points to look at:

• Excessive timing
• Excessive choices
• Excessive explanations

The choice of specific metrics depends on your product; we can group them under the umbrella of:

• Complexity index, %

Fighting complexity is tough, but it’s worth the time you spend on it. Here is an example of how eliminating unnecessary complexities can help organizations to improve.

Quality KPIs to Estimate Financial Impact

Sometimes, quality specialists have to justify their budgets and strategies in board meetings. Here are some finance-related quality metrics to use.

Metrics that were analyzed below:

Cost of Poor Quality

Use this metric to understand the impact of poor quality. Estimate all costs associated with problem detecting and solving:

• Customer support specialist costs
• Solution costs
• Brand impact costs

In many cases, the goal is not to find the exact dollar value, but to understand how big the impact is on the stakeholders´ perception of quality.

For this article, I´ve created an indicator:

• Cost of Poor Quality

The indicator ‘Cost of Poor Quality’ is configured to sum the values of the indicators inside.
Source: Quality Scorecard.

This indicator summarizes the values of the indicators inside, giving us the total cost of quality. In my example, the calculations are for the average cost per accident.

Cost of High Quality

Similar to the “Cost of Poor Quality,” but in this case, we are estimating the costs of trying to achieve excellence.

Achieving high quality also has its cost. For example, decreasing the defects rate from 1% to 0.01% might be a cost-effective goal, but reducing the defects rate down to 0.001% might be too costly and won’t make sense for the quality stakeholders.

In this case, our indicator is:

• Cost of high quality, $

Its formula depends on the specific situation. For this example, we can calculate the budget needed to decrease the defects rate from 0.01% to 0.001%.

Opportunity or Risk Metrics for Quality

The resources of any organization are limited. We cannot treat all quality issues equally; some need to be solved immediately, and some will stay in your backlog for a longer time.

How to justify decision making in this case? A simple opportunity (or risk) estimation will help:

• Estimate the impact of the quality issue
• Estimate the probability of this impact

Now we have:

• Opportunity estimation metric, % = Impact, % x Probability, %

How to estimate the impact of the quality issue? Look at the:

• The number of customers who are affected by the problem
• Customers’ financial potential
• Customers’ involvement in the issue (the number of emails customer sent)

How to estimate probability? Try to map it on a scale from 0% to 100% where:

• 0% is something that potentially influences your customers, but you have no proof of it.
• 100% is for the critical factors that, for example, will prevent the customer from buying or will result in important costs.

Let’s use a case with BSC Designer software. A few years ago, we got two feature requests from the customers (both functions are implemented now):

• (A) Add a feature to allow/deny users to edit historical data in their scorecards
• (B) Change the way initiatives are displayed on the strategy map

(A) was a critical issue for a big potential client, and three other clients had previously asked about this.

(B) was a less critical issue for the existing client and only one more client had requested a similar feature.

Our efforts to “fix the problem” (actually, develop new functions) were about the same.

Here is an example of opportunity estimation for these two cases:

Impact
Probability
Opportunity
A
70% (was critical for buying decision of high-end customer)
80% (3 more clients asked for a similar feature)
= (0.7 * 0.8) * 100% = 56%
B
50% (would make life of existing customer something easier)
60% (1 more client asked for a similar feature, and our developers liked the idea)
= (0.5 * 0.6) * 100% = 30%

In this example, the value of the opportunity metric was higher for “A,” so we gave it a higher priority during the implementation.

KPIs to Prevent Quality Problems

For now, we were quantifying and measuring the impact of (poor) quality. Let’s try a different focus:

• Instead of fixing quality issues, let’s be proactive. Let’s work to decrease the possibility of future quality problems.

Here are the metrics that we can track to achieve this goal:

Near Miss Metric

How can we predict something that has not happened yet?

To predict some important quality problems, we need to look at the preceding, less important problems.

It’s that simple.

• That’s what broken windows theory teaches us about.
• That’s how aviation administration tries to prevent accidents in the air.
• That’s how Alcoa approached achieving their zero injuries goals (see “The Power of Habit: Why We Do What We Do in Life and Business” by Charles Duhigg).

How to build a near miss metric?

• Check out general thoughts about near miss
• Find near miss events relevant for your business processes
• Quantify, measure and track these findings

Implementing Near Miss metric is about changing the culture of the organization.

You need to teach your team to report and discuss minor quality issues that might lead to more important problems.

The metrics can be formulated as simply as:

• Leading metric: Near miss awareness training penetration, %
• Lagging metric: The number of near miss issues reported

The ‘Near miss reporting training’ goal from the strategy map with leading indicator, lagging indicator, and initiative.
Source: Quality Scorecard.

Requirements Ambiguity Metric

Similar to the Near miss metric, but in this case, we want to be proactive.

Instead of waiting for some quality issues to appear, we ask our best experts to review the product/service requirement documentation and spot possible problematic points.

The metric in this case might be:

• Ambiguities per requirements page, %

In software development, the experts review the software requirements for ambiguity.

The requirements, in this case, are the preceding documentation, so if we make the requirements better, we can prevent future quality problems proactively.

Product Age and the Number of Modifications

We are in the software business, and I’m always curious about the best ways to quantify the software quality and specifically the quality of software code.

A classical “number of lines of code” doesn’t make a lot of sense. Intuitively, software developers understand that:

• Frequently changed code will contain more defects than code that remains unchanged for a long time

In other words, we expect code quality to be the function of code age and the number of code modifications. That what was confirmed empirically in the “Predicting Fault Incidence Using Software Change History” .

Replace “code” word with “product,” and I’m sure the same findings will make sense for any business with a complex product. Thinking about this in the context of quality strategy, we can predict defects more effectively and update testing strategy, respectively.

Expert Review Metric (WTF/Minute)

Similar to the requirement ambiguity metric, but in this case, we can ask experts to review anything – our product, user manual, customer service.

How to quantify such reviews? A humoristic (still practical) response to this question was given by Thom Holwerda in his comics for osnews:

• WTF/minute (What’s This For / Minute).

The original metric was for software code quality reviews, but as you can imagine, it works in any domain.

Aleatory Test Metric

Another metric is useful to estimate the quality of large samples.

This is applicable mostly to physical products.

1. Mix the samples
2. Take a few random items
3. Check out their quality
4. If some defects were found, estimate proportionally the real scale of the problem

The metric, in this case, might be:

• % of defects in aleatory sample

For example, in a supermarket, you don’t need to check out the expiration date on each pack of milk or cheese – just pick few at random, and if you found some of them have expired, then you can have a rough estimation of the problem.

Obviously, this approach works on early product stages where there are often quality issues. With the development of quality control systems, the size of the aleatory sample should be increased.

Quality Self-Test Metric

Sometimes, we have to check the “quality of quality,” e.g., how the quality control system works itself.

• Do quality reviews actually work?
• Do automated tests actually detect the problems?

The idea of self-test is to introduce the input with known errors and see if those pre-engineered problems were detected.

Medical analysis laboratories have this routine implemented. The laboratory equipment is a combination of complex optic devices and chemical reagents. The factors of quality are the way reagents are stored, the quality of distilled water, the stability of the electric current, human factors, and random errors off course.

The easiest quality control, in this case, is to load a known set of samples and compare actual results with expected ones.

Regulators use the same approach doing blind tests when the properties of the test samples are not known beforehand.

The metrics in this case are:

• The number of self-tests per period
• Defects detection efficiency, % = Defects detected / Defects injected

This approach works as a starting point before doing more expensive and time-consuming tests.

Automated Quality Control Metrics

Similar to the self-test, but in this case, we are not looking at the system as a black box, we can do more specific automated tests.

The benefits of automated tests compared to manual tests are obvious:

• No human factor involved
• Less costs

Still, we need to balance automated test coverage with the cost to develop and maintain such tests.

The possible metric in this case:

• Automated test coverage, %

Stress Test Metric

The quality of the system can change when the environment changes significantly.

The metric in this case:

• Stress changes readiness, %

The idea is to foresee the possible conditions and try to be prepared. Imagine a best- and worst-case scenario and try to come up with emergency plans.

• Have a smartphone? Try mountain hiking during the winter season and see how long your battery will last.
• Do you run a website? Will it survive a DDoS attack?
• Designing consumer products? Will it work properly if used in India?

Manual Test Metrics

Old good manual tests are a must. Here are two recommendations:

• Be the user of your product
• Take into account different stakeholders of quality

The metrics in this case:

• Test plan coverage, %
• Test plan execution time

Less formal, but still useful metric:

• Time being end-user of the product, hours

For example, to illustrate this article and create a scorecard for quality, I was using our BSC Designer software. This always helps in generating some improvement ideas.

Change Risk Analysis

When the complexity of the product is high, changes in the product might produce many quality issues. The solution to this challenge lies in the level of product architecture. For example, software engineers build their solutions using microservices to avoid bad dependencies.

From the viewpoint of quality, the starting point is to estimate risks of change. Ask your engineers:

• If we change function “A” of the product, how likely is that something will be broken?

Following this logic, we can use:

• The number of recent changes metric

As a leading factor for future quality issues.

Quality Response Metrics

We tried to fix the problem on the design stage. We tried to fix the problem before the customer (or other stakeholders) saw it. But sometimes, a problem passes all these filters, and customers face a quality issue. We discuss below the metrics that we can track in this case.

Here are the metrics that we discuss below:

Time to Solve Issue

Yes, there was a problem, but it was fixed overnight. That’s a wow experience for any customer.

Metrics:

• Average time needed to solve the critical problem
• Average time needed to solve a minor problem

Track separately timing for different stakeholders. See the risk paragraph for the detail.

Another important idea is that normally, time to solve the problem doesn’t include:

• Time needed to develop prevention plan, or
• Time needed to implement prevention plan

Time to Respond

In many cases, the problem is not that critical for the client, and what is more important is that your team care. A quick email, “Hey, thank you for reporting. We are on it, expect a fix in 2 days” will show a customer that you care.

Metric:

• Time to respond to quality issue

Quality Backlog

Similar to the sales backlog, a quality backlog is an indicator of how quality flows throughout your organization.

No backlog of quality issues most likely indicates that you:

• Have not changed your product or service for a long time (and this fact itself is a problem), or
• You are not listening to your clients well enough

Overloaded backlog shows that your response efforts are not sufficient, and something needs to be changed.

My recommendation is to track in dynamic this metric:

• Quality issues backlog

The sine function is used as a performance function for the indicator ‘Quality issues backlog’ – to be in the green zone there should always be some issues in the backlog.
Source: Quality Scorecard.

Quality Team Metrics

A popular metric for quality teams is:

• Quality personnel / Normal personnel, %

If analyzed together with issue fix time and returning problems rate, % it might give some information about the team’s performance.

Findings Closure Ratio

Imagine that a company did a quality audit. For example, using the requirements ambiguity analysis described above. We need to make sure that those findings are analyzed and closed.

Metric to track in this case:

• Findings closure ratio, %

We are not always talking about implementing the recommendations, but at least they need to be reworked into some tangible form (for example, updated requirements).

Quality of Communication

The complexity is a known leading factor of quality.

Track the communication redundancy metric in this case:

• Communication redundancy, % = (Average message size – Shortest message size) / Average message size.

Did the customer need to make just one call to fix the problem, or it was an exchange of five emails to solve a simple issue?

For example, in the case emails exchange, we can calculate the metric:

• Communication redundancy, % = (5 – 1) / 5 = 80% redundancy.

Track also:

• First contact resolution rate, %

Solution Effectiveness

We fixed the problem, and we communicated efficiently and effectively along the process, but was the problem actually solved?

Metric to track:

• % of returning problems

or similar

• Right first time, %
• OTIF, % – On Time in Full in logistics

These metrics work best if bundled with the time to solve metric. The organization needs to find a balance between the speed of providing solutions and their effectiveness.

Quality Culture

Quality culture is about establishing new behaviour habits that will help to avoid critical problems and maximize the effectiveness of solving less critical issues.

Does your team have a habit of tracking and improving quality?

KPIs that we review here:

Rejected Defects

Imagine a scenario when a user reports a problem in your product, but it appears not to be a problem but normal behaviour. The developers “reject” this report as a defect.

As long as quality is perceived by the clients, this situation will be considered by a customer as a defect. What does a rejected defect mean? It means that some other system did not work well.Metric to track:

• % of rejected defects

Here is an example:

• A small fridge is making too much ice and is causing a problem to its owners. They call customer service, a repairman comes, and tells them that this small fridge is not suitable for continuous use; it is supposed to be used in motels where people come and go. This is a quality issue, and it already has its costs (brand impact, guaranty service fees), but this defect will be rejected by the engineers. Where is the problem then? I guess with marketing…

A similar situation is when a tester of the product reports a problem that is rejected by the engineers. Where is the problem in this case? I believe it is in the ambiguity of the initial requirements.

Root-Cause Analysis Metric

Fixing quality issue on time is a good starting point, but an organization also needs to learn from its mistakes (see the quality assurance article on Wikipedia). Do a root cause analysis for quality issues:

• What was the reason for the problem?
• Why didn’t we detect it before? What can we change in our quality control system to detect similar problems?
• What can we change in our systems to eliminate the possibility of such problems?

Off course, the degree of root-cause analysis should correspond to the importance of the product.

Metric to track:

• Leading: Root-cause analysis training penetration, %
• Lagging: % of the problems where root-cause was analyzed

Systematic Error

Part of the quality assurance challenge is to detect and find systematic errors. Let’s say your SaaS is hosted on a slow server, and it results in many other problems with system usability. It doesn’t make sense to review all those usability problems when they are caused by a bigger systematic error.

Quality Awareness Metric

Quality is everyone’s job, so it makes sense to explain the necessary quality procedures to all involved members of your team:

• How should people report a “near miss”?
• What is root-cause analysis and how to conduct it?
• What is a response plan to the critical issues?
• What are our quality standards?

The metrics to track:

• Leading: Quality training penetration, %
• Leading: Outdated training materials, %
• Lagging: Quality program awareness, %
• Lagging: Quality standards awareness, %
• Lagging: % of policies understood by employees

As you can see, the lagging awareness metrics are balanced with the leading training metric. Professionals responsible for the training might focus their measurement efforts even more by tracking:

• Immediate reaction
• Change in actions
• Business impact

We discussed this approach in the training scorecard article.

Improvement Acceptance Ratio

Your team and your customers might suggest a lot of improvement ideas. The question is, how many of them do your engineers actually analyze and implement? If ideas go directly to the rubbish bin, then we cannot expect high morale from the employees.

A better approach would be to establish a procedure to deal with ideas, similar to what we have discussed for the innovations scorecard.

An additional map was created to explain the idea collection process and its alignment with the goals of the Innovations scorecard.
Source: Innovations Scorecard.

A starting point for performance measurement in this case is:

• Improvement Acceptance Ratio, % = Suggested ideas / Implemented ideas

Retention Rate

The poor quality affects the stakeholders of quality.

We might see that customers are no longer recommending your product, or valuable members of a team are looking for a new employer. Track these two metrics to have an early warning signal:

• Customer retention rate, %
• Top performers turnover, %

These are lagging metrics, but we can find some that have some shorter time horizon:

• Instead of seeing how customers leave, we can look at what they share in social media.
• Talent managers can track early indicators of turnover using the HR scorecards.

Your Unique Quality Metrics

I believe most of the metrics discussed in this article can be applied to your domain. Still, there are always some unique metrics that need to be tailor-made for a specific business. How to find these metrics? I do recommend having a look at the KPI System that we have for you.

My best guess is that:

Your unique quality metrics will be related to bottleneck process in your organization

Did my guess work for you? It’s easy to explain; if something happens to a process that limits the system, it will affect the rest of the system.

Quality Measurement by Design

Another topic that we discussed in the KPI System is measurement implemented “by design.” Instead of trying to measure something afterwards, we need to build a product with future measurement needs in mind.

Applying this idea to quality, we need to find out where the quality measurements can be implemented by design.

Ask your quality team these questions to get started:

• When our end customer reports some issue, how will we know what is causing the problem?
• What can we change in the product or service to make quality measurement easier?

The answer can be a simple checklist or a kind of black box with operation log recording.

And here is a trick:

If you find ways to better measure quality, it automatically results in quality improvement.

For example, in the software development domain, adepts of DevOps practices first create tests for software functionality, then build the functionality itself. On the one hand, it allows measuring quality better; on the other hand, it improves the quality out of the box.

Implementing Quality Culture

We discussed some ideas related to the quality measurement culture. A more important topic is how to focus a company’s culture in general on quality. There are many books written on the topic, and in part 3 of this article, you will find my short list of top 10 tips.

Part 2. Quality Balanced Scorecard

The Balanced Scorecard is a strategy execution framework. What are the goals of quality? How do those goals fit into the overall strategy of the organization? How to track the execution of those goals? Below, we’ll create an example of the Balanced Scorecard for quality.

An example of the Balanced Scorecard strategy map for quality.
Source: Quality Scorecard.

Here is our plan:

Mission

What’s the purpose of focusing on quality? Let´s formulate some inspiring mission statement for a quality team:

Solve quality problems effectively, think about prevention plan

That´s a simple message that anyone in the quality team should understand. We are here to:

• Solve quality issues fast
• Investigate and implement a prevention plan

Let’s continue with the perspectives of the Balanced Scorecard.

The expectations of the stakeholders of quality are formulated in the Financial perspective of the Balanced Scorecard.
Source: Quality Scorecard.

Financial Impact of Quality

What is the financial impact of quality? In other words, what will happen if stakeholders of quality are not satisfied with the quality of the product?

The answer depends on the stakeholders:

• If clients are not satisfied – they won´t buy or won´t do repeat purchase.
• If your team is not happy about product quality – they may leave the company to find a better place to apply their talents.
• If you don’t comply with the standards set by the regulator – your organization might be fined or closed.

I suggest mapping these ideas under the goal:

• Sustainable growth

This is a very vague goal. How can we make it something more specific? We can align some performance metrics with it:

• Cost of Poor Quality – will show the financial impact of the quality

Customer Perspective

To ensure the sustainable growth of the organization, we need to satisfy the needs of our clients (stakeholders of quality like end users, regulators and team).

The Customer perspective includes the needs of external (end-users) and internal (team) customers.
Source: Quality Scorecard.

Goal: Avoid critical defects, fix non-critical defects quickly. That’s what all stakeholders want. How can we estimate the degree of achievement of this goal? Let’s use some of the indicators discussed above. These will be lagging indicators in the context of the goal:

• Number of Complaints per Period
• NPS
• Customer retention rate, %

What about leading indicators? The leading performance will come from the goals in the internal perspective that we will discuss below.

Another customer goal is more related to the internal stakeholders of quality:

• Goal: Maintainable product. A product that is relatively easy to maintain, change, test, etc. That’s what your team want.

What lagging metrics can we use to track the execution of this goal? For example, these two:

• Quality or Complexity-related complaints from team
• Top performers turnover, %

That’s a simple model of how product complexity and team engagement is connected:

• If a product gets too complex, the members of our team will complain about this, and if we do nothing – they will leave.

What about leading metrics? What can we do to prevent such cases? It depends on the specific situation, but I would say that we need to understand the reasons for quality/complexity problems. For that purpose, we’ll have “Root-Cause Analysis and Quality Assurance” goal in the Internal perspective that drives leading performance to the “Maintainable product” goal.

Internal Perspective

In this perspective, we need to answer two questions:

• How are we going to “Avoid critical defects” and “fix non-critical defects quickly”?
• How can we ensure that we have a “Maintainable product”?

I will formulate these three goals, and the links on the screenshot will show how they contribute to the goals in the Customer perspective:

• Prevent Critical Quality Problems
• Solve Quality Issues Quickly
• Root-Cause Analysis and Quality Assurance

Again, these goals are too vague for now. To make them more specific, we need to align some indicators with them.

Let’s start with “Prevent Critical Quality Problems” goal before we discuss different approaches to preventing quality problems, respectively lagging indicators can be:

• Near miss issues reported
• Ambiguities per requirements page, %
• Expert Review Metric (WTF/Minute)

How can we ensure that our team mastered, for example, “Near miss reporting”? For this purpose, we’ll map the “Near miss reporting training” goal in the Learning and Growth perspective.

What about the “Solve Quality Issues Quickly” goal? The basic measure will be Time to solve the critical problem.

It is also important to train all members of the team to report quality issues properly. We can estimate the achievement of this goal by using the Improvement Acceptance Ratio metric.

Internal perspective includes the drivers of quality strategy, e.g., the goals that need to be achieved internally to satisfy the needs of stakeholders.
Source: Quality Scorecard.

Finally, the “Root-Cause Analysis and Quality Assurance” goal contributes to the goal of having a “Maintainable product”. How can we make sure that our team is actually doing good root-cause analysis and quality assurance?

Let’s start with the basics. We can track that our team is thinking in the right direction by measuring:

• Time to develop prevention plan
• Time to implement prevention plan

You might ask:

How do we know that this time budget is spent effectively?

Well, with these metrics, we cannot know for sure, but if things go wrong, you will see that indicators aligned with “Maintainable product” goal from the Customer perspective will appear in the red zone:

• Internal stakeholders will start complaining about product quality/complexity (see the “Quality/Complexity Complaints from the Team” indicator).

OK, we know how the quality team should behave. What should we do to equip them with the right skills and tools? The answer is explained below in the Learning and Growth perspective.

Learning and Growth Perspective

For the Prevent Critical Quality Problems goal, we have two supporting goals in the Learning and Growth perspective:

• Near miss reporting training
• Learn to Prevent Quality Issues

I’ve separated them because I believe near miss reporting deserves special attention.

In the Learning and Growth perspective, we explain where the organization needs to focus their learning efforts to support the drivers of the quality strategy (Internal perspective).
Source: Quality Scorecard.

Let’s start with the Near miss reporting training goal.

The leading metrics will be:

• Near miss awareness training penetration, %

Obviously, we want all involved stakeholders to be properly trained.

What about lagging metrics? If we want to do it seriously (see training scorecard approach for the detail), we need to understand how we are going to measure the:

• Results of the training
• Change in actions
• Impact on business

The results of the training is an exam score. Change in actions is what we measure with Near miss issues reported aligned with the Prevent Critical Quality Problems goal.

How to measure the impact of “near miss” training on business? I’m looking at the Avoid critical defects, fix non-critical defects quickly goal, and I believe the Number of Complaints per Period is an excellent indicator for this purpose.

Specific initiative aligned with one of the training goals on the scorecard includes a detailed action plan, expected budget and timeline.
Source: Quality Scorecard.

The last goal in the Learning and Growth perspective is Quality Awareness. Before, we discussed that it’s important to keep quality standards and respective training material up to date. For that purpose, we have:

• Outdated training materials, % leading metric.

Again, we need to make sure that all stakeholders were trained, and we use this metric:

• Quality training penetration, %

To track the results of the Quality Awareness program, we will look at these lagging metrics:

• Quality program awareness, %

How can we quantify Quality program awareness, %? For example, by doing some simple surveys or tests.

How can we make sure that the Quality Awareness program changed the way stakeholders behave? We need to look at the specific behaviour metrics. I’ve selected these two:

• Communication redundancy, %
• Time to respond to quality issue

The ‘Quality awareness’ goal is quantified with the indicator ‘Time to response to quality issues.’ Its performance function is configured for ‘Minimisation’, e.g., faster response (lower value of the indicator) means higher performance.
Source: Quality Scorecard.

 

Let’s use this goal as an example and track how the logic of the strategy map works.

Logic on Strategy Map in Action

If our Quality Awareness program is a success, we can expect that, for example, customer support specialists will be responding quickly (Time to respond to quality issue metric metric) and efficiently (Communication redundancy, % metric).I’m moving to the “Internal” perspective.

The Quality Awareness goal is a leading factor for the Solve Quality Issues Quickly goal. Basically, our hypothesis is that by improving quality awareness, we expect to solve quality issues better. And we will check this hypothesis by looking at the Improvement Acceptance Ratio, % and the Time to solve the critical problem indicators.

We are now looking at the “Customer” perspective.

Solve Quality Issues Quickly goal contributes to the Avoid critical defects, fix non-critical defects quickly, that finally contributes to the Sustainable growth goal in Finance perspective.

The logic of the quality strategy map – the drivers of the strategy (Learning and Internal perspectives) create the outcomes (Customer and Stakeholders perspective).
Source: Quality Scorecard.

Relevant Importance of the Contribution

From the strategy map, we see that Avoid critical defects, fix non-critical defects quickly is influenced by the leading performance of two goals:

• Prevent Critical Quality Problems
• Solve Quality Issues Quickly

Do these goals contribute to the Avoid critical defects, fix non-critical defects quickly goal with the same weight?

Let’s suppose that preventing problems is more important than fixing them.

To reflect this logic, I can change the weight of leading indicators respectively:

• Prevent Critical Quality Problems – weight 70%
• Solve Quality Issues Quickly – weight 30%

BSC Designer software that I used to create a scorecard for this article will take these weights into account when calculating the leading performance of Avoid critical defects, fix non-critical defects quickly goal.

If you want to learn more about the way the scorecard values are calculated, check out the Scorecard 101 article.

In the context of the ‘Avoid critical defects…’ goal, the outcomes of the goal ‘Prevent Critical Quality Problems’ and goal ‘Solve quality issues quickly’ are the leading indicators. Use used relative weight to explain their impact on the total performance.
Source: Quality Scorecard.

Explanatory video

Here is an explanatory video that summarizes some important findings of the article.

Quality Balanced Scorecard and KPIs Explained

Watch this video on YouTube

Quality Management Frameworks and Strategy Execution

There is a number of well-known quality management frameworks, such as TQM, Six Sigma, Kaizen (Continuous Improvement), Lean Manufacturing, and, of course, the latest version of the ISO 9001 .

Let’s discuss how to combine:

• The ideas from these quality management frameworks

with

• The ideas of the Balanced Scorecard, a strategy execution framework discussed in this article.

Quality Management Framework vs. Strategy Map for Quality

Before, we were talking about the Balanced Scorecard for quality. The Balanced Scorecard (together with its visual diagram – strategy map) is one of the tools for strategy planning and execution.

Let’s match the concepts of the quality management frameworks with the ideas of the Balanced Scorecard framework:

Quality Management Frameworks
Balanced Scorecard for Quality

Continuous analysis of customer needs
Most of the frameworks require the focus on the customer’s needs and continuous analysis of those needs.
The ideas of customer focus and continuous analysis of customer needs fit perfectly into the Customer and Learning and Growth perspectives of the Balanced Scorecard.

Systematic changes of the business systems
Quality frameworks recognize the benefit of a systematic approach to adjusting business systems according to the challenges of quality.
In the Balanced Scorecard framework, these changes are mapped into the Internal Processes Perspective that is connected by the cause-and-effect logic to the Customer and Learning and Growth perspective.

Alignment with the interests of stakeholders
The frameworks, in particular, the latest ISO 9001, recognize that the ultimate goal of quality improvement is the improvement of the overall performance of the organisation.
Exactly the same idea is articulated in the Balanced Scorecard framework, all goals from the strategy map are connected to the Stakeholders/Finance perspective.

Other typical components of a quality management framework include:

• Continuous improvement
• Plan-Do-Check-Act cycle
• Involvement of top management

Let’s see how these ideas can be implemented in the strategy scorecard.

Make Improvement “Continuous” with KPIs

Any quality management framework underlines the importance of continuous and systematic work on quality.

• What does it mean in the context of a quality scorecard?

Having property defined indicators already implies systematic approach to quality improvement.

What about the continuity of this systematic approach. How could we guarantee it?

The easiest way is to define update intervals for quality indicators on the scorecard:

We used the Values Editor to configure the update interval of ‘Time to solve the critical problem’ to monthly update interval.
Source: Quality Scorecard.

If you are using automation software, then setup notifications to be informed when an indicator needs to be updated:

The owner of indicator will be notified about an upcoming update interval 5 days before the update date.
Source: Quality Scorecard.

PDCA Cycle Based on Strategic Initiatives

Under different acronyms, the PDCA (Plan-Do-Check-Act) cycle can be found in all quality frameworks.

• How does quality scorecards support the PDCA?

Any improvement plan needs to be developed in the context of certain business challenges. In this sense:

The quality scorecard with a properly designed strategy map provides everything we need to understand the context – the business goals and connections between them.

Additionally, the initiatives of the strategy map help to describe the quality improvement plans by assigning budget, timelines, linking to the more detailed action plans:

Use the status of the initiative to track the strategy execution process.
Source: Quality Scorecard.

A typical quality improvement initiative passes certain phases. On the strategy scorecard, those phases can be reflected via the status of the initiatives.

The “check” part of PDCA is presented by the KPIs of the quality scorecard. If you are using automation software, then the dashboard and analysis functions will make the review of the performance results easier.

Involvement of Top Management

Most quality management frameworks recognize that the best quality improvement is achieved when top managers are responsible for quality. 

• How is it implemented in practice?

A starting point is a strategy map for quality. It helps to involve all stakeholders, including the top management, in the discussion around the strategy of quality.

The next step is cascading of the quality strategy.

Some quality indicators (quality KPIs), will be aligned with a scorecard of the CEO. For example, the “Perceived system quality” indicator discussed in the CEO’s scorecard article:

The ‘Perceived system quality’ indicator from the CEO scorecard takes its performance from the Quality Scorecard.
Source: CEO Scorecard.

Certain quality indicators will fit well into the corporate governance scorecard. For example, the KPIs that quantify the quality of management, and the overall product quality index.

As a leading factor of quality, the corporate governance scorecard might include the “Standards and processes defined” metric:

The indicator ‘Standards and processes defined’ contributes to the Corporate Governance Scorecard.
Source: Corporate Governance KPIs.

Similar to the top management, other stakeholders need to be involved in quality strategy. For example, on the procurement scorecard, we can find the “% of defects” as one of the metrics used for the “Compliance index.”

‘% of defects’ metric used in ‘Compliance index, %’ in the Procurement scorecard.
Source: Scorecard for Procurement.

Conclusions

As you can see, the quality management frameworks and the strategy execution frameworks complement each other.

In the context of quality management, one of the main benefits of having a quality scorecard is the alignment between quality initiatives and overall strategy of the organisation.

The formalized strategy of quality helps to align quality initiatives across the departments as well as make quality the job of everyone in the organization.

Part 3. Quality In Practice

10 Tips About Using KPI Scorecard for Ongoing Quality Control.
In the previous parts, we discussed how to build a Balanced Scorecard for quality and execute a quality strategy effectively. What about quality on the operational level? In this part, we will discuss ten ideas about using quality KPIs for ongoing quality control.

1. Business Context Comes First, then Come Indicators

In the previous parts, we discussed how to build a Balanced Scorecard for quality and execute a quality strategy effectively. What about quality on the operational level? In this part, we will discuss ten ideas about using quality KPIs for ongoing quality control.

Always start with business context! The Balanced Scorecard framework is one of the possible approaches to describe and execute your strategy. If you do quality metrics on the operational level, then BSC might not be an appropriate tool; still I would recommend having clear answers to these questions:

• Why do you need to measure the quality?
• What aspects of quality are important to your customers?
• How exactly is the company planning to achieve current business goals?
• What quality-related problems need more attention?
• What are the business challenges of your company right now?

Quality indicators help to make the goals more specific.
Source: Quality Scorecard.

In BSC Designer: go to the “KPIs” tab. There, you can formalize your business goals and align KPIs with these goals.

2. Visualize Your Quality Challenges and Their Reasons

Quality control or speaking more generally, quality assurance, is not only about having a set of indicators and reporting them. The idea is to have an understanding of the processes in the organization, their strong and weak points and trying to prevent possible quality issues.

One needs to do a regular analysis of:

• Typical processes
• Quality challenges
• Their reasons
• Possible solutions

The result of this analysis can be formalized in the form of a flow chart.

An example of the process map that combines goals and indicators from the Quality scorecard as well as some custom graphical elements.
Source: Quality Scorecard.

In BSC Designer: go to the “Strategy Map” tab, create a new map and use it to visualize your process. Later, you will be able to visualize some

3. Balance Between Lagging and Leading Indicators

Have a look at the flow charts created in the previous step. On each step, there is some input used and some result produced according to the defined standards. You can have quality indicators associated with those control points.

Those might be leading or lagging indicators.

• Lagging indicators tell you a story of what has happened. This information is useful, but you are just observing what happened, and you cannot change anything.
• In contrast, leading indicators reflect inputs that influence the process.

On your quality scorecard, you need to have both – leading and lagging indicators.

A goal ‘Quality Awareness’ quantified by two leading and one lagging indicator.
Source: Quality Scorecard.

In BSC Designer: select the type of indicator on the “Context” tab, leading and lagging indicators are calculated in different ways.

4. What are the Most Important Indicators?

You say:

• “All of them are important!”

The question is:

Are they equally important?

Reducing the defects number from 0,01% to 0,001% might slow down the whole production system, and while your clients were happy with a 0,01% defect rate, for them, other product parameters are much more important.

You need to find a balance and reflect the importance of each of the indicators on your scorecard.

Relevant weight specified for the leading indicator ‘Prevent critical quality problems’.
Source: Quality Scorecard.

In BSC Designer: you have a weight property for an indicator on the “Performance” tab. Learn here what weight means and how it is calculated.

5. What is the Performance of an Indicator?

Let’s say you have a 1% defect rate, then you optimized the process and achieved a 0.1% defect rate; finally, after an introduction of additional quality measures, you got a 0,01% defect rate on average.

My question is:

How was the performance of the respective quality indicator changing? Was it improved 10 times and then again improved 10 more times?

Not necessarily! From a business viewpoint, clients might be unhappy with a 1% defect rate, and really happy when you have a 0.1% defect rate and achieving a 0.01% defect rate may not have made any visible difference to them.

In real life:

• Some indicators have a linear performance function (you improve the value of an indicator 10 times, and the performance improves 10 times).
• Some indicators have non-linear performance, e.g., the performance of an indicator might grow slowly in the beginning and then the function converts into an almost flat line, meaning that further improvements don’t improve the end performance.

When designing your indicators, choose an appropriate performance function for them.

For example, on the quality scorecard discussed above, we have Time to develop prevention plan indicator measured in days that have a customized performance function:

• If we stay below the 6 days’ time frame, the performance is in the green zone.
• If we develop prevention plan for a longer time, then performance drops quickly to the red zone and then to 0%.

Non-linear performance optimisation function used for the indicator.
Source: Quality Scorecard.

In BSC Designer: there is an “Optimization” drop list on the “Context” tab for indicators. Change there the linear optimization function to a more relevant one or specify your own function.

6. The Total Score of the Quality Scorecard

When you have 50+ indicators on your quality scorecard, each with its own performance function, benchmarks, values, and weights, then theoretically you can calculate the performance of each category and then the performance of the whole scorecard.As a result, you will have a single number (some call it a “quality score”).

I know some top managers ask for a single quality score, but in most cases, it won’t make any sense as this number aggregates too much data. It looks nice in the reports, but it won’t give any meaningful information for a top manager.The better option is to group indicators into categories and report the performance of each category.

For example, from the scorecard discussed above, I can see how the organization performs in terms of learning efforts or internal improvements.

The total performance of the quality scorecard is calculated considering the performance of sub-items and their relevant weights.
Source: Quality Scorecard.

In BSC Designer: the program can calculate the performance figures for indicators, goals, perspectives and for the whole scorecard, but the end usage is up to you.

7. Plan Your Actions

Be sure that you have (preferably in writing) some plan aligned to each quality indicator.

• What will you do if this indicator gets into the red zone?
• Are there any routine procedures associated with this indicator?
• What’s the proposed budget?
• How long will it take to execute the action?
• How often do you need to revise this indicator?
• Who is responsible for an indicator?

For example, in the quality scorecard created above, we have a Company-wide training initiative, aligned with a Near miss reporting training goal where we have specified all the necessary details:

In BSC Designer: select an indicator or goal, click the “Initiatives” button, specify relevant details, budget, link to some documents, and choose a person responsible.

8. Build a Quality Dashboard

Dashboards help to visualize quality indicators. For some of them, you will need to use a gauge-style chart, for others the best option is to display a time chart with a trend line. A well-designed dashboard will give you a top-level view. Use it together with indicators visualized on the map.

An example of the performance dashboard that visualised key quality indicators and the Gantt diagram with initiatives.
Source: Quality Scorecard.

In BSC Designer: you can create a number of dashboards to visualize your indicators.

9. Track the Trend

Even if you check your indicators regularly, you overlook a negative trend. For that purpose, track the indicators’ value for a longer period of time. For example, taking a 3-month period as a base for calculations, you might see that an indicator is trending into a red zone. This will serve as an excellent early-warning signal for your team.

Analysis function forecasts the value of the indicator using its historical data.
Source: Quality Scorecard.

In BSC Designer: Use “Forecast Analysis” on “Analysis” tab to forecast indicators’ value.

10. Get Bad News Early

If some quality indicator has gotten into the red zone, make sure that all of the quality specialists involved get a notification about the problem immediately.

Also, make sure that your quality scorecard is updated with fresh data, and you are looking at up-to-date quality information.

In BSC Designer: setup notifications for the indicators on the “Alerts” tab. The software can send out a notification if an indicator has got into the red zone, or if an indicator was not updated on time with fresh data.

Summary

There are many stakeholders of quality that perceive it in different ways – prospective clients, customers, partners, our team, regulators.

• Good news: we can still quantify, measure, and even predict quality.

In this article, we discussed:

• Part 1. KPIs that help to measure, predict, and report quality
• Part 2. A Balanced Scorecard example that helps to align quality initiatives with overall business strategy
• Part 3. 10 improvement ideas for ongoing quality control

Feel free to share in the comments some interesting findings about measuring quality.

What’s next?

• Access templates. Sign-up with a free plan at BSC Designer for immediate access to 31 scorecard templates, including Quality Scorecard discussed in this article.
• Master skills. Learn how to break down ambiguous goals like “improve quality” and “increase resilience” into specific strategies.
• Automate. Learn what Balanced Scorecard software is and how it can make your life easier by automating strategy execution, KPIs, and strategy maps.

More About Strategic Planning

Strategic Planning Process:

Examples of the Balanced Scorecard:
Strategy Maps:

CEO | Trainer | Author

BSC Designer is a Balanced Scorecard software that is helping companies to better formulate their strategies and make the process of strategy execution more tangible with KPIs.