How to Measure QA Success | Mailosaur

The benefits of quality assurance testing in software are widely accepted, but quantifying those benefits and optimizing performance can be tricky. Software developer performance can be measured by the amount and difficulty of code that they commit in a given sprint, but measuring the performance of QA engineers is harder when their success is measured by the lack of problems in software applications deployed into production.

In this post, we will look at some common techniques for measuring the effectiveness of your QA team, as well as how to measure the return on investment for automation and tooling.

How to Measure Team Effectiveness
There are many different ways to measure the effectiveness of quality assurance teams. For example, you could measure efficiency by how long it takes to develop and run a test or measure accuracy by looking at how many bugs made it into production. The ‘right’ metrics depends on your specific organization, but it’s generally a good idea to measure both efficiency and performance for a well-rounded guide for performance evaluations.

Measuring Defects
The whole idea behind QA processes is to reduce the number of defects between builds over the course of a project. While the total number of defects in a given project may depend on a variety of factors, measuring the rate of decline in the number of defects over time can show you how efficiently QA teams are finding and addressing the defects. This can be calculated by plotting the number of defects for each build and measuring the slope of the resulting line.

Plotting Defects per Build

An important exception is when a new feature is introduced, which may increase the number of defects found in the following builds. These defects should begin to steadily decrease over time until the build becomes stable, but you may need to plot each new feature as a separate line to account for these differences. The rate of decline in the number of defects following each new feature launch can then be analysed independently.

Measuring Time
Efficiency often boils down to the time that it takes to accomplish a task. While it may take a while to execute a test for the first time, subsequent executions should become much smoother and test times should begin to fall. You can determine the efficiency of a QA team by measuring how long, on average, it takes to execute each test in a given cycle. These times should decrease after initial testing and eventually plateau at a base level.

QA teams can improve these numbers by looking at what tests can be run concurrently or automated. For example, Mailosaur enables QA engineers to automate many email testing tasks via our API while Selenium enables QA engineers to automate common UI-related tasks. The use of these tools and techniques can also help produce metrics that can help to justify the cost of QA automation and tooling designed to improve efficiency.

Investing in QA Automation Tools
Automation is widely accepted as a way to improve quality assurance, but it can be difficult to quantify the return on investment in a way that justifies additional expenditures. For example, a Chief Financial Officer may want to know how many tangible dollars will be saved each month using testing automation tools rather than relying on the abstract notion of cost savings. Calculating these exact numbers can prove difficult given the nature of software testing.

The most basic way to demonstrate value when it comes to testing automation is to look at the number of hours worked. A manual tester might run tests for eight hours per day, while testing automation may be run twice as frequently for the same cost, which reduces the average cost of each testing hour. Since it costs less to fix bugs earlier in the development cycle, greater test coverage using the same budget can also reduce development costs over the long-run.

Beyond the financial benefits, QA automation helps simplify routine tasks and free up time for QA engineers to work on more challenging problems. Automated tests also help find hard-to-detect defects earlier when they are easier for the software development team to fix, which frees up the development team’s time to focus on features rather than bugs. Both QA engineers and software developers tend to be happier with testing automation in place.

Examples
Let’s look at a real example: A senior QA engineer might write automated tests for eight hours per day at $75.00 per hour while a junior QA engineer might write run traditional tests for eight hours per day at $50.00 per hour. Since the senior QA engineer tests can be run for 16 hours instead of just the eight hours worked, the cost per testing hour is reduced to $37.50, which is actually cheaper than paying the $50.00 per hour to a junior QA engineer.

Another example might be looking at a tool like Mailosaur to automate email testing. If manual testing for certain email tasks takes 20 hours, and Mailosaur can cut that down to 10 hours, then the return on investment can be calculated by multiplying the 10 hours saved by the QA engineer hourly rate and comparing that to the cost of Mailosaur. Mailosaur may also have a lower error rate and catch more bugs than manual testing given its automated nature.

Conclusion

Quality Assurance has well-known benefits in software development, but it can be challenging to evaluate the performance of QA engineers and determine the return on investment when it comes to investing in automation tools. By using the techniques outlined in this article, you can gauge the efficiency and effectiveness of QA engineers and calculate some solid numbers to justify the ROI of a given QA automation tool.

Try Mailosaur for free now to see how it can help improve your email testing automation.