Quality Gates and Build Failure

Introduction

A Quality Gate is a check on a code quality fact that must be enforced before releasing and eventually, before committing to source control.
A Quality Gate can be seen as a PASS/FAIL yardstick for software quality.

A dozen of default Quality Gates are proposed by CppDepend related to measures like technical debt amount, code coverage or amount of issues with particular severity.

Notice that special red / yellow / green losange icons shows Quality Gates status: fail / warn / pass.

Video tour

Quality Gates and Build Failure

Quality Gates can be used to fail the build when certain yardsticks are not-verified.
When at least one Quality Gate fails, CppDepend.Console.exe returns a non-zero value that can be used to fail the build.

Creating and Customizing Quality Gates

What makes CppDepend unique is that a Quality Gate is a C# LINQ Query that can be easily created, edited and customized.
For example if you wish to enforce a certain amount of code coverage through a Quality Gate, you can just write:

// <QualityGate Name="Percentage Code Coverage" Unit="%" />
failif value < 70%
warnif value < 80%
codeBase.PercentageCoverage

Notice also the special clauses failif (mandatory clause) and warnif (optional clause) that are specific to CppDepend CQLinq.
codebase.PercentageCoverage is actually a C# statement that CppDepend evaluates.

Notice the special header if comment that defines the Quality Gate name and unit.
The unit here is the string “%”. It can be optionally repeated after after the thresholds for readability purpose.

A Quality Gate can also determine a measure based on the diff since baseline.
For example the Quality Gate below is ensuring that the overall technical-debt doesn’t grow beyond certain thresholds, since the baseline.

Notice that a <Description> tag can be used to describe the Quality Gate for non-developers.

// <QualityGate Name="New Debt since Baseline" Unit="man-days" />
failif value > 2 man-days
warnif value > 0 man-days
let debt = Issues.Sum(i => i.Debt)
let debtInBaseline = IssuesInBaseline.Sum(i => i.Debt)
select (debt - debtInBaseline).ToManDay()

//<Description>
// This Quality Gate fails if the estimated effort to fix new or worsened
// issues (what is called the *New Debt since Baseline*) is higher
// than 2 man-days.
//
// This Quality Gate warns if this estimated effort is positive.
//
// Debt documentation: http://www.CppDepend.com/docs/technical-debt#Debt
//</Description>

Finally notice that when defining a Quality Gate threshold, the keyword value must be replaced with the keyword count
if the LINQ query is returning rows instead of returning a scalar.
This is useful to make the Quality Gate result more informative when it makes sense.

Exploring Quality Gates Status

The set of Quality Gates status is queryable from C# LINQ queries.
The Dashboard summarizes the Quality Gates status. A single click generates a LINQ query that shows the detailed status for each Quality gate.
Quality Gates statuses are also provided in HTML+js report.

Prioritizing issues fix and the Breaking-Point metric

The Breaking-Point of an issue or of a set of issues,
is the time point from now to when the estimated cost-to-fix the issue(s) will reach the estimated cost to leave the issue(s) unfixed.

The breaking point is the debt divided by the annual-interest.
For example if the estimated cost-to-fix the debt is equal to 10 man-days and the estimated annual-interest is equal to 2 man-days per year,
then the breaking point is equal to 5 years from now.

Notice that a breaking point which is lower than a year means that during the next 12 months, it is estimated that it would be cheaper to fix the debt than not to fix it.

Notice also that a breaking point is not measured through man-time like debt or annual-interest (a man-month or a man-year),
but rather through regular duration (months or years). Breaking point values are typed with TimeSpan.

When it comes to prioritizing issues to fix first, the issue severity is an important parameter.
As a reminder: the severity is the discrete measure of the annual-interest.
Hence the higher the annual-interest, the more important it is to fix.

However, given a certain severity level, not all issues are equal. Some will demand more effort to fix. This is estimated through the technical-debt measure.
Hence, to estimate the Return On Investment (ROI) of an issue fix, it makes sense to estimate the debt divided by the annual-interest.
This estimation is the breaking-point for which the lower the value, the higher the ROI.

Let’s specify that in the set of default rules, issues that are relative to new problems since the baseline,
such as API breaking changes, code elements quality getting even worse, new code elements not tested…
are issues which produce a higher annual-interest and thus a higher severity than the other issues.
This complies with the best practice to fix recently introduced issues first.