Quality Department Performance Indicators

1. You are here:
2. Engineering
3. Quality Department
4. Quality Department Performance Indicators

Maintained by

:

On this page

Executive Summary

KPI
Health
Status

Master Pipeline Stability

Attention

• Improved to 95% in December but dipped to 92% in January due to ongoing broken master incidents resulting from flaky tests, test selection gap, and runner infrastructure issues.

Review App deployment success rate

Okay

• Increased to 94% after taking corrective actions to deploy Review App more sustainably
• Improved team process on managing GKE cluster upgrade
• Improved Review App resource cleanup with monitoring in place for automated health check

Time to First Failure

Okay

• Reduced to 16 minutes through the ongoing work to improve pipeline efficiency
• Please note that the current average TtFF does not account for Review App child pipeline failures, as we allowed pipelines to ignore them temporarily while we worked on resolving Review App deployment issues. We will resume accounting for Review Apps child pipelines once Review App deployment success rate is restored to above 95%.
• Sustainable Review App deployments and critical path optimizations will help keep the average TtFF low once we resume accounting for Review App child pipelines

S1 OBA

Okay

• Under target at 34 days
• Open issues decreased from 39 to 19.
• Two oldest performance bugs are getting auto-scheduled for upcoming milestones but haven’t been resolved yet.
• Auto-scheduling has been helpful in reminding product teams to schedule S1s

S2 OBA

Okay

• This metric was changed to use median rather than average as a better representation of OBA.
• S2 bug refinement is underway to stabilize open age and reduce backlog. Backlog is decreasing.
• Closed 40 customer-impacting S2 bugs in [FY23Q4](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1518).

Quality Team Member Retention

Okay

• Above target

Software Engineer in Test Gearing Ratio

Attention

• 62% of gearing ratio target.

Key Performance Indicators

Master Pipeline Stability

Measures the stability of the GitLab project master branch pipeline success rate. A key indicator to the stability of our releases. We will continue to leverage Merge Trains in this effort.

Target: Above 95%

Health: Attention

• Improved to 95% in December but dipped to 92% in January due to ongoing broken master incidents resulting from flaky tests, test selection gap, and runner infrastructure issues.

DRI: Daniel Croft

URL(s)

Review App deployment success rate

Measures the stability of our test tooling to enable end to end and exploratory testing feedback.

Target: Above 95%

Health: Okay

• Increased to 94% after taking corrective actions to deploy Review App more sustainably
• Improved team process on managing GKE cluster upgrade
• Improved Review App resource cleanup with monitoring in place for automated health check

DRI: Daniel Croft

URL(s)

Time to First Failure

TtFF (pronounced “teuf”) measures the average time from pipeline creation until the first actionable failed build is completed for the GitLab monorepo project. We want to run the tests that are likely to fail first and shorten the feedback cycle to R&D teams.

Target: Below 15 minutes

Health: Okay

• Reduced to 16 minutes through the ongoing work to improve pipeline efficiency
• Please note that the current average TtFF does not account for Review App child pipeline failures, as we allowed pipelines to ignore them temporarily while we worked on resolving Review App deployment issues. We will resume accounting for Review Apps child pipelines once Review App deployment success rate is restored to above 95%.
• Sustainable Review App deployments and critical path optimizations will help keep the average TtFF low once we resume accounting for Review App child pipelines

DRI: Daniel Croft

URL(s)

S1 OBA

S1 Open Bug Age (OBA) measures the total number of days that all S1 bugs are open within a month divided by the number of S1 bugs within that month.

Target: Below 60 days

Health: Okay

• Under target at 34 days
• Open issues decreased from 39 to 19.
• Two oldest performance bugs are getting auto-scheduled for upcoming milestones but haven’t been resolved yet.
• Auto-scheduling has been helpful in reminding product teams to schedule S1s

DRI: Vincy Wilson

S2 OBA

S2 Open Bug Age (OBA) measures the total number of days that all S2 bugs are open within a month divided by the number of S2 bugs within that month.

Target: Below 250 days

Health: Okay

• This metric was changed to use median rather than average as a better representation of OBA.
• S2 bug refinement is underway to stabilize open age and reduce backlog. Backlog is decreasing.
• Closed 40 customer-impacting S2 bugs in FY23Q4.

DRI: Vincy Wilson

URL(s)

Quality Team Member Retention

This is a subset of an existing KPI. Please see the definition for the parent KPI.

We need to be able to retain talented team members. Retention measures our ability to keep them sticking around at GitLab. Team Member Retention = (1-(Number of Team Members leaving GitLab/Average of the 12 month Total Team Member Headcount)) x 100. GitLab measures team member retention over a rolling 12 month period.

Target: at or above 84%

This KPI cannot be public.

Health: Okay

• Above target

URL(s)

Software Engineer in Test Gearing Ratio

Amount of Software Engineers in Test against the targeted gearing ratio We have a detailed gearing prioritization model that informs important us which group we will hire an SET first.

Target: At 50 Software Engineers in Test

Health: Attention

• 62% of gearing ratio target.

DRI: Vincy Wilson

Regular Performance Indicators

Time to First Failure p80

TtFF (pronounced “teuf”) measures the 80th percentile time from pipeline creation until the first actionable failed build is completed for the GitLab monorepo project. We want to run the tests that are likely to fail first and shorten the feedback cycle to R&D teams.

Target: Below 20 minutes

Health: Attention

• Track this metric in addition to average starting FY23Q4
• Plan to optimize selective tests in place for backend and frontend tests

DRI: Daniel Croft

URL(s)

Merge request pipeline duration

Measures the average successful duration for the GitLab project merge request pipelines. Key building block to improve our cycle time, and effiency. More pipeline improvements and critical code paths planned.

Target: Below 45 minutes

Health: Unknown

• Metric is not accurate at the moment due to a bug with the pipeline duration calculation. Fix is WIP, see this issue for details.

DRI: Daniel Croft

URL(s)

Average duration of end-to-end test suite

Measures the speed of our full QA/end-to-end test suite in the master branch. A Software Engineering in Test job-family performance-indicator.

Target: at 90 mins

Health: Attention

• Good month of October, but spiking up again in November
• Long-term – rethinking design of test execution to innovate and drive significant reduction

DRI: Vincy Wilson

URL(s)

Average age of quarantined end-to-end tests

Measures the stability and effectiveness of our QA/end-to-end tests running in the master branch. A Software Engineering in Test job-family performance-indicator.

Target: TBD

Health: Unknown

• Chart to track historical metric is broken. Our visibility is limited to a table of currently quarantined end-to-end tests.
• We made good progress on our FY23Q2 OKR to un-quarantine all quarantined tests past SLO; will continue burning down this backlog in FY23Q3.
• Collaborating with product teams to fix bugs blocking end-to-end test execution via the cross-functional prioritization process.
• We made good progress on our FY23Q2 OKR to un-quarantine all quarantined tests past SLO; will continue burning down this backlog in FY23Q3.

DRI: Vincy Wilson

URL(s)

S1 OCBA

S1 Open Customer Bug Age (OCBA) measures the total number of days that all S1 customer-centric bugs are open within a month divided by the number of S1 customer-centric bugs within that month.

Target: Below 30 days

Health: Okay

• Newly created performance indicator, target needs to be reviewed periodically.
• Under target for 3 consecutive months

S2 OCBA

S2 Open Customer Bug Age (OCBA) measures the total number of days that all S2 customer-centric bugs are open within a month divided by the number of S2 customer-centric bugs within that month.

Target: Below 250

Health: Attention

• Newly created performance indicator, target needs to be reviewed periodically.
• Customer-impacting S2 bugs are been actively worked on in FY23 Q4.

Quality Budget Plan vs Actuals

This is a subset of an existing KPI. Please see the definition for the parent KPI.

We need to spend our investors’ money wisely. We also need to run a responsible business to be successful. Latest data is in Adaptive, data team importing to Sisense in FY22Q2

Target: See Sisense for target

Health: Unknown

• There is currently a data lag to resolve

URL(s)

Quality Handbook MR Rate

This is a subset of an existing KPI. Please see the definition for the parent KPI.

The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching /source/handbook/engineering/quality/** over time.

Target: Above 1 MR per person per month

Health: Problem

• Declining in last 3 months
• Will require focus in FY24Q1 to increase

Quality MR Rate

This is a subset of an existing KPI. Please see the definition for the parent KPI.

Quality Department MR rate a key indicator showing how many changes the Quality Department implements directly in the GitLab product. It is important because it shows our iterative productivity based on the average MR merged per team member. We currently count all members of the Quality Department (Director, EMs, ICs) in the denominator because this is a team effort. The full definition of MR Rate is linked in the url section.

Target: Above 10 MRs per Month

Health: Attention

• Increased in October & November. Slight decline in December
• We made progress on increasing codebase maintainers and will continue along with pushing for smaller iterations

URL(s)

Quality Department Promotion Rate

The total number of promotions over a rolling 12 month period divided by the month end headcount. The target promotion rate is 12% of the population. This metric definition is taken from the People Success Team Member Promotion Rate PI.

Target: 12%

Health: Okay

• Under target for 4 months, which is expected after being above target for 8 months

Quality Department Discretionary Bonus Rate

This is a subset of an existing KPI. Please see the definition for the parent KPI.

The number of discretionary bonuses given divided by the total number of team members, in a given period as defined. This metric definition is taken from the People Success Discretionary Bonuses KPI.

Target: at or above 10%

Health: Attention

• We have not been close to target for 10 months, not worrisome for the amount of new people in the department, but we have to pay attention in the following months.

Other PI Pages

Legends

Health

Value
Level
Meaning

3

Okay

The KPI is at an acceptable level compared to the threshold

2

Attention

This is a blip, or we’re going to watch it, or we just need to enact a proven intervention

1

Problem

We’ll prioritize our efforts here

-1

Confidential

Metric & metric health are confidential

0

Unknown

Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists – a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files.
Each – denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property
Type
Description

name
Required
String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by ” – ” (Ex. {Stage Name}:{Group Name} – {PI Type} – {PI Name}

base_path
Required
Relative path to the performance indicator page that this (K)PI should live on

definition
Required
refer to Parts of a KPI
parent
Optional
should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.

target
Required
The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.

org
Required
the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)

section
Optional
the product section (Ex: dev) as defined in sections.yml
stage
Optional
the product stage (Ex: release) as defined in stages.yml
group
Optional
the product group (Ex: progressive_delivery) as defined in stages.yml
category
Optional
the product group (Ex: feature_flags) as defined in categories.yml
is_key
Required
boolean value (true/false) that indicates if it is a (key) performance indicator

health
Required
indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.

health.level
Optional
indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI.

health.reasons
Optional
indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.

urls
Optional
list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url

funnel
Optional
indicates there is a handbook link for a description of the funnel for this PI. Should be a URL

sisense_data
Optional
allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.

sisense_data.chart
Optional
indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard
Optional
indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard
Optional
indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed
Optional
indicates the Sisense embed version. For example: v2
sisense_data_secondary
Optional
allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart
Optional
Same as sisense_data.chart
sisense_data_secondary.dashboard
Optional
Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard
Optional
Same as sisense_data.shared_dashboard
sisense_data_secondary.embed
Optional
Same as sisense_data.embed
public
Optional
boolean flag that can be set to false where a (K)PI does not meet the public guidelines.

pi_type
Optional
indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)

product_analytics_type
Optional
indicates if the metric is available on SaaS, SM (self-managed), or Both.

is_primary
Optional
boolean flag that indicates if this is the Primary PI for the Product Group.

implementation
Optional
indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.

implementation.status
Optional
indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI.

implementation.reasons
Optional
indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.

lessons
Optional
indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.

lessons.learned
Optional
learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned

monthly_focus
Optional
indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.

monthly_focus.goals
Optional
indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal

metric_name
Optional
indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.