Quality Management of Statistical Processes Using Quality Gates

Quality Management of Statistical Processes Using Quality Gates Narrisa Gilbert Australian Bureau of Statistics May 2011

Overview • Background • What are Quality Gates • Attributes of Quality Gates • Six Components of Quality Gates • Outcomes of Quality Gates • Further information

ABS approach to quality management • What? • Reinforcing a high degree of credibility for the ABS and its outputs • Maintaining the relevance of ABS outputs • How? • Effective relationships with respondents • Processes that produce high quality outputs • Regular review and evaluation of statistical activities • Staff skilled and motivated to assure the quality of ABS outputs

Background • The Australian Bureau of Statistics produces a lot of statistics • On occasion the quality (accuracy) of data has been called into question at the last minute (right before release to the public) • Sometimes no issues are identified • Sometimes errors are discovered • Sometimes errors in data are identified after release

What are quality gates? “checkpoints or decision points at various strategic places in a statistical process at which the quality of the process at or up to that point is explicitly assessed”

Attributes of Quality Gates Planned in advance of issues occurring Designed to facilitate the detection, discussion and resolution of issues and problems with processes earlier Constantly reviewed and revised

Attributes of Quality Gates Collaborative Communicate openly the progress/issues associated with the collections Provide a model of accountability and responsibility for survey processes

Attributes of Quality Gates Knowledge Management Create a store of corporate knowledge Document and monitor issues and actions throughout the collection cycle

Six components of quality gates Placement Quality measures Roles Tolerance Actions Evaluation

1. Placement Assigning priorities Related to risk profile of collection cycle Want to detect errors ‘upstream‘ What can go wrong? When can this problem occur? What impact can it have?

Placement of Quality Gates Handover • For example Within UNECE (United Nations Economic Commission for Europe) Secretariat 2009, Generic Statistical Business Process Model, Version 4, April 2009, UNECE, Geneva.

Areas of risk changes to processes, systems, methodologies data transformations overlap / coordination / integration with other areas knowledge management

2. Quality measures Choose good indicators of potential problems Prioritise! Definitions May need to drill down Time series useful How would we know if this problem occurred?

3. Roles Owner area Sign-off person Operational person / Gate Keeper Stakeholders Gate definition Gate assessment / sign-off Who is responsible? Who will this affect?

4. Tolerance Driven by user requirements Consider size of natural variation, sampling errors, level of detail, importance of outputs Need to form expectations Historical data may help Predetermined! What is an acceptable level of quality?

5. Actions reflect extent of the problem reflect consequences “traffic lights” concept What will we do (if there is a problem)? Who needs to be informed?

6. Evaluation evaluation of processes evaluation of quality gates What has this information told us about our quality? How can we improve in the future?

Outcomes from Quality Gates Business processes well-defined and known by staff Responsibilities for processes and quality clearly defined Statistical risk explicitly identified & assessed

Outcomes from Quality Gates Explicit sign-off occurs at Quality Gates Quality Measures clearly defined and understood Errors identified earlier (and fixed)

Further information • Quality Management of Statistical Processes Using Quality Gates, Dec 2010, (cat.no. 1540.0) on the ABS website. • http://www.abs.gov.au/ausstats/[email protected]/mf/1540.0

Questions • Contact: • Narrisa Gilbert: [email protected]