Evolving quality improvement support strategies to improve Plan–Do–Study–Act cycle fidelity: a retrospective mixed-methods study

By retrospectively capturing the experience of the programme and project teams, we aim to provide insight into the reality of using the PDSA cycle method and providing support to teams to do so. The overall intention of the paper is to support future programmes and project teams in using the method effectively to improve patient care.

This study explores the PDSA cycle conduct of front-line healthcare improvement teams supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) Northwest London (NWL) programme 2008–2013. It takes advantage of the documentation collated by the CLAHRC NWL programme to conduct a retrospective study. Specifically it aims to (1) assess the fidelity of a range of PDSA cycles documented in real time by front-line improvement teams; (2) determine whether any change in PDSA fidelity occurred over time; and (3) explore the strategies deployed by the programme team to support and improve the use of PDSA cycles.

The Plan–Do–Study–Act (PDSA) cycle method is widely recommended as an effective approach to QI; however, previous research has demonstrated that the fidelity of the method reported in peer-reviewed literature is low 3 and barriers are encountered in its use. 4–6 PDSA cycle fidelity has been defined as the degree to which a PDSA cycle is carried out in accordance to the guiding principles of its use ( table 1 ). 3 Measuring fidelity of the PDSA cycles demonstrates whether the method has been used as intended, which in turn can inform assessments as to whether its desired benefits have been achieved: learning to inform the evolution of a change idea to support achievement of a stated aim. 7 There is little overarching empirical evidence, however, of the fidelity of PDSA used by front-line teams or understanding of factors that may influence the fidelity of PDSA cycle use. 8

Quality improvement (QI) approaches continue to grow in popularity in healthcare. This increased emphasis and uptake of the approaches needs to be balanced by an understanding of how to ensure their effective use to enable the delivery of improvements in patient care. Without such assurances there is a danger that QI remains a ‘slogan of intent’ to improve quality rather than an authentic application of the concepts in practice. 1 2

Project reviews and training materials were initially reviewed to identify high-level themes relating to PDSA cycle conduct and QI support strategies. These themes informed the interview questions with QI support team members. Selected training materials were brought to interviews to act as prompts. Drawing on both documentation and interview transcripts, a full thematic inductive analysis using the constant comparative method was undertaken to identify themes relating to the QI support strategies used and the experiences of both the QI teams and QI support team. Detailed open-coding of themes within text were identified along with code definitions. These were grouped to high-level categories before further conceptualisation within each category. Coders met to discuss and refine coding in an iterative manner. All data (documentation and interview transcripts) were also coded to project round(s) that they related to. Themes were discussed with other authors and other members of the QI support team to support sense-making of the analysis in light of the historical development and conduct of the programme, and to discuss and clarify gaps in analysis. Full roles of the authors in the data analysis are presented in online supplementary appendix 1 .

The quantitative outputs for the measures of fidelity from the first stage of analysis were divided by the year the project teams were initiated. A one-way analysis of variance and post-hoc t-tests were first used to determine change in the mean number of PDSA cycles conducted per project overtime. χ 2 tests and a subsequent trend test, the Marascuilo procedure, were used to assess the significance in changes observed for each fidelity assessment over time (see online supplementary appendix 1 for further details). 17

Two reviewers (CM and LL) first coded a third of the 421 cycles against the principles in Microsoft Excel. Before coding, they were familiar with and had discussed the principles outlined in Taylor et al’s 3 systematic review of PDSA cycles (of which CM was an author). They also reviewed a small number of PDSA cycles together to learn how to apply the framework. Both reviewers had completed QI training on a range of methods, including PDSA cycles. The reviewers were blinded from the project name and round, and while they may have delivered training for the teams they were not involved in the delivery of the projects. Intercoder reliability, as indicated by Cohen’s kappa, ranged between 1 and 0.77, with percentage agreement between 100% and 82%. Discrepancies were resolved by discussion and consensus and a shared understanding was developed. The remainder of the cycles were then coded by one reviewer (CM).

A structured framework was used to assess the fidelity of PDSA cycle use against the key principles of the method 3 ( table 1 ). The documented PDSA cycles of the CLAHRC NWL project teams were assessed by deductive content analysis against this framework. 15 16

Feedback from project teams about the use of PDSA cycles (and other QI methods) and QI support strategies was collated throughout the programme. This included project reports (n=96, including details of how QI methods were used); minutes from formal project review meetings (n=84, including discussion of project team perceptions of QI methods and QI support strategies, held at 6 monthly interviews); and teaching and support materials (including planning documents, slides and activity handouts). In total, 180 project documents were assessed (number of documents review per round: R1=11, R2=80, R3=89) and training materials from 20 events teaching or referring to PDSA cycles. In addition, to triangulate data derived from the document analysis, interviews were conducted retrospectively after all three rounds of projects had been completed. Three programme team members who had been involved in teaching and support of PDSA across all three rounds of the programme were interviewed. The data from the interviews were intended primarily to clarify and explain our findings.

Each project documented their use of PDSA cycles in real time on an online tool, the Web Improvement Support for Healthcare system. 14 A total of 421 PDSA cycles were documented and are included in the study.

The projects were conducted over the three rounds of projects, each round lasting 18 months, with 6 starting in April 2009, 16 in 2010 and 17 in 2011 ( figure 1 ). 13 Project team members tended to be QI novices, with little or no prior QI experience. No entire team participated in more than one round, although a small number of individuals participated in different project teams over more than one round. The initiation of projects annually was purposeful so that teams overlapped and shared experiences, and so that modifications to the QI support strategies could be made based on feedback of both the programme and project teams.

Between 2009 and 2012 the NIHR CLAHRC NWL programme supported 39 projects. 9–12 Using a QI collaborative structure, a central programme team provided training and support to help front-line improvement teams use a suite of QI methods, including PDSA cycles, to improve the quality of healthcare through the implementation of research evidence into practice. The QI support team were made up of members of the programme and were from a range of backgrounds—clinical, managerial, information analysts and researchers. Each project team was assigned a main point of contact in the QI support team. The majority of the QI support team stayed the same throughout the programme, with one senior member moving on and being replaced after 2 years, and four additional junior posts starting in 2010. The support provided by the programme is referred to as ‘QI Support Strategies’.

“We didn’t want them to rush off and change practice, we wanted them to sort their measures out and get their baselines and then test changes.” (Interviewee 2)

In recognition of these challenges, the role of QI support team shifted from an arm’s length advisory role to working much more closely with the project teams. This included greater presence of QI support staff within projects to facilitate structured discussion about how changes should be tested and role-model the use of PDSAs. Teams were also supported to develop aims and measure definitions earlier so that timely data were available to inform PDSA cycles. 18 Additionally, the programme invested in providing greater support and training to the QI support team members themselves.

“I don’t know if there would be many teams who would use it in a daily project meeting – sit around and say well this was the PDSA we said we were going to do – how did it work out? I think it was still a bit more of one person’s responsibility.” (QI support team member, interviewee 1)

Project team members were expected to work together to design, conduct and review PDSAs with the intention of all members sharing their professional perspectives. However, this was reported to rarely occur, and the use of PDSA was often delegated to an individual team member. Practical time constraints or competing priorities also presented challenges to completing PDSA in real time and with high fidelity. The method was often used retrospectively to frame past actions rather than prospectively plan and test changes iteratively. This meant that principles such as use of predictions or consideration of scale were not applied.

Application of use referred to the way teams went about using the method in practice. It included social challenges, such as bringing a team together to discuss a PDSA, or technical challenges such as the difficulties collecting and analysing data. It was interlinked with the other factors as true understanding could only be achieved through experience of application in practice and the appreciation that the method may be simple in theory but hard to apply in practice.

As outlined in table 3 , original teaching was frontloaded at the beginning of the programme and delivered by external QI experts. Analysis of teaching materials from rounds 2 and 3 demonstrated a different approach that staggered teaching of the method over time, delivered by the QI support team. Initial training sessions focused on the rationale of using the method, and over time evolved to conducting a single PDSA well, before considering iterative chains of PDSA and use of data over time. Training sessions were also designed to include more relevant examples of PDSA cycle use, and individuals with past experience in projects were invited to present them. These examples were perceived to be of greater relevance to the new project team members, and less ‘push back’ was experienced.

In some cases, project team members were observed to embrace the PDSA method as an alternative way of working that empowered them to make rapid changes in their local settings. In these cases, however, there was a tendency to ‘PDSA everything’ with little critical consideration of whether the method was being applied well, nor in following the premise of iterative development of a change over time.

Understanding referred to the capability to use the methods and included knowledge of the concept and also the specific principles of the PDSA method. Understanding and intention were distinct factors but interlinked: some teams may have had little intention to use the method as they did not understand it; some may have understood the method and consciously intended not to use it; and some may have intended to use but had insufficient understanding to use with high fidelity.

Time was also invested for QI support staff to facilitate debate and critical thinking in regard to the method’s use. These discussions were seen as important for project teams to cognitively engage with the PDSA method and position its use within their prior experience and knowledge.

“I think the biggest changes we’ve made is trying more to appeal to the hearts and minds of people, so rather than trying to explain it as a technical process was trying to appeal to why might you want to do this? Why might it be useful for your projects and for patients?” (QI support team member, interviewee 1)

To manage expectations in relation to the use of QI methods, rounds 2 and 3 were required to attend introductory workshops prior to applying for funding and support, and the application process required them to demonstrate their intention to use and initial understanding of QI methods. Changes to training sessions were also made which reflected a recognition that the effective use of PDSA required more than just technical knowledge, but a willingness and motivation to use the method and to change previous ways of working.

There was also a reported perception that the use of PDSA cycles was for the benefit of the QI support team rather than adding value to the project team itself. Areas such as documentation and data collection were seen as a form of programme assurance, rather than as mechanism to help the team learn, and therefore inhibited motivation.

A lack of intention to use the method was reflected in instances of low levels of use and fidelity of PDSA cycle. Qualitative analysis suggested this was influenced by the team’s beliefs and plans on how to tackle change and improvement, as well as their understanding of why the method could be helpful. Some teams had predefined intentions on how to conduct their projects, and clinical academic project team members were also reported to have raised concerns about the iterative nature and small sample sizes that PDSA cycles used.

As the QI support team gained experience and expertise, they recognised that the introduction of PDSA methodology required a fundamental change to how project team members thought about and approached change. This was clearly observed as changes in the reviewed training materials. Interviewees reported that, in light of the observations and the learning gained by the QI support team, deliberate actions were taken to improve the support for PDSA cycle conduct. Table 3 provides details on the original and revised QI strategies, and their reported consequence. Online supplementary appendix 3 provides additional supporting quotes.

By assuming that PDSA was easy and that project teams would be receptive to its use, the original QI support strategies failed to address the challenges encountered, particularly in the first round of the projects. The QI support strategies in round 1 ( table 3 ) were felt to have exacerbated these issues and were seen as a contributing reason for low levels of understanding and intention resulting in the low levels of PDSA use and PDSA fidelity identified in the quantitative analysis for round 1.

Overall, thematic analysis of all data identified three areas of challenge for project teams using PDSA: intention to use, understanding of how to use and the application in practice. They were evident through three corresponding assumptions, described by interviewees, in designing the original QI support strategies: a belief that people would be motivated and willing to use PDSA, that PDSA was easy to understand, and that PDSA was easy to apply in practice.

Improvements in fidelity were observed across project rounds for all PDSA cycle principles, except for the presence of a learning activity within PDSA cycles which was high (above 98% of cycles) across all three rounds ( table 2 ). These improvements were statistically significant for documentation (all PDSA cycle stages documented, p<0.001, moderate improvement 50%–77%; ‘Study’ documented in past tense, p<0.001, moderate improvement 67%–92%), predictions (explicit prediction documented, p=0.001, modest improvement 0%–18%) and iterative cycles (PDSA cycle within iterative series of cycles, p<0.001, substantial improvement 0%–60%). Improvements were seen for incremental scale and use of regular data over time, but these findings were not statistically significant. The seven cycles adhering to all indicators of fidelity were all from final round projects. Online supplementary appendix 2 presents the full statistical results.

Over the period of study, 2% (7/421) of PDSA cycles reviewed adhered to all six measures of fidelity, 12% (49/421) adhered to >5 principles and 29% (121/421) adhered to >4 principles as described in the framework. Table 2 presents individual results by each measure of fidelity. Only PDSAs with full documentation were included in full fidelity analysis (299), the rest (122) being excluded from further analysis.

A total of 421 PDSA cycles were documented and included in the study. There was a statistically significant increase in the mean number of PDSA cycles initiated by project teams across the three rounds ( figure 2 ) (p<0.05) ( online supplementary appendix 1 provides further details on the analyses).

Discussion

Over a three-and-a-half-year study period, moderate yet significant improvements were seen in the number of PDSA cycles conducted and the fidelity of these cycles against the key principles of the method. However, across the total sample of projects, PDSA cycle fidelity remained low with key principles of the method frequently not met. The study presents a theoretical framing and practical solutions to support better use of the PDSA method. It suggest that project teams’ intention, understanding and application of the PDSA cycle method are three areas in which QI support teams should consider when supporting the method’s use.

The study reiterates previous findings suggesting that the PDSA cycle methods,3 6 and QI methods in general,19 20 are not always applied as they are intended. The study also provides a detailed reflection on how QI methods are introduced influences their uptake and use, adding to the literature on the influence of context on QI approaches.21–24 This provides empirical grounding to support recent claims cautioning against the oversimplification of QI methods when they are taught and adopted into mainstream healthcare practices, demonstrating that the full benefits of these methods are often not realised25 26 and suggesting that challenges can be accentuated when the use of QI methods is new to individuals or teams.27

The actions taken by the QI support team to revise the QI support strategies align with improvements in fidelity of PDSA cycles. Given some limitations of the retrospective nature of this work, findings about the relationship between changes in QI support strategies and improvements in PDSA fidelity are intended to be exploratory (hypothesis-generating) rather than indicative of causality (hypothesis-confirming). The programmed yearly project initiation cycle allowed an iterative approach to be taken by the QI support team to respond to challenges faced and revise QI support strategies over time. The QI support strategies were revised in the recognition that developing intention to use PDSA, understanding of how to use it and mastery of its application in practice are a gradual and negotiated process.

Of note, the measures of fidelity that did not see significant improvements were those requiring users to revisit the method, including the increasing scale and use of data over time. Even with an adequate level of intention and understanding, these principles are arguably more complex and harder to achieve as they require skills and behaviours to work effectively as a multidisciplinary team to make decisions and plan between cycles, as well as the application of ‘measurement for improvement’ principles.

In establishing a rationale for the changes in fidelity over time, we also considered other changing contextual factors. Over the study period, the majority of the QI support team stayed with the programme and were likely to have gained in skills and competencies over this time. A small number of project team members also moved between teams over rounds. This could provide an alternative explanation as individuals increase experience and understanding of how to use PDSA over time, although perceptions from the QI support staff indicated that disengagement and misunderstanding of the method were equally likely to persist over rounds for some individuals. The authors are not aware of any other major contextual influences that happened during the study period. The extent of these or other contextual influences remains to be tested in future research.

Implications

This study reinforces growing research that emphasises that the use of QI methods is not simple. The use of QI methods must be considered as complex sociocultural interventions that require significant technical and social skills. This understanding needs to inform future use and the design of QI support strategies and PDSA education.

Through the challenges of intention to use and understanding and application of the methods in practice, this research raises questions about the most appropriate teaching, training and support mechanisms required for effective use of QI methods. The findings present new learning to inform the design, delivery and evaluation of QI training including PDSA cycles. Frequently QI training is perceived as deliverable in a short period of time28 29; however, existing studies have demonstrated the limited impact of such approaches in the rigorous and effective use of QI methods.27 30

Previous studies have identified the challenges of adopting new methods into existing organisational cultures and practices,19 20 31–33 and that to use QI methods well requires people to adopt fundamentally different ways of working.31 32 Introducing QI approaches into new settings needs to be carefully designed and delivered to consider how to support the introduction of behaviours that are often counter to prevailing organisational norms. The idea of conducting pragmatic and scientific tests of change locally to ensure that interventions were fit for purpose in a particular setting occupies a middle ground between the rigour of traditional academic research and the pace of change in healthcare organisations. While PDSA has the potential to bridge between these two worlds, it also presents a very different way of working which was not readily accepted or implemented by teams using the method.

Future work and limitations

This is the first study to provide a detailed assessment of a large number and range of PDSA cycles documented in real time during an improvement project. The fidelity assessment framework presented in this paper could be used in the future to provide a formative assessment of PDSA fidelity and provision of real-time feedback to project teams. This can support teams to identify and respond to factors within their local context and improve PDSA cycle conduct.

While this research would ideally have been conducted prospectively, the challenge of PDSA use and how to improve it only drew the attention of programme leaders, and researchers, as the work progressed.34 A resulting limitation is that interviews were only conducted with the three QI support staff who had been present and involved in teaching across the different project rounds and were still working with the programme at the time of the study. To counter this challenge, common themes were drawn from triangulating data from project team reports and review meeting minutes, training material and QI support staff interviews to ensure findings were reflective of the project team’s perspectives as well as the QI support team. The theories and strategies proposed in this paper require further investigation and, in particular, prospective application to assess if improvements in fidelity can reliably be achieved in practice.

Additionally, a limitation is that the study was reliant of PDSA cycles documented by front-line teams. While this provides a greater depth of insight to published reports of PDSA,3 it provides only a partial and selective reflection of how PDSAs were used in practice. Further research is needed to observe actual PDSA conduct in practice and to understand the perspective of front-line QI teams.

A further limitation of the study is that PDSA cycles were only assessed quantitatively against the principles of use, with no qualitative assessment of the nature of changes made, nor the success of the learning and adaptions introduce through subsequent cycles. The principles can therefore be considered necessary, but not sufficient, to determine the quality of PDSA cycle use. As such, this study simply reports on a minimum standard of PDSA fidelity. The findings demonstrate that engaging and motivating people to use PDSA at all and achieving these minimum standards in themselves are challenging, and therefore provide learning to others. Future research would, however, benefit from including additional work to understand how change ideas are adapted, with and without success, over time.7