The Cult of Quality Matters

In The Trouble with Rubrics, Alfie Kohn writes “consistent and uniform standards are admirable, and maybe even workable, when we’re talking about, say, the manufacture of DVD players.” Teaching is nothing like the manufacture of DVD players. Teaching online is nothing like the manufacture of DVD players. There’s no technical guide, no instruction manual, and not even a “missing manual,” for this work.

In early 2020, as colleges and universities struggled to adapt to remote learning, we weren’t surprised as many turned to rubrics, checklists, and pre-packaged online course templates. In a matter of weeks, dozens of new tools for this purpose emerged as schools rushed to adopt or adapt existing models. In a time of crisis, reliance upon rubrics and checklists to focus faculty efforts and provide a straightforward framework for action may indeed be understandable. Nevertheless, in our recent piece, “Counter-Friction to Stop the Machine,” we write, “Many of these approaches to instructional design reinforce and rely on systems that oppress students. On the ground at our institutions, the people who are tasked with promoting and supporting these models have little space or authority to push back on oppressive systems. Too often, the schools they work at have consistently devalued their labor and humanity.” Stock, prescriptive models for online learning exacerbate problematic power dynamics and structural inequities, and they deny students, faculty, and staff the agency necessary for the work of education.

Quality Matters, with its 42-point rubric, is one of the most pervasive and insidious examples of a standardized approach to the development of online courses. Insidious because it offers solutionism (or solution theater) more than an actual solution. Pervasive, because it has been so widely adopted, because it is looked to as a “gold standard,” because it is often universally implemented as a requirement across entire institutions, and because the items on the rubric itself are inflexible but also vague enough that they gesture to an all-seeing panoptic gaze. Unsuspecting teachers who encounter the QM rubric feel certain it’s right (because of how it’s rhetorically positioned) and that their teaching is being watched and controlled (because of how QM gets rolled out at institutions). They internalize authority, following rules that aren’t even laid out for fear of running aground on the rubric. In The Trouble with Rubrics, Kohn continues, rubrics are an “attempt to deny the subjectivity of human judgment…” The problem with the Quality Matters rubric is that it not only denies the subjectivity of human judgment, but that it appears to live in a world where there are no humans altogether.

When Quality Matters is critiqued, the QM organization has pointed out that their program is more than a rubric. It purportedly includes access to research, workshops and other programming, certification of peer-reviewers, and the actual peer-review and course certification process. These resources (particularly the full, annotated version of the QM rubric) are supposed to provide more context, support, examples, and direction.  Moreover, the use of the rubric, according to QM staff, is not meant to be prescriptive; faculty can take what they like and leave behind or adapt what doesn’t work. In addition, they point out that many of those who dislike QM are not victims of Quality Matters (or even of the Quality Matters rubric) but rather of the bad choices their administrations and schools have made when implementing the QM program.

All of the above may well be true. Since it’s impossible to catch a glimpse of the full QM program and the complete annotated rubric without becoming a member (which is costly), we can’t comment upon the ways in which the complete package decenters or deemphasizes the rubric or deepens practitioners’ understanding and use of the standards. Nor do we wish to comment upon who is to “blame” if adopters or their administrations misconstrue, misuse, or misrepresent to their campus community the purpose of the QM model and how it should be successfully implemented. There is certainly truth to the point that no tool is perfect, and we all bear a responsibility to think critically not only about the adoption of new tools and techniques but how we choose to implement them within our specific institutional contexts.

All that said, we do want to discuss what is at the center of this debate: the 42 standards that make up the Higher Education (Sixth Edition) Rubric. Even as we recognize that there is more to QM than just the rubric, it is impossible to ignore that the rubric sits at the center of the course certification process and that access to the rubric and standards (and course certification) is why schools adopt the QM model. QM’s own promotional materials on their website mention the rubric and standards again and again; their openly available research library associates every curated resource to a QM standard (and allows you to search by the standards); and their showcase of best practice is organized around the rubric and standards. We are not the only ones focusing upon and centering the rubric and standards; QM is most certainly doing this as well.

While QM may argue that the rubric is not meant to be prescriptive or that the goal is not to templatize every course, it relies on and encourages overwrought course structures, all in the name of “clarity.” In fact, the words “clear” or “clearly” appear over a dozen times in the rubric. Clarity certainly has some value, but more instructions does not necessarily lead to clarity. Writing longer syllabi and spelling out every expectation for interaction, policy compliance, technology requirements, and prerequisite knowledge leads to a learning environment where compliance becomes more important than human relationships. In an attempt to meet these standards, faculty are likely to err on the side of prescriptive expectations that leave little room for exploration or risk-taking.

QM Standard 1.3 (“Communication expectations for online discussions, email, and other forms of interaction are clearly stated”) could lead to faculty imposing heavy-handed rules about how many discussion forum posts a student must write and respond to each week, how long they must be, and how many days apart they should appear. Standard 1.7 (“Expectations for prerequisite knowledge in the discipline and/or any required competencies are clearly stated”) could result in a syllabus with a laundry list of expectations of pre-existing knowledge instead of an open conversation with students about past learning and how it dovetails with the goals of the course. Standard 2.3 (“Learning objectives or competencies are stated clearly, are written from the learner’s perspective, and are prominently located in the course”) encourages a paternalistic centering of students and predetermined instructor-prescribed objectives. Standard 2.4 (“The relationship between learning objectives or competencies and learning activities is clearly stated”) encourages a kind of tidiness of design that obstructs learners from choosing their own path or embracing emergent activities.

The QM rubric leaves little space for alternatives to traditional assessment. Ungrading, for example, is a complex, multi-faceted, emergent practice. How does one reconcile this with standards like, “the course learning objectives, or course/program competencies, describe outcomes that are measurable” or “the assessments measure the achievement of the stated learning objectives or competencies.” Alternative assessment is just one example of instructional flexibility that one might want to build into a course, only to find it difficult within the mold of the rubric. How do we design classes that respect the messiness and complexity of learning when so much of the rubric demands that objectives, assessments, materials, activities, and technologies be codified before the first day of class? How do we make space for students as coauthors of their own learning, if the most basic structure of that learning must be predetermined?

What we find most frustrating though about the QM rubric is the lack of human presence and humanity in the language it uses. People rarely appear in the standards; instead the standards are written as sterile instructions, as though the humans making choices and enacting those choices are irrelevant to the process of design or teaching. Many of the standards use passive voice, literally erasing the human actors of teaching and learning. Standard 1.2 (“Learners are introduced to the purpose and structure of the course”) removes the teacher, as though the course itself and not a human being is doing this introducing. In Standard 1.9, “Learners are asked to introduce themselves to the class.” Introducing oneself to a group of classmates (particularly in a fully online environment) asks for vulnerability and trust, so why is the human subject removed from this sentence?

The use of passive voice is pervasive: connections between learning materials and activities “is clearly explained,” a variety of materials “is used,” requirements for interaction “are stated,” accessibility statements “are provided.” It’s worth noting that not every standard employs passive voice. Several use active verbs. In these cases though, the subject is never a person. In the QM rubric “tools promote,” “instructions articulate,” “assessments measure,” and “materials contribute.” Tools, instructions, assessments, and materials are at the center of these standards; humans are not. Here are a few words that do not appear anywhere in the rubric: “community,” “agency,” “inclusivity,” “flexibility,” “joy,” “compassion,” “question,” and “human.”

In the absence of humans and community, QM promises efficiency and objectivity. Even their approach to accessibility (arguably the most human-centered standard in the rubric) is loaded with administrative box-checking instead of actual care. The word “privacy” appears once, but the onus is put on students to protect their data and privacy; the course needs only to provide the necessary “information.” The words “access,” “accessible,” and “accessibility” appear five times in the rubric, but each use refers to either merely providing “policies” and “statements,” or having accessible “text and image files” and “multimedia content” that meet “the needs of diverse learners” (the only context in which “diversity” is mentioned). This barely scratches the surface of what it means to address accessibility (or diversity); following the rubric provides cover for the biases that continue to exist and are left unchecked. Sean Michael Morris writes, “When we talk about decolonising the university or education, we are inviting ourselves to participate in a radical realignment of power […] However, we cannot decolonise education by inviting indigenous students to campus; we must be willing and ready to unseat the teacher and to evolve a university where learning isn’t necessarily always linear.”

Again, there are two things that can be critiqued with regard to Quality Matters: how it is deployed by institutions, and also the words and rhetorical positioning of the rubric itself. When teachers encounter the rubric, certainly they are affected by structures, requirements, and restrictions at their institutions, but they also encounter it as a text, and a tool, one they have to grapple with, both pedagogically and professionally. The precarious positionality of adjunct and contingent teachers influences that encounter, and the rubric (whether intentionally or not) exploits this precarity.

In “Why I threw Away My Rubrics,” Jennifer Hurley writes, “It was only when I was on the receiving end of a rubric, while taking a graduate-level education class, that I had my first critical thought about rubrics. After looking at the rubric the professor had completed for me, I wondered, where is the human response in all of this?” Hurley argues rubrics are especially damaging to those who would be most successful and to those who are struggling. When faculty are handed a rubric like QM to facilitate the design of their courses, their experience mimics what Hurley recognizes among students graded against a rubric. The best teachers are encouraged to step backward as they jump through hoops not really designed for them. The advice they get from Quality Matters runs counter to their better instincts, which see learning (and online learning) as a complex set of human experiences, behaviours, and interactions — all of which can’t be neatly measured by a rubric. There is no incentive (and, in fact, there are risks) to pushing past the boundaries set up by the QM rubric.

Meanwhile, the teachers who are struggling (or teaching online for the first time) find themselves bewildered by the sea of categories the QM rubric contains and the way the rubric patronizes its users. For the new or struggling teacher, the rubric ends up feeling like a crude (and mechanistic) tool for administrators and institutions to police teaching – a primer for “cop shit.” This is not a good entry point into the work of being a teacher, especially the very human work of being a teacher online.

Rubrics, models, and “best practices” proliferate so rampantly because institutions want assurance of quality, but also because they want to control for “extraneous” variables. They are mechanisms for keeping complexity at bay, but they are also mechanisms for controlling people. Rubrics are authoritarian, even colonizing, at their core. Teachers are made to feel precarious within an educational system that insistently devalues their labor. Meanwhile, students are under constant suspicion, their motivations scrutinized and their agency undermined.

Rubrics, inflexible models, and stock approaches to teaching are symptoms of a much larger set of structural problems. The people who make them are not the problem. The teachers who use them are not the problem. The problem is their efficiency, and that they feel necessary at all. From a pedagogical perspective, Quality Matters is the least necessary of rubrics, empty of ethos, and so nebulous as to be also empty of practical advice. It gives administrators a safety blanket, confuses teachers (or gives them a false sense of “security”), and does harm to students.

None of this is to say the Quality Matters rubric has never ever been used to support good pedagogy. The point is that it is not the first place we should be turning as we begin to imagine what online and hybrid learning could be, especially when students (and teachers) are struggling. Last fall, when a conversation about the relative merits and problems with QM arose on Twitter, several of the staff members at the organization pointed out that the rubric could be bent to accommodate the kind of flexibility and humanity that it lacks on its surface. Between the lines, they argued, was a great deal of opportunity for creative interpretation. While we take their point, we also wonder: why not just create something that has these values baked in from the start? As an exercise, we attempted to use the structural frame of the rubric to create our own set of “standards.” We wondered what would happen if we rewrote the rubric for a human audience, centering the people in a course instead of the tools, policies, and materials (our attempt at that is included below).

Ultimately, we aren’t really making an argument about Quality Matters. The Quality Matters rubric is just one trenchant example of what we describe as “the end point of traditional models of instructional design. […] what happens when we don’t support nuanced, complex conversations about pedagogy and design, when we don’t valorize the collaborative (and sometimes messy) work of teachers and designers, when we don’t design for and with the students who actually show up to our physical and virtual classrooms.” We don’t subscribe to any model that relies upon a simplified set of bullet points to do the vitally important work of instructional design. We think this practice should be emergent, complex, and human. There can be no neat and tidy distinction between the kind of design we do for learning on-ground vs. the kind of design we do for learning online. Any model we or anyone else develops should serve only as a set of possible starting points, opportunities for critique and conversation. Instead of prescriptive checklists, we need to build our capacity with new modalities, be always open to learning from and with our colleagues and students, and listen to and trust our own pedagogical instincts, no matter where (or with what tech) we teach.

Course Design Considerations

1 Introductions and Invitations

1.1 Start with “Hello, how are you?”
1.2 Directly engage students in helping build and structure the course.
1.3 Invite students to talk to you.
1.4 Share institutional resources that may help students learn. Don’t ask students to “comply” because that reinforces notions of students as people who need policing.
1.5 Provide students with computers if they don’t have them.
1.6 Invite students to share what they know and how it excites them to be in the course.
1.7 Explain what kind of technology knowledge students may need and help them find ways to learn what they do not already know.
1.8 Listen to student introductions before sharing your own; put their voices first.
1.9 Invite students to make themselves known to their classmates in whatever way is comfortable for them.

2 Learning Goals and Trajectories (Possibilities)

2.1 Leave room for personal goal-setting and growth
2.2 Invite students to share their personal progress in whatever way makes sense for them.
2.3 Encourage students to inspect and re-author all learning objectives and competencies.
2.4 Collaborate with students to develop activities and work that helps them achieve their personal goals.
2.5 Create flexible points of entry for every student, opportunities for students to challenge themselves, and space for them to share their accomplishments.

3 Feedback and Reflection

3.1 Design assessment that rises out of the pedagogical goals of the course rather than dictating specific outcomes.
3.2 Do not leave students feeling like a rug might get pulled out from under them.
3.3 Find clarity in conversations with students, not in pre-authored policies and rules.
3.4 Recognize students as complex human beings, needing varied and flexible approaches to assessment.
3.5 Be available to talk to students about their learning and goals.

4 Content and Representation

4.1 Choose content that invites diverse voices into the classroom and helps students feel recognized and welcome.
4.2 Engage students in conversation about the relationship between the content and the activities of your course.
4.3 Model sharing practices for students and invite them into a discussion of the various ways they can honor their sources.
4.4 Engage students in thinking about the world they live in, the present moment, the material circumstances influencing them, their specific context, and their education.
4.5 Make the world your classroom; invite students to explore the world in whatever way helps them meet their learning goals.
4.6 Stop requiring stuff. Offer invitations.

5 Community

5.1 Consider your learning outcomes and objectives and ask yourself whether they help in any way to create a learning community.
5.2 Encourage students to talk to each other about their learning.
5.3 Be available to students to talk to them about their learning. Be a human being with real reactions to student work, not a template of prescribed responses.
5.4 Recognize that different students engage at different times, in different ways, and that engagement looks different depending on context, material circumstances, the bodies we bring to a learning space, etc. Honor and celebrate those differences.

6 Ethical and Flexible Approaches to Technology

6.1 Do not use tools that farm students’ data or intellectual property. Use tools that are designed in a way that respects student privacy and agency.
6.2 Invite students to use the tools or technologies that allow them to explore their learning goals.
6.3 Encourage alternative approaches, invite experimentation with alternative technologies, and keep requirements flexible.
6.4 Do not use course technologies that harm students.
6.5 Have conversations with students about how they use technology, but also about how we decide what tools to use and about the ethical and legal obligations of technology companies.

7 Student Support and Basic Needs

7.1 Make sure students have the help they need. Or that they can find it. Or that they are supported in helping each other. Check in with them beyond the first week of classes about this.
7.2 Build a course that can be a collection of human beings not instructions and policies.
7.3 Keep making sure students have the help they need — especially that their basic needs are being met. Collaborate directly with your staff colleagues in financial aid, disability resources, tutoring, student success, the library, IT, etc.
7.4  Do not turn your course into an island. Help students connect to the community of the school beyond your course, particularly students from marginalized populations who are more likely to feel isolated from this community.

8 Accessibility and Human-centered Approaches

8.1 Make sure your students can find what they need to succeed in your course.
8.2 Choose technologies that do not put up barriers for students; be prepared to provide alternatives and flexibility if a student is having trouble accessing your course technology.
8.3 Meet legal requirements regarding accessibility; they are not a choice. Learn more about them through the experts on your campus that work in offices like disability services.
8.4  If your students are having trouble understanding you or your course content, meet with them and help them. Revise the course for the future so fewer students struggle. Realize that more instructions (especially patronizing instructions) don’t necessarily help people understand better.
8.5 Caption your videos. Use alt tags. But engage more thoughtfully than just meeting bare minimum legal requirements.

*Note: Please don’t print this rubric out and distribute it to faculty. That’s exactly not the point. Instead, if you or your faculty are required to engage with the Quality Matters rubric (or some other standardized model), undertake a similar activity, reimagining and rewriting the rubric or model, from your own pedagogical perspective, for your own idiosyncratic work.