Written by:
Alice Huguet
Our research investigates how evidence was used in three policy decisions in a mid-sized urban school district. Policy discussions related to mathematics classes and curriculums and involved district leaders from various departments and two outside consultants. We identify two main themes that could be considered in future research.
As an educational researcher, I’m interested in how and when evidence enters practitioners’ and policymakers’ conversations. Yet, limitations often prevent us from being deeply embedded in schools and districts in a way that would allow us to understand the nuances of evidence use. My research team and I wanted to challenge that by conducting a rich, longitudinal, qualitative study of one district’s decision-making processes.
We collected and coded field notes from over 350 hours of observations of internal district-level meetings, analyzing the kinds of reasons that individuals used to support their positions about policy decisions. Think of these reasons as justifications that actors used to convince colleagues about the “right” path to take on a particular topic. Evidence like research and data did emerge in these conversations, but made up a minority of the reasons that the teams employed. Other types of reasons included invoking constituents’ anticipated reactions and needs, statements of values, references to existing policy constraints, and others.
Some novel findings emerged from our analysis. First, if a policy decision was perceived to be potentially controversial, the kinds of reasons used in negotiations were different from those used in less contentious policy discussions. Second, the makeup of individuals at the table influenced the degree to which people invoke reasons at all, including research and data. These themes raise questions for further research in the evidence-use field.
Potential controversy can factor in how a team makes decisions
We studied conversations about three separate mathematics policy decisions. This gave us the opportunity to compare reasons that people used when discussing different topics, and we saw that anticipated controversy surrounding a policy decision was associated with greater invocations of evidence, and more frequent references to stakeholders like parents and teachers.
One of the three policy decisions that we studied was viewed as significantly more likely to be controversial with parents and school leaders. The reasons used to debate this decision were notably different from the other two deliberations. Data was referenced about twice as often in the controversial policy deliberation than in the others. We hypothesize that this is because participants wanted evidence to add authority to their decision in case they needed to diffuse any frustration from interested parties (e.g., parents, teachers, principals).
In addition, reasons in this deliberation were significantly more likely to invoke constituents—for example, “Parents are going to want their kids to take calculus in high school”—than in the other policy conversations that we studied. This finding raises additional questions: What other characteristics of the policy at hand might change the kinds of evidence used in deliberations? Would this be true across sectors?
Who is at the table can change how evidence is used
Evidence use is relational, and this is often examined in the context of relationships between researchers and policymakers. In our research though, the relationships that mattered existed among the decision-makers.
We found that participants in a deliberation were more likely to frame a problem or offer a solution without any reasons at all—in other words, without justifying their claims—when they were in the company of people from their own department, and with whom they had shared beliefs about mathematics. They may not feel the need to invoke research and data in their conversations, because they know others already share similar knowledge. This points to a perennial challenge of studying evidence use; how can researchers better understand when shared knowledge is rooted in evidence, if so much of it is “baked in” to conversations rather than explicitly referenced?
Analyzing the gamut of reasons in these district deliberations introduced additional questions; foremost on my mind is whether patterns of reason-giving are similar in other sectors. Or—as the variation between our three different policy questions might suggest—do closed-door negotiations in other sectors look significantly different? If we want to understand how evidence can enter important discussions, mapping the character of the conversations in-depth may help provide the blueprints.