BetaGov Blog
We Don't Care About Your Trial's Results, And That's A Good Thing
Posted on April 12, 2017 By Lauren Rilling
How many batches does it take to make a perfect chocolate chip cookie? If you're a baker at America's Test Kitchen, the answer is 30,000. Never content to stop at "good enough," these chefs painstakingly and repeatedly test every facet of a single recipe for an average of seven months in the lab kitchen, and then continue testing and altering the recipe based on the feedback they receive from home cooks around the country. While success is the ultimate goal, a recipe flop produces valuable knowledge that fuels motivation to keep testing.
We can all learn from America's Test Kitchen's attitude toward research. Let's face it: the world of public policy isn't exactly known for holding failed "recipes" in high regard (have you ever heard a politician campaign on ideas that didn't work?). In a system that tends to settle for "good enough," BetaGov is inspiring innovation and improvement in the public sector through the rigorous testing of policy ideas.
When it seems like policy stakeholders are constantly quoting statistics from the latest study, why should practitioners be interested in conducting more research? After all, it's relatively easy to find data that can be used to support virtually any opinion. The problem is that there are so many kinds of research (some of which are better than others), a multitude of factors that affect a study's validity, and nuances in data that can be easily misinterpreted. The human brain is full of assumptions and subconscious biases, and the more invested one becomes in proving an idea right or wrong, the more likely one is to taint (even accidentally) the research process with methodologically unsound procedures or data analysis. It's only natural!
That's where a well-designed randomized controlled trial (RCT) comes in handy. RCTs remove human bias as much as possible by comparing outcomes for two similar populations, only one of which receives the intervention.* Conducting an RCT can be as simple as finding out what happens when baking the same cookie recipe at 350 degrees compared to 400 degrees. Fortunately, this research design can also handle more complex innovations, such as testing the effects of a writing curriculum in a school district.
BetaGov does not hinge a trial's success on obtaining positive results, but on accurate results. Our only passion is the research process itself, ensuring that the practitioners we partner with have the resources and professional assistance they need to design and execute their own RCTs.
We embrace testing ideas, and then testing them some more. We don't take political positions or promote policy agendas, and this is essential to our role as researchers who assist practitioners in conducting their own research. It's because we don't have a stake in what the results are that we can confidently guide practitioners through RCTs that produce reliable results they can use to make their agency better.
*Intervention: research jargon for the idea being tested