BetaGov spotlight: Jason Potts
Date: September 25, 2017
Jason Potts is a lieutenant with the City of Vallejo Police Department in Vallejo, CA. He joins BetaGov to discuss his role leading an RCT to test the effectiveness of equipping patrol vehicles with automatic license plate readers.
Tell us about the trial and your role in it.
I partnered with Maureen at BetaGov to develop the trial design and methodology. We wanted to know if an automatic license-plate reading (ALPR) system would work.
We equipped three patrol cars with ALPR that were randomized on a schedule. Two were filtering and alerting the officers. The third car, the control, was set to alert me, but not the officers. Again, the officers in the control group were not notified (blinded) that they were using an ALPR car in the control group for that day. Our purpose was to isolate the intervention from the control group to test the effectiveness of the technology.
An ALPR scans thousands of plates. The reader alerts us to stolen vehicles, among other notifications such as wanted subjects and lost or stolen plates.
Intuitively, we figured the ALPR would work better instead of having an officer manually look for stolen vehicles. Now we have the results that prove it: the data show that we recovered 37 stolen vehicles and made 17 arrests in 90 days as opposed to a much smaller number for the control group.
We also wanted to know if officers arrest more subjects on a particular shift. Are less tenured officers more inclined to use the technology? Are younger or more senior officers more adept at locating stolen vehicles without the technology? Do the days of the week and two-person cars versus one-person cars affect the data?
We also started looking at fixed license-plate reader locations and the number of arrests and stolen-vehicle recoveries compared to the mobile readers. Fixed locations are posted on traffic lights or light posts at four different thoroughfares throughout the city. Some questions that we aimed to answer: Were fixed locations not as susceptible to human error, such as crashing a police car with a mobile ALPR unit or the having the mobile ALPR computer in the vehicle go down? We also hope to answer via a survey at the end of the trial: How can we improve the technology? Do we need to reposition cameras—one facing rear for cars without front plates? How do repetitive lost- or stolen-plate hits affect officer behavior? And do we need more fixed readers as opposed to mobile readers?
Where did this idea come from?
I would give credit to my colleague at NIJ LEADS, Jeremiah Johnson. While at IACP in San Diego last year, we discussed ideas for various randomized controlled trials and he mentioned an RCT related to license plate readers. I reached out to Maureen at BetaGov, and we tweaked the idea, and tailored it specifically to Vallejo PD.
Had you ever done a research study like this before?
Never. I'm excited and enthusiastic about where evidence-based policing can take the profession. I think the sky's the limit for the things we can test to benefit law enforcement with RCTs, testing police practices, and learning whether they work or not. And it doesn't have to be that complicated.
This trial can be easily replicated, and that's the point I want to drive home. If more and more agencies can do similar trials, we can build the data by figuring out what works and what doesn't with data and science.
The thing that's positive about this study is that human error is minimized because we just randomized the vehicles—not the drivers. The other thing is the officers didn't know they were in the control group. They could operate as normal and by doing so, we could test and isolate the intervention. We were also able to identify unintended consequences or surprises.
Were there unintended consequences with this trial?
Yes. More like surprises. 30% of all vehicles scanned by mobile units were misreads. I didn't think the number would be that high. Also, some of the officers were trying to use the fixed license-plate reader locations in conjunction with the mobile unit. These were challenges especially when identifying if the stolen-vehicle recovery was the result of a fixed LPR or a mobile ALPR. We're trying to isolate data on the mobile readers. We discovered that most of the hits were from lost or stolen plates, and that many were duplicates.
Dealing with the record-management system presented an opportunity to talk about how difficult it is to grab data in law enforcement. We should be able to pull up information almost like a Google search. Just the data collection alone has been a challenge for me to navigate through our system. You find a way to do it, but I shouldn't have to go through complex windows and fields to accomplish it.
What was unique about this experience?
I think it was the exposure and opportunity to run a randomized controlled trial. I have a little bit of academic background, and I'm a practitioner, but it was still a challenge. However, it was an opportunity to learn further about evidence-based policing practices, and to think about other creative ways to use RCTs. Being exposed to RCTs, working with Maureen at BetaGov, getting to share our experience at the American Society of Evidence-Based Policing (ASEBP) conference, and being able to share evidence-based policing with other agencies was definitely valuable and unique.
Has your agency made any changes to its policy or procedures as a result of the trial?
We're reviewing that right now—such as improved efficiency, data retention, and investigative value. We were attacked with a computer virus, so the two vehicles that are in service right now are unable to scan plates, but we're going to get them up and running. We're going to evaluate the cost-benefit analysis of ALPRs and the need for more fixed cameras as opposed to the mobile units.
Do you think you'll lead another trial in the future?
Yes, I hope so. My chief and command staff have embraced this and has been very supportive. I'd like to do continue developing homegrown RCTs that can be easily replicated. I don't want to overcomplicate things.
If we can do this, anyone can. We're short-staffed and have the second most violent city in California for our size. We have staffing and budgetary issues but we successfully ran this trial. Far too often in policing we put up roadblocks and identify why we can't, instead of how we can. A big take away from this trial was proving that we can and doing so with accurate data collection. However, I would say, data without context are just data, but with the evidence, we can test and contextualize them. We shouldn't just confirm what we already think we know, instead evidence-based policing, such as running and completing an RCT, advocates for challenging our conclusions. The public is seemingly no longer satisfied with accepting the conclusions of the police without evidence and data.