As part of its federal court settlement agreement to eliminate racial profiling and other systemic problems, the Oakland Police Department is required to build a computer system that can identify individual officers who engage in risky behavior or misconduct. The database tracks evidence of racial profiling, the frequency with which officers use weapons like tear gas and firearms, whether they get involved dangerous situations like high speed chases and auto accidents, and other factors. The purpose of the sophisticated system — similar versions have successfully been implemented by other departments across the country — is to help OPD supervisors intervene early with officers and correct problematic behavior while also weeding out bad cops.
But according to the court-appointed police monitor who oversees OPD’s compliance with the settlement agreement, the Microsoft-built computer system, known as the Performance Reporting Information Metrics Environment (PRIME), has suffered from cost over-runs, delays, and numerous bugs.
The city has already spent at least $6 million building PRIME, but now the department might have to scrap it entirely and spend millions more to develop a new system from the ground up.
The failure to launch the risk management database has the city and police department scrambling to find a solution ahead of an upcoming status conference with U.S. District Judge William Orrick on Feb. 2. Orrick oversees OPD’s progress under the federal settlement agreement.
In his January 2018 report to Judge Orrick, Independent Monitor Robert Warshaw wrote that there “have been many concerns over the status of PRIME, beginning with delays in implementation, to continuing issues with ‘bugs’ in data collection, and errors in data available. …”
As a result, the department is behind schedule in launching a fully operational version of the database. The existing system, called PRIME 1.0, has few of the capabilities that OPD promised.
For example, it doesn’t allow OPD supervisors in the field to access body camera video footage and is unable to integrate information such as field training records and police academy records into the database so that they can be reviewed alongside internal affairs information. Stop data analyzed by Stanford University professor Jennifer Eberhardt also hasn’t been folded into the database, meaning that supervisors can’t track evidence of racial bias and profiling.
The monitor wrote that OPD’s compliance with two of the settlement agreement’s tasks is now “in jeopardy given the problems with PRIME.”
Altogether, it’s yet another setback in the department’s 15-year federal court oversight saga.
“We appreciate that a reasonable period of time may be needed when new technology is implemented,” wrote Warshaw, “but also note that the definition of ‘reasonable’ is being strained by the continuing problems with PRIME.”
The project began back in 2006 when the city’s information technology department built a precursor system called the Internal Personnel Assessment System. IPAS, which drew on very limited data, was considered “antiquated” by 2014, so that year, the city council selected Microsoft to build PRIME at a cost of $2.2 million. The system was supposed to become operational in late 2016. However, Microsoft and city staffers ran into several technical problems that delayed the launch until May 2017, according to city records.
Last July, the council voted to spend another $1.8 million on PRIME to incorporate many of the missing sources of data. And according to city spokesperson Karen Boyd, the city has spent another $2 million building the system.
But Warshaw wrote in his report this month that OPD’s process of identifying and intervening with officers who engage in risky behavior and misconduct is “severely affected” by PRIME’s ongoing technical problems. In fact, inconsistencies in the database mean that all of the information supervisors pull from it has to be tediously checked by hand with separate personnel files. “The condition of the database is described as worse than under the earlier data system which was seen as needing replaced,” wrote Warshaw.
Jim Chanin, one of the attorneys who sued the city back in 2000 in the infamous Riders case that put OPD under federal court oversight, said he’s worried about the database’s problems, but he also remains committed to helping OPD build a system that works. “I have hope we’re finally turning a corner,” said Chanin, “but that corner means recognizing we have a problem, not that we’re close to solving the problem.”
Chanin said he’s especially concerned about three features that were supposed to be built into PRIME, but aren’t close to being implemented. First is the inability of field training officers to access the academy records of police trainees. After trainees graduate from the academy, they’re assigned to field training officers — veteran cops who show them the streets during a probationary period. It’s crucial that FTOs have access to records from the academy, among other sources, in order to know about an officer’s strengths and weaknesses and help him or her unlearn behaviors that can cause trouble or create dangerous situations for cops and civilians.
Second is the inability for supervisors to view body camera video in the field. Chanin said if sergeants, lieutenants, and captains can’t view videos in their squad cars, it’s unlikely they’ll do it later at a station unless a complaint is subsequently filed. It makes early intervention in possible breaches of policy or misconduct much less likely. Problems will fester and turn into more serious issues later.
Finally, Chanin said he’s worried about the inability of Microsoft and OPD to integrate the racial profiling stop data prepared by Stanford into the PRIME system. “This will allow better control over training and possible coaching of biased police officers,” he explained, adding that much of what the Stanford team has uncovered is implicit bias that could be consciously avoided if officers understand how and when they treat different racial groups unfairly.
“We have been told all this is doable,” said Chanin, “but to this day, we don’t have any of it.”
Independent experts say it’s not unusual for large police departments, even those in tech-savvy regions, to have trouble building an advanced early warning risk management database like Oakland’s PRIME.
Joe Walsh, a data scientist with the Center for Data Science and Public Policy at the University of Chicago who helped build an early warning database for the Charlotte-Mecklenburg Police Department in Charlotte, N.C. , said the biggest challenge is getting a police department’s numerous existing databases to feed into one single system like PRIME. For many departments, academy files, personnel records, field training documents, use of force reports, and other crucial records are collected and stored in separate places. Some of the information is already in a database that can be merged into a risk management system, but many departments still keep paper records, making access very difficult.
“Building the models is not a huge lift,” Walsh said about the formulas that can predict which officers need intervention and training based on their current behavior. “Most of the time spent is getting all those systems to bring all the data together.”
So far, OPD and the city have kept the details of the PRIME system and its numerous shortcomings to themselves. The city did not make police and IT staff who are working on PRIME available for an interview. Boyd wrote in an email responding to questions from the Express that “the project suffered early on from staff transitions, and areas where the requirements were not completely understood and/or properly implemented,” but that it’s now back on track to be completed in 2019.
Councilmember Rebecca Kaplan reviewed the monitor’s latest report about PRIME and said she’s appalled by the lack of progress and by the fact that the city administration and OPD haven’t given the council a recent update on the project.
A briefing on PRIME is currently listed on the council’s legislative calendar without a hearing date.
At last week’s Oakland Police Commission meeting, OPD Capt. Paul Figueroa provided a brief update on PRIME. “We spend a lot of time and effort fixing these bugs,” he said about the system. “We continue to struggle every week to do that.”
Microsoft’s performance on the project has been so poor that the city invited new bids from at least three other companies to take it over. Assistant City Administrator Stephanie Hom told the commissioners that an evaluation process is underway for several bids submitted by new vendors and that the city will consult with Warshaw before picking a company to try to salvage PRIME.
According to the city administrator, four vendors submitted bids, including Accenture, Outsystems, Sierra Systems, and Microsoft. Microsoft officials did not respond to a request for an interview. But if the city scraps Microsoft’s version of PRIME and starts over, it could cost as much as $2.4 million more in public funds.