In my work at The Trace, I wanted to better understand why so few shootings in America get solved. I worked with investigative reporting fellow Sean Campbell and BuzzFeed News’s data editor Jeremy Singer-Vine to request violent crime data from more than 50 police and sheriff’s departments. We used some of this data for our story, “Shoot Someone In a Major U.S. City, and Odds Are You’ll Get Away With It,” and posted raw data from 56 agencies online.
We also filed a request to each agency for documents that list all the fields in the databases that they use to track information on major crimes, and that explain how those databases function, such as the data dictionaries, record layouts, and user guides. We got back records from two-dozen agencies, which together cover some of the most widely used law enforcement databases in the country, extending their utility far beyond the cities we reported on (click here to jump to the full list).
For example, the Baltimore County Police Department sent us the 2,000-page user manual for Intergraph’s inPURSUIT Records Management System, which is used to track information on practically every aspect of policing, from traffic stops to property seizures. The manual gives the name and definition of every field in the database, along with screenshots of what the officer would see during every step of the data entry process.
We hope that by posting database documentation like the above, and sharing the strategies we’ve learned for making data requests, we can help you with your own investigations — whether through filing more specific requests, deciphering the results, or pushing back when agencies say release is impossible.
If you’ve been able to obtain documentation for other police databases not already listed, we encourage you to share the records with the MuckRock community for other reporters to use by emailing [email protected] with the subject line “Police Records Database.”
The basics of making a solid data request
This request to the Pittsburgh Police Department for officer staffing data annotates some of the phrases I use in my data requests. I make it clear that I want data, not PDFs or printouts of tables, and I try to pre-empt the most common objections and points of confusion I’ve run into.
In my list of data fields, I always include some version of the following:
- Before listing specific fields, I state that I am seeking “all fields that contain non-exempt information, including but not limited to the following types of information.” If you are requesting a single incident report, you do not need to list every piece of information you want. The same logic should apply to the database that contains the information. Additionally, this provision gives you legal recourse to appeal if you realize you’re missing something important after you’ve gotten your data.
- “All identifiers, primary keys, and/or metadata that distinguish between unique records and individuals, and that link the database tables together, including but not limited to Incident IDs, Person IDs, and [any other relevant fields you’re aware of].” Your data may be unusable without these fields. For example, if Pittsburgh has two officers named John Smith, you won’t be able to differentiate between them in the data unless you have their officer ID.
- “All documentation that describes the layout of the database’s tables and fields, and that provides the definitions of any field names, codes, and abbreviations used in the database.” Otherwise, you may get data that looks like alphabet soup. You can also use the list of fields to appeal if the list shows fields that contain non-exempt information that you weren’t given.
Pre-empt these common objections
It’s helpful to cite the provisions in the law, or court rulings, that address the most common objections to data requests.
- “We don’t have to create a new record.” Public records laws often expressly state that requests for raw data do not constitute the creation of a new record. (Technically, requests that require some type of calculation, like “the top 20 most-sued officers,” count as the creation of a new record, but agencies often offer them instead because it gives them more control over what you can report.)
- “We would need to write a program to extract this data, which will cost you $40,000.” Some laws prohibit agencies from creating a new database that can’t export data in response to public records requests. If the database was created after this amendment was implemented, you should not be held accountable for any hardship associated with exporting data.
- “Your request is overly burdensome. We would need to manually review thousands of records for redactions and that would take us 30 years.” Only request fields that don’t require manual redaction and specify that you are not requesting “free text fields” like narratives that “contain a mix of exempt and non-exempt information,” because that would, in fact, require the agency to read through thousands of records.
It can be frustrating when agencies insist that they only need to release a handful of fields, regardless of what the law actually makes public, as if they are allowed to make their own rules for information that is stored as data.
In fact, public records laws often expressly state that the disclosability of information is not limited based on format. To argue this point, I’ve often attached a document that the agency has redacted that contains the same type of information I’m seeking as data. The same rules should apply. For example, the below image shows an incident report generated by the New York Police Department’s Omniform database that the agency redacted and released as a document, compared to the same report, but revealing only what’s in the crime data that the NYPD posts online. The document version gives a much clearer picture of what happened. The online data doesn’t even include a column indicating whether the crime involved a weapon.
Your writing could make or break your request
Think of your initial request and all communication that follows as legally-binding documents. The records officer is entitled to take everything you write very literally, and if you must appeal, it could all wind up in a court file.
- Request data in a way that can be queried. My biggest FOIA fail of all time was filing dozens of requests for data on “homicides and nonfatal shootings.” Many agencies do not track incidents as nonfatal shootings because it’s not an official crime classification. We quickly learned that we needed to amend our requests to ask for “murders, robberies, and assaults,” the weapon used, and any fields that indicate if shots were fired and if anyone was injured. (We dropped rapes if the state’s law had a lot of special exemptions for sex crimes.)
- Write for all levels of data literacy. Your request and follow-up emails need to be understandable to a records officer who doesn’t know how to use Excel, yet specific enough for the person who will be generating your data.
- Follow all phone calls with an email summarizing the conversation. Some records officers negotiate over the phone, then fail to make good on their promises. This way, you have a record of the conversation.
- Treat everything as if it could become part of a lawsuit. If your request is ignored, follow up several times, offering to clarify or amend your request if necessary (agencies must give you the opportunity to do this before denying your request). Respond to baseless objections by citing the law and other supporting evidence. This greatly increases your chance of succeeding with the agency. But if you must appeal, you’ll need to attach all written communications. Showing a good faith effort to work with the agency will go a long way.
- Exhaust all agency-level negotiation and appeal options. You often can’t sue or appeal to an outside agency unless you’ve done this.
Once you’ve received data, make sure you got everything you requested. Frame any follow-up as missing records, not as questions. The law does not require agencies to answer questions.
“I received records on 500 homicides, yet your agency reported 600 homicides to the FBI. Please regenerate the data to include the missing 100 homicides…” is allowed.
“Why am I missing 100 homicides?” is not.
I scan my follow-up emails for question marks and rewrite those sentences as statements.
Don’t assume something has been withheld for nefarious reasons. It’s often a mistake in the query or a misunderstanding. Be friendly but firm in stating your rights.
Learn as much as you can about the database
It’s always a good idea to learn as much as you can about the database before making your request.
- Before filing a request, check your city’s open data portal, or the Police Data Initiative if you’re compiling data from multiple agencies. You may find the information you need is already online.
- Incident reports and other types of forms, and public access sites for things like court records or inmate locations, are usually generated by a database. Therefore, each piece of information separated by a header should correspond to a separate field in the underlying database.
- Patrol guides and other employee training manuals, like this “Report Writing Manual” from the Bakersfield Police Department, often include very specific instructions for data entry, including field names, code definitions, and screenshots of the various steps in the data entry process.
- The database’s “documentation” gives the most detailed breakdown of the database, such as the user guide, data dictionary, record layout, entity relationship diagram, and a glossary of terms. Sometimes it helps to attach examples of each, because most people aren’t familiar with these types of records.
How we got the documentation for police databases
We obtained these documents by filing a very simple request to each agency for:
- “The Requests for Proposals (RFPs), Requests for Qualifications (RFQs), and contracts pertaining to the creation or maintenance of the database(s).”
- “The data dictionary, glossary of terms, record layout, entity relationship diagram, user guide, and any other records that describe the database(s).”
- “The instruction manual, or any other type of guide, distributed to law enforcement personnel dictating how they should use the database(s).”
Agencies often cite security concerns or trade secrets to deny requests for the database documentation, or they simply claim the records don’t exist. To counter the first two objections, look for provisions in your open records law, or court rulings, that explicitly address database documentation, metadata, and employee manuals. To counter the argument that the records don’t exist, you could look through state regulations, contracts, and RFPs for stipulations that data dictionaries and other documentation be included with databases.
We could not have obtained data from the Newark Police Division, New York Police Department, and numerous California police and sheriff’s departments, without pro bono representation from Jean-Paul Jassy and Kevin Vick of Jassy Vick Carolan LLP; Gideon Oliver of Gideon Law; Adam Marshall and Katie Townsend of the Reporters Committee for Freedom of the Press, and C.J. Griffin of Pashman Stein Walder Hayden. Jason Schultz, director of New York University’s Technology Law & Policy Clinic, and JD candidates Suchita Mandavilli and Jake Karr, assisted with legal research.
Below are the links to the documentation that we received from various police and sheriff’s departments, and the brand of database covered by the documents. If your police or sheriff’s department uses one of these databases, you should be able to use the records in your request.
Atlanta Police Department, Georgia: In-house system
Bakersfield Police Department, California: Versaterm Versadex
Baltimore County Police Department, Maryland: Intergraph (Hexagon) inPURSUIT Records Management System, version 12.7.0
Boston Police Department, Massachusetts: Intergraph (Hexagon) inPURSUIT Records Management System, version 12.6.1
Buffalo Police Department, New York: CHARMS Globalquest
Charlotte-Mecklenburg Police Department, North Carolina: In-House System
Flint Police Department, Michigan: New World Systems Aegis MSP Version 10.2
Indianapolis Police Department, Indianapolis: Interact
Jefferson Parish Sheriff’s Office, Louisiana: Automated Records and Mapping System (ARMMS)
Kansas City Police Department, Missouri: Tiburon Field Automation System version 7.4, Tiburon Records Management System version 7.4.1, and Tiburon LawRecords version 7.6
Kern County Sheriff’s Office, California: Intergraph (Hexagon) iLeads
Maricopa County, Arizona: Maricopa County Sheriff’s Department
Miami Police Department, Florida: Motorola Infotrak LRMS Version 5.3
Milwaukee Police Department, Wisconsin: Tiburon
New Orleans Police Department, Louisiana: In-House System
Phoenix Police Department, Arizona: Intergraph (Hexagon) inPURSUIT Records Management System, version 12.7.0 and inPursuit RMD Explorer version 3.0.0
Sacramento County Sheriff’s Department, California: Northrop Grumman CommandPoint
San Antonio Police Department, Texas: Intergraph (Hexagon/Denali) inPURSUIT Records Management System
San Jose Police Department, California: Versaterm Versadex Records Management System, version 7.3
Stockton Police Department, California: Tiburon LawRecords version 7.10
Tucson Police Department, Arizona: Intergraph (Hexagon) iLeads
If you have records on a database not listed here that you’d like to share with the MuckRock community, please email [email protected].
