For decades, the Judge Rotenberg Center, a school for children with developmental and behavior disorders in Canton, Mass., employed brutal methods to discipline students, including electric shock therapy. My colleague, Heather Vogell, and I anticipated that government data on student complaints would shed light on the school’s practices, but realized that student privacy laws protected those records from disclosure. By requesting the records with certain personally identifiable information removed, we were able to abide by the law and still document incidents of harsh punishment at the school.

Most journalists who cover health or education struggle to obtain government records and data that are vital to our stories and have compelling public interest. While some agencies are reasonably accommodating, others exploit every loophole or gray area in the law to deny public records requests—or delay in the hope that the journalist will move on to another story and stop bothering them. Health and education records are especially elusive because of federal laws that protect the privacy of patients and students.

ProPublica has often negotiated with or contested rulings by government institutions to pry data out of them. Our persistence has led to groundbreaking findings, such as our analysis of birth complications for our “Lost Mothers” series. Over the years, I have amassed a variety of tips and tricks on how to overcome or circumvent these restrictions. I shared the following strategies last week with more than 1,200 reporters at the National Institute for Computer-Assisted Reporting (NICAR) conference.

Know the Laws

The federal Freedom of Information Act (FOIA) allows records officers to deny your request under nine restrictions, or exemptions. They protect records related to national security, internal agency rules, trade secrets, internal agency memos, personal privacy (also known as (b)(6) or exemption 6), law enforcement, banks, oil and gas wells, and any records that are exempt under other laws. States also have open records laws, and their exemptions frequently echo federal restrictions.

Two key federal laws protect the private information of patients and students: the Health Insurance Portability and Accountability Act (HIPAA), and the Family Educational Rights and Privacy Act (FERPA). Along with exemption 6, these laws are commonly cited in denials of health and education data requests.

HIPAA, a 1996 law, aims to make it easier for health care organizations and companies to use electronic records so that medical data can quickly be transferred. It applies to health care offices and institutions (for example: doctors, clinics, nursing homes, pharmacies, universities, insurance companies and more), as well as any organization that electronically transmits health care data, including schools, prisons, and detention facilities.

HIPAA’s privacy protections last 50 years after a patient dies. After death, an executor or surviving family member may decide whether to disclose personal health information.

Enacted in 1974, FERPA protects the privacy of student’s “education records” and limits disclosure. Federal funds may be withheld if schools violate FERPA. Because nearly all public K-12 schools, colleges and universities receive public funds, nearly every educational institution in the country is covered by FERPA. The protected data includes information such as student or parent names, addresses, Social Security numbers, fingerprints, place and date of birth, as well as educational records.

Be Prepared Before Filing a Data Request

• First, find out if the data exists. Look online or phone the government entity and ask for data schemas, dictionaries and repositories (specifically discussing with a records officer which “limited use” data sets may be available). If you plan to file a request with a particular state, familiarize yourself with the state’s data reporting requirements. Ask a public affairs officer for the data before filing a formal request. Explain to the records officers that you don’t want to waste their time or yours.
• Track down the internal data wizard. Try to speak with data custodians, or the people responsible for maintaining the databases, instead of, or in addition to, a public affairs or records officer. They know the data best and can help you tailor your request.
• Always request an itemized cost estimate. Government agencies sometimes calculate exorbitant cost estimates for fulfilling your request. Make sure you seek an itemized estimate to see if they are over-charging you.
• Negotiations are crucial. Explain to the records officer that you are willing to negotiate the scope of your request to avoid privacy restrictions by redacting or removing certain fields.
• If you plan to request data from a state government, familiarize yourself with its open records memoranda and legal decisions. State attorneys general frequently rule on thorny records requests. If your request seems legally dicey, look up recent attorney general decisions related to privacy restrictions.

Even with HIPAA, You Can Still Get ‘De-Identified’ Data

If a data set has been “de-identified,” HIPAA’s privacy rules do not apply. There are two methods for de-identification: “safe harbor,” which suppresses fields that reveal personally identifiable information, and “expert determination,” which relies on experts to verify that there is a limited risk of identifying patients.

• Check if de-identified data is available for download online. Local and state health agencies sometimes put de-identified data sets online. These data sets have minimal, if any, restrictions on their use.
• Ask a records officer to remove personally identifiable fields. If the health data you want includes any personal identifiers, consider requesting the data with these variables removed or redacted. If there are account or Social Security numbers to identify each patient, ask for dummy IDs (but make sure to find out which variables have been replaced by dummy numbers).
• Request aggregate data. Some records officers may deny your request on the grounds that aggregating data is the same as “creating” data, which they may not be legally obligated to do. So ask nicely, and negotiate! If you are able to get aggregated data (or data you can only publish in an aggregate form), you may be prohibited from publishing data on small groups of people in order to protect the privacy of patients.

If the Government Won’t Help, Try These Sources

• Depending on the hospital, you may be able to request information from the “hospital directory,” which has basic facts about current and recent patients, including patient names and conditions, and where they are being treated within the hospital.
• Hospitals, state agencies and the federal government can release statistical data on hospital billing.
• Medical examiners are not covered by HIPAA, although they may be covered by state privacy laws.
• In some states, you may be able to get information from public ambulance or emergency medical services.
• People are entitled to their own records, and can share them with journalists if they so choose.
• At ProPublica, we’ve had success obtaining patient-level data through an institutional review board (IRB) process. We found a non-university-affiliated board that was willing to review and approve our protocol, satisfying a government agency that had initially rejected a FOIA request. You can read more about our IRB process here.

When De-Identified Data Isn’t Enough, Try a Restricted-Use Data Set

Organizations covered by HIPAA are allowed to create data sets containing protected health information that may be disclosed for research purposes, with the understanding that the researcher signs a data use agreement. Some agencies give journalists access to limited- or restricted-use data sets, as long as they abide by the same rules as researchers.

• Make sure the data exists. Call the institution and ask what data sets are available in the limited and restricted form.
• Inquire if the data would be available if you sign a data use agreement. Ask about what process researchers generally undergo to obtain data and whether it’s available to journalists as well.
• Review data use agreements with an attorney or your newsroom’s counsel. Make sure you understand the fine print.
• Understand the publishing restrictions. Many agreements limit how the data can be published (for example, an agreement might not allow you to publish raw data). Make sure you are aware of the restrictions before you sign a document, and negotiate the terms if you need to.
• Keep track of termination dates. Some agreements only last a few years. Make sure you update the contract when necessary.

What to Include in the Data Use Agreement:

• Statement of intent: This statement should include information about your news organization, what you plan to study, and the goals of your research.
• Description of the data: The agreement should identify the specific data files that the organization is providing, including the time periods of the data sets. Always double check that you’re being given the time frame you want.
• Payment: You should negotiate the fee for the data before seeing the data use agreement. Prices may be pre-set and listed on a health agency’s website.
• Data uses: Most data use agreements explain clearly how the data can be used and what text should be included in your story related to source of data.
• List of all reporters: With every agreement, include a list of all reporters who are or may be interested in working with the data.

Even with FERPA, You Can Obtain Student Data

Depending on the school or school system, you may still receive “directory information,” which describes aspects of a student’s educational record that would not “be considered harmful or an invasion of privacy if disclosed.” It typically includes: name, address, phone numbers, emails, photo, participation in activities or sports, dates of attendance, major field of study, grade level, enrollment status, weight and height of athletes, degrees, honors or awards and most recent educational institution attended. Under the law, parents may request to remove their child’s information from a directory.

Aside From Directory Information, FERPA Allows You to Request:

• Instructional data: Data related to teaching and administrative roles is not considered “education records,” and is not covered by FERPA.
• Law enforcement or campus security data: School police units are generally not covered by FERPA. If their records are created for a non-law-enforcement purpose (for example, student suspensions), the data may be subject to FERPA restrictions. Additionally, states have varying privacy laws related to juvenile crime.
• Health or emergency records: Institutions may release records without a student’s or parent’s permission if necessary to safeguard student health or safety.
• School employment data: Data related to staff members at the school is not covered by FERPA, unless the employee is also a student.
• Scholarship and award data: Because it can be considered directory data, this information may not be subject to FERPA restrictions. Still, colleges and universities have tried to use FERPA to deny requests for this information, prompting some reporters to file lawsuits seeking its release, with mixed success.
• Spending or contracting data: As this data does not relate to “education records,” it is, for the most part, not covered by FERPA.

To sidestep FERPA restrictions, you may be able to request aggregated data or for the personally identifiable fields to be redacted or removed. Sometimes, as with HIPAA, an entity will require a journalist to sign a data use agreement. Many of the same tips related to HIPAA apply.

Have any additional questions? Feel free to email Annie at annie.waldman@propublica.org.