Study Reveals Racial Disparities in Performance Evaluations at The New York Times

2023

Study Reveals Racial Disparities in Performance Evaluations at The New York Times

Entry type: Single project

Country/area: United States

Publishing organisation: The NewsGuild of New York

Organisation size: Big

Publication date: 2022-08-23

Language: English

Authors: Davey Alba, Bill Baker, Larry Buchanan, Quoctrung Bui, Alex Carp, Ben Casselman, Nick Confessore, Stacy Cowley, Susan DeCarava, Michael H. Keller, Ella Koeze, Dan Lenos, Denise Lu, Jim Luttrell, Emma Odette, Shira Ovide, Frenchie Robles, Jennifer Valentino-DeVries

Biography:

This was a project carried out by members of Times-Guild Equity Committee, a group of journalists and other Times employees represented by the NewsGuild union. The Equity Team reviews pay data, performance evaluation data and other records provided to the union by the company to identify — and press for action to address — disparities that may be influenced by age, race, gender, sexual orientation or other demographic characteristics.

Project description:

A group of New York Times journalists — members of the NewsGuild of New York — conducted a statistical analysis of The Times’s performance review system, and found that their news organization had for years given significantly lower ratings to employees of color.

The report, which spurred changes in the practices of one of the country’s largest newsrooms, relied on data journalism techniques to further the goals of diversity and fairness, in service not only of journalists but also of readers.

Impact reached:

The goals of this project were similar to those envisioned regularly by journalists — revealing systemic inequalities and inspiring change to correct them. But this time, the journalists were turning a lens on an unusual center of power: their own newsroom.

The project had almost immediate effects: Despite having previously denied and downplayed the racial disparities in the company’s performance review process, The Times’s leaders soon formed an internal team of top editors to tackle the problem.

As a result of that review, the company this year is providing additional training for managers and a more rigorous review of managers’ proposed ratings for their employees. Journalists in the NewsGuild will conduct follow-up analyses to see whether these procedural changes lead to improvements, and will continue to press the company to provide a fair work environment. The goal is to ensure journalists from diverse backgrounds are fully welcome and supported at The TImes, so that the newsroom can better report on New York City and the country as a whole.

Since the project was published, it has been discussed in dozens of other newsrooms across the country. We understand that several are now working on their own analyses.

Techniques/technologies used:

We used a wide range of statistical modeling techniques, including:

* linear regression
* ordered logistic regression (our main model)
* Poisson regression
* Hierarchical mixed-effects models
* Kruskal-Wallis and Cochran-Mantel-Haenszel tests

We also replicated the methodology the company’s consultant had used in claiming that its system did not show evidence of disparities — and ran simulations using dummy data to test it. We also made several static data visualizations to help communicate our findings to readers.

All of this was done in the statistical programming language R. We shared outputs from our analysis with a range of experts, including leading econometricians who study racial discrimination.

Context about the project:

Obviously, a group of journalists and other employees coming together to write a report that criticizes theit own employer is uncomfortable. The protections labor law gives us, as union members, to speak freely about the terms of our employment were critical in our ability to both do this project and to publish it publicly.

We also relied on data that the company is required, under our contract, to provide to the union. Performance review data is quite sensitive, and workers would not normally have access to this kind of information absent a negotiated union contract requiring the company to provide it.

What can other journalists learn from this project?

It can be useful to train the kind of investigative and analytical lens we often apply to companies and organizations on our own newsrooms. These newsrooms can be as powerful and as important to society as any corporation or local government entity, and it is important that they be held accountable.