A group of New York Times journalists — members of the NewsGuild of New York — conducted a statistical analysis of The Times’s performance review system, and found that their news organization had for years given significantly lower ratings to employees of color.
The report, which spurred changes in the practices of one of the country’s largest newsrooms, relied on data journalism techniques to further the goals of diversity and fairness, in service not only of journalists but also of readers.
The goals of this project were similar to those envisioned regularly by journalists — revealing systemic inequalities and inspiring change to correct them. But this time, the journalists were turning a lens on an unusual center of power: their own newsroom.
The project had almost immediate effects: Despite having previously denied and downplayed the racial disparities in the company’s performance review process, The Times’s leaders soon formed an internal team of top editors to tackle the problem.
As a result of that review, the company this year is providing additional training for managers and a more rigorous review of managers’ proposed ratings for their employees. Journalists in the NewsGuild will conduct follow-up analyses to see whether these procedural changes lead to improvements, and will continue to press the company to provide a fair work environment. The goal is to ensure journalists from diverse backgrounds are fully welcome and supported at The TImes, so that the newsroom can better report on New York City and the country as a whole.
Since the project was published, it has been discussed in dozens of other newsrooms across the country. We understand that several are now working on their own analyses.
We used a wide range of statistical modeling techniques, including:
* linear regression
* ordered logistic regression (our main model)
* Poisson regression
* Hierarchical mixed-effects models
* Kruskal-Wallis and Cochran-Mantel-Haenszel tests
We also replicated the methodology the company’s consultant had used in claiming that its system did not show evidence of disparities — and ran simulations using dummy data to test it. We also made several static data visualizations to help communicate our findings to readers.
All of this was done in the statistical programming language R. We shared outputs from our analysis with a range of experts, including leading econometricians who study racial discrimination.
Context about the project:
Obviously, a group of journalists and other employees coming together to write a report that criticizes theit own employer is uncomfortable. The protections labor law gives us, as union members, to speak freely about the terms of our employment were critical in our ability to both do this project and to publish it publicly.
We also relied on data that the company is required, under our contract, to provide to the union. Performance review data is quite sensitive, and workers would not normally have access to this kind of information absent a negotiated union contract requiring the company to provide it.
What can other journalists learn from this project?
It can be useful to train the kind of investigative and analytical lens we often apply to companies and organizations on our own newsrooms. These newsrooms can be as powerful and as important to society as any corporation or local government entity, and it is important that they be held accountable.