Project description:

This in-depth investigation is of Twitter’s fact-checking system called Community Notes (formerly known as Birdwatch). Musk claimed it to be “a game changer for improving accuracy on Twitter.” By analyzing more than 40,000 notes, parsing the algorithm itself, and talking to current and former Twitter engineers, Bloomberg found major limitations and fundamental flaws in the system. Twitter’s algorithm looks for broad consensus from “both sides” of the ideology spectrum. As a result, the algorithm tosses out 96% of all fact-checking notes and leaves the vast majority of inaccurate tweets—especially on the most divisive topics such as abortion, Covid-19 and elections—unaddressed.

Impact reached:

This timely piece following Elon’s controversial takeover of Twitter and the Community Notes program’s public launch provided crucial context for a new fact-checking technology that could potentially impact millions of Twitter users around the world. It reached a wide audience and was included in many influential newsletters, including the Nieman Lab.

Techniques/technologies used:

We conducted data cleaning and analysis in Python with knowledge in statistics and machine learning, especially on matrix factorization.

The Community Notes algorithm places users along an opinion spectrum, which generally aligns with the political left and right, based on their voting history. However, Twitter does not publish these measurements. We extracted these hidden metrics by running the public data through Twitter’s algorithm, which has been made public on GitHub.

Community Notes’ public data also does not include any information about the tweets themselves and tweet authors (handles) associated with each note. We obtained such additional data through the publicly available Twitter API and matched them back to the Community Notes.

We validated our results with current and former Twitter engineers, as well as results in Twitter’s own research paper.

Because many readers would not be familiar with how Community Notes work, we framed the story’s narrative around examples that illustrated our findings. The first interactive scrolling example of Musk’s tweet stepped readers through how Community Notes work. A simple, stacked histogram shows the breakdown of data. In our graphics presentation we repeated chart forms to help readers understand the topic without having to learn new visual forms all the time. After showing more findings through examples, we included a second, more in-depth scrolling section to explain the larger dataset and findings. The visuals aid readers in understanding how the algorithm works and its limitations.

The graphics were developed using JavaScript and specifically Svelte, supported by front-end development and design work. Sketches were initially made in Adobe Illustrator, to plan out the flow of the graphics. Svelte was used to build scrolling interactive sections, with D3.js used for the charts. CSS animations make the transitions and animations flow easier.

Context about the project:

●Since taking over the company in late October, Elon has revoked Twitter’s policy about fact-checking Covid misinformation, dissolved the company’s Trust and Safety council, and fired thousands of employees, including staff responsible for tracking dangerous or inaccurate posts.
●Research has found misinformation spiked after Elon took over & traditional fact-checkers were fired.
●Leaders from the EU and other parts of the world have issued warnings about misinformation on Twitter following Musk’s takeover.
●Elon Musk has repeatedly touted Community Notes as “a game changer” that would make Twitter “the most accurate source of information about the world.”
●Twitter is rapidly expanding the Community Notes program around the world.

What can other journalists learn from this project?

1.Augment/match public data with additional API data and make connections
2.Test run open source algorithm with real data, to reveal underlying parameters and hidden metrics (such as the opinion/ideology score assigned to each user/note).
3.Break down complex, technical concepts into concrete examples and accessible language.

Project links:

https://www.bloomberg.com/graphics/2022-twitter-birdwatch-community-notes-misinformation-politics/