The Spanish Urban Exodus

2020

Category: Best data-driven reporting (small and large newsrooms)

Country/area: Spain

Organisation: El Confidencial

Organisation size: Big

Publication date: 27/09/2019

Credit: Javier G. Jorrín, María Zuil, Jesús Escudero, Laura Martín, Antonio Hernández, Pablo López Learte, Luis Rodríguez, Pablo Narváez

Project description:

The Spanish Urban Exodus project did a microdata analysis of Spain’s internal migration data from 1988 to 2018, based on data from the Spanish Statistics Institute. After compiling more than 51 million records and analyzing more than 15 million inter-provinces migrations, this data-driven project details how the two biggest cities of Spain, Barcelona and specially Madrid, attract high-skilled young people from mid-sized Spanish cities in a brain drain process linked to the 21st century phenomenon of the metropolization of Spain.

Impact reached:

The project shows an under-reporting new phenomenon in Spain: the internal brain drain capitalized by Madrid and Barcelona of getting high skilled young workers from mid-sized cities, impoverishing these towns and provinces. This is mostly known as capital effect: national capitals and big cities are a pole of attraction of skilled labor because big companies and high-tech jobs are located in these areas. Unlike the rural exodus, where cheap labor left small villages going to cities and towns where factories and industries were located, the urban exodus attracts high skilled workers from these towns, decapitalizing these areas, employing them as high skilled workers in services companies.

The seven pieces of the project shows the extent of this recent phenomenon in Spain, specially in Madrid. Nevertheless, the arrivals of new skilled workers to Madrid and Barcelona have a side effect: the rise in housing prices and the expulsion of low skilled workers to the neighboring provinces, where homes are more affordable, making commutes more difficult. The project also shows two exceptions to this capital effect, Basque Country and Málaga, who have found a way to retain and even attract skilled workers.

The impact of the project was huge, both quantitatively and qualitatively. The seven pieces reached nearly 500,000 views since their publication, and all the articles reached at least the top 3 most read articles of every day of publication. Qualitatively, we were able for the first time to compile 31 microdata files in one file to explore more than 51 million migration records of the last three decades (the R scripts with the process are published in our GitHub). Also we could show the inter-province migration flows of every Spanish city with more than 10,000 inhabitants in an interactive dashboard.

Techniques/technologies used:

The seven pieces of the Urban Exodus project combines a great number of journalist techniques: data analysis, interactive and static infographics, maps, interactive dashboards, reporting trips, photographs, layout and design.

The compilation and the analysis of the microdata files published every year by the Spanish Statistics Institute were made with R due to the large size of the database (the final csv file has more than 51 million records and 15 columns and has a size of 3.3 GB; the R script of the compilation process is published in our GitHub: https://github.com/ECLaboratorio/unidad-de-datos/tree/master/proyectos/migraciones_espana). We also produced several csv files, tables and infographics with R to help us to identify reporting leads for our series.

The interactive dashboards, infographics and maps were made with D3.js and Javascript and rendered with Canvas to improve the data loading on desktop and mobile.

What was the hardest part of this project?

The project has two main difficulties: the compilation of the 31 microdata files published by the Spanish Statistics Institute (one per year between 1988 and 2018) and the data loading in the interactive dashboards and infographics. The data compilation were made with R and, due to the large size of the files and the lack of enough computer memory, the process was very slow and restarted back and forth. The solution found was to split the compilation process to obtain a file for every five years and then to join these five year files in an only one. With this original raw file, we did the data analysis, categorizing the migrations in three types: intra-province, inter-province and foreign migration. We focused on the inter-province migrations because the main reason for a person to change his residence to another province is for work reasons, so this type of migration depends mainly on work opportunities.

To solve the data loading problems in the interactive dashboards and infographics, we used D3.js and Javascript and rendered then with Canvas to improve the loading on desktop and mobile. This process was a great success, especially in the map that heads the overview (https://www.elconfidencial.com/economia/2019-09-27/exodo-urbano-espana-migraciones-provincias_2240119/) and in the interactive dashboard which shows the urban migration flow of every Spanish city above 10,000 inhabitants (https://www.elconfidencial.com/economia/2019-09-27/exodo-urbano-espana-migraciones-ciudades-buscador_2251515/). Nevertheless, to improve the data loading we also selected the 396 main flows between provinces (15% of the total) which represent more than 10.77 million migrations since 1988 (71% of the total).

What can others learn from this project?

The first thing others can learn from the Urban Exodus project is the potential of the microdata published by National Statistics Institutes all around the world. These microdata files requires a hard work to compile them in an unique file or database to explore the data in all its extension because National Statistics Institutes used to publish the microdata yearly, so the first step to work with microdata must be to compile and combine them. The big advantage of this process is that once you have combined them, you can upgrade it every year with new data releases by National Statistics Institutes. And also these big databases offer journalists leads to several reports and news articles.