Project description:

Alex Homer from the BBC’s Shared Data Unit carried out the most in-depth analysis to date of the controversial Right to Buy policy, which allows council tenants to purchase their council homes at a discount.  He used a variety of investigative and data journalism techniques to establish whether there was any evidence that the purpose of the policy was being undermined by former tenants treating it as a get-rich-quick scheme.  The story involved sourcing multiple datasets and using data analysis to paint a comprehensive UK-wide picture of a policy that has come under fierce scrutiny.

Impact reached:

This report and research sparked widespread discussion. It was carried by three national newspapers, it had 1.6m readers on the BBC website and it was broadcast on television by the BBC News Channel. It was debated on TV, radio and across social media. A further 65 regional media outlets used Alex’s research in reports of their own, as did nine BBC local radio stations and a further local BBC TV news programme. 

It was published against a background of Scotland and Wales having brought the policy to an end, and while Northern Ireland consulted on the future of its equivalent scheme. An extension of the policy to include housing association tenants was however being trailed in England.

Supporters said this policy had given millions of people the chance to get on the housing ladder and secure their families’ financial future. Opponents blamed the policy for distorting the housing market and for a huge reduction in the amount of social housing stock. Our analysis – the first on this UK-wide scale, covering 18 years’ of data – revealed around 92,000 former council homes which were later resold, made a collective £6.4bn profit in real terms.

Alex’s findings led Labour’s London Assembly housing spokesman to renew his call for the policy to end https://twitter.com/tomcopley/status/1106173993881649152 and the Mayor of Newham to say the same https://www.newhamrecorder.co.uk/news/newham-council-right-to-buy-resales-1-5956677.

Many more of Alex’s interviewees called for the existing scheme to be halted. His Interviews held the politicians of the day to account over the trial extension of the policy and the author put further questions about its impact to one architect of the scheme nearly 40 years ago, Lord Heseltine, although he declined to answer.

This report was featured in the top 10 round-up of data journalism projects by the Global Investigative Journalism Network: https://gijn.org/2019/03/21/gijns-data-journalism-top-10-frances-shocking-yellow-vest-injuries-yield-curve-music-migration-maps/.

Techniques/technologies used:

A variety of techniques were used to unearth these previously-unpublished data.

Alex submitted requests for information under the Freedom of Information Act (FOI) to HM Land Registry for data covering England and Wales. He submitted similar requests to the Northern Ireland Housing Executive and requested a bespoke dataset from the Registers of Scotland. Northern Ireland’s FOI response was 83 pages of scans of a paper ledger in PDF files. It was not possible to digitise with OCR software so Alex spent days manually inputting it into a public-facing Googlesheet to enrich the Commons.

In his analysis, Alex factored in inflation to establish real-terms price differences by sourcing CPI data from the Office for National Statistics to match to dates of sales. He normalised the findings on profits by establishing the profits in Pound sterling generated per day of ownership by former council tenants before they resold their homes. He also matched homes sold to government regions. He visualised the report with charts and an interactive Carto map, illustrating that vendors in the south east of England made more money in real terms from the year 2000-2018.

Interviews with people with direct experience of this policy or impacted by the shortage of social housing, made this report engaging, meaningful and relatable to a wide audience. The data were published on the BBC Shared Data Unit’s public-facing Github page and are available for reuse and further interrogation.

What was the hardest part of this project?

A challenge that came up during the data wrangling for this story was because these were previously-unpublished data, they had not been previously exposed to the same level of scrutiny as datasets which are routinally and habitually used in news reports. During Alex’s Right to Buy investigation, he was presented with around 150,000 rows of data – each row representing a property title in Great Britain formerly sold under the policy. Both HM Land Registry and Registers of Scotland underline they cannot guarantee their datasets are error-free. Alex found some 60,080 rows did not have comparable sales prices. When he highlighted this, HM Land Registry re-ran its script and found 2,582 more price entries so he updated his calculations. Similarly, he encountered 734 dates anomalies in Scottish data. This underlined the importance for any journalist not to treat a dataset as a source of objective truth but, just like any interviewee, to ask it questions and not simply accept its first answers.

Another challenge was posed by the lack of digital records of property sales in Northern Ireland. When Northern Ireland Housing Executive sent its response to Alex’s request for data under the Freedom of Information Act (FOI) it sent 83 pages of scans of a paper ledger in PDF files. First, Alex sought to establish if these data could be digitised with OCR software, but when that could not be achieved, Alex decided it would still be of public value to make these data open themselves so he spent days manually inputting it into a public-facing Googlesheet to enrich the Commons. The data were published on the BBC Shared Data Unit’s public-facing Github page, in an inline link from the BBC online report and are available for reuse and further interrogation because of his labours.

What can others learn from this project?

A significant takeaway from this project was carrying out this data analysis, and having the opportunity to have an open dialogue with the data holders about data quality, led to improved data quality and presentation so those data were easier for everyone to interpret and they had greater public value as open data, when they were brought into the public domain. The different approaches of the devolved governments across the UK might also have provided a reason to restrict the scope of this study, which Alex refused to do and it led to a more impactful investigation. Comparing sales across the UK was complex due to differences in policy: Scotland had brought the policy to end in 2016, Wales had done the same in January 2019 and Northern Ireland did not hold data on the prices paid for properties nor the duration of their ownership. Despite the difficulties, Alex realised these nations’ data would provide valuable insights into the consequences of the policy there. Together with patterns found in England, the research was able to inform questions that needed to be put to government ministers in England, as they intended to further extend the policy in England to include housing association tenants in addition to council tenants. Another key takeaway was Alex realised the limitations of the dataset and was clear on them in his reporting. Each row in the data represented a property title: sometimes they were merged or split as different owners bought/sold/developed on their land. Due to rare enormous profit margins in the data, Alex used median averages for fairness as he could not investigate the idiosyncracies of every individual property title changing over time, as there were tens of thousands of them. The data also did not include the levels of discount repaid by ex-council tenants

Project links:

www.bbc.co.uk/news/uk-47443183

github.com/BBC-Data-Unit/right-to-buy