Content removal bias in web scraped data: A solution applied to real estate ads

Gabriele Marconi

Open Economics, 2022, vol. 5, issue 1, 30-42

Abstract: I propose a solution to content removal bias in statistics from web scraped data. Content removal bias occurs when data is removed from the web before a scraper is able to collect it. The solution I propose is based on inverse probability weights, derived from the parameters of a survival function with complex forms of data censoring. I apply this solution to the calculation of the proportion of newly built dwellings with web scraped data on Luxembourg, and I run a counterfactual experiment and a Montecarlo simulation to confirm the findings. The results show that the extent of content removal bias is relatively small if the scraping occurs frequently compared with the online permanence of the data; and that it grows larger with less frequent scraping.

Keywords: Scraping; Missing data; Survival analysis; Inverse probability weighting; Crawling; Bias; Real estate; Housing; Luxembourg; Big data (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1515/openec-2022-0119 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:vrs:openec:v:5:y:2022:i:1:p:30-42:n:2

DOI: 10.1515/openec-2022-0119

Access Statistics for this article

Open Economics is currently edited by Friedrich Schneider

More articles in Open Economics from De Gruyter
Bibliographic data for series maintained by Peter Golla ().