Data cleaning in Stata using Internet search engines

Sergiy Radyakin

Canadian Stata Users' Group Meetings 2009 from Stata Users Group

Abstract: Open-ended questions can be a nightmare for statistical processing. Any mistake in spelling can result in a mismatch during merging, or multiple counting of the same object. For example, the answers to the "place-of-birth" question might be "Chicago" and "San Francisco", but in practice they are often "Chicaga" and "SanFrancisko". Manual correction of hundreds of answers is tedious, and becomes infeasible with a larger dataset. For a long time, algorithms like SOUNDEX remained the only alternative for researchers. A new Stata command allows taking advantage of Internet search engines, like Google or Yahoo to find proper substitutes for an unclear word or multiple words. The distinctive feature of the search engines is that they rely not only on the spelling similarity, but are also context driven: other words may affect the suggestion, such as including "city" into the query. This will hint to the search engine to give more priority to the names of cities. This presentation will demonstrate this new command and explain the main steps necessary to programmatically acquire information available on the Internet and convert it into Stata-usable format.

Keywords: data cleaning; search engine; spelling correction (search for similar items in EconPapers)
Date: 2009-11-06
References: Add references at CitEc
Citations:

Downloads: (external link)
http://fmwww.bc.edu/repec/csug2009/ca09_radyakin.pdf (application/pdf)
http://fmwww.bc.edu/repec/csug2009/ca09_radyakin.wmv

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:boc:csug09:08

Access Statistics for this paper

More papers in Canadian Stata Users' Group Meetings 2009 from Stata Users Group Contact information at EDIRC.
Bibliographic data for series maintained by Christopher F Baum ().