Automated Data Harmonization (ADH) using Artificial Intelligence (AI)

Anjan Dutta (), Tomal Deb and Shrikant Pathak
Additional contact information
Anjan Dutta: Tata Consultancy Services Ltd
Tomal Deb: Tata Consultancy Services Ltd
Shrikant Pathak: Tata Consultancy Services Ltd

OPSEARCH, 2021, vol. 58, issue 2, No 1, 257-275

Abstract: Abstract Organizations in the business of Information Services deal with very large volumes of data which is collected from a variety of proprietary, as well as, public sources in multiple languages with different formats, naming conventions, and context. Mapping such data into enterprise master data for reporting and prediction is an effort-intensive, time-consuming process which is prone to errors. Machines cannot match these sources and map to master data accurately. Enterprises are eager to automate the human intensive tasks of data harmonization so that their resources can focus on finding the insights to drive the business. We undertook one such automation initiative for a global Market Research Major (MRM) and achieved a significant level of success leveraging Artificial Intelligence (AI) techniques. The Automated Data Harmonization (ADH) solution has been a multi-step approach of Dictionary Matching, Fuzzy Text Similarity, and different Machine Learning techniques. It has been implemented on the Big Data stack for better performance and scalability. In order to streamline the overall business process, runtime rules and workflow has been implemented. The Proof of Concept has yielded an overall F-Score within the range of 82–93% depending on the variation of the dataset. The deployed version is continuing to deliver high accuracy and gained adoption as a core micro-service across the organization. The Business as Usual (BAU) cycle time has been reduced by 80% (from 14 days to 3 days). While the solution is unique and tailored to meet a set of specific business requirements, it can be extended for media metadata standardization across multiple devices, author name and citation resolution in scholarly journals, leads resolution in multi-channel marketing and ad campaigns etc.

Keywords: Automation; Data Quality Management; Entity Resolution; Artificial Intelligence; Machine Learning; String Similarity; Big Data (search for similar items in EconPapers)
Date: 2021
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://link.springer.com/10.1007/s12597-020-00467-4 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:opsear:v:58:y:2021:i:2:d:10.1007_s12597-020-00467-4

Ordering information: This journal article can be ordered from
http://www.springer. ... search/journal/12597

DOI: 10.1007/s12597-020-00467-4

Access Statistics for this article

OPSEARCH is currently edited by Birendra Mandal

More articles in OPSEARCH from Springer, Operational Research Society of India
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().