Optimizing ETL by a Two-Level Data Staging Method

Xiufeng Liu, Nadeem Iftikhar, Huan Huo and Per Sieverts Nielsen
Additional contact information
Xiufeng Liu: Department of Management Engineering, Technical University of Denmark, Kongens Lyngby, Denmark
Nadeem Iftikhar: University College of Northern Denmark, Aalborg, Denmark
Huan Huo: Shanghai University of Science and Technology, China
Per Sieverts Nielsen: Technical University of Denmark, Kongens Lyngby, Denmark

International Journal of Data Warehousing and Mining (IJDWM), 2016, vol. 12, issue 3, 32-50

Abstract: In data warehousing, the data from source systems are populated into a central data warehouse (DW) through extraction, transformation and loading (ETL). The standard ETL approach usually uses sequential jobs to process the data with dependencies, such as dimension and fact data. It is a non-trivial task to process the so-called early-/late-arriving data, which arrive out of order. This paper proposes a two-level data staging area method to optimize ETL. The proposed method is an all-in-one solution that supports processing different types of data from operational systems, including early-/late-arriving data, and fast-/slowly-changing data. The introduced additional staging area decouples loading process from data extraction and transformation, which improves ETL flexibility and minimizes intervention to the data warehouse. This paper evaluates the proposed method empirically, which shows that it is more efficient and less intrusive than the standard ETL method.

Date: 2016
References: Add references at CitEc
Citations:

Downloads: (external link)
http://services.igi-global.com/resolvedoi/resolve. ... 018/IJDWM.2016070103 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:igg:jdwm00:v:12:y:2016:i:3:p:32-50

Access Statistics for this article

International Journal of Data Warehousing and Mining (IJDWM) is currently edited by Eric Pardede

More articles in International Journal of Data Warehousing and Mining (IJDWM) from IGI Global
Bibliographic data for series maintained by Journal Editor ().