Towards Efficient Big Data Storage With MapReduce Deduplication System

Vijesh Joe, Jennifer S. Raj and Smys S.
Additional contact information
Vijesh Joe: VV College of Engineering, India
Jennifer S. Raj: Gnanamani College of Technology, India
Smys S.: RVS Technical Campus, India

International Journal of Information Technology and Web Engineering (IJITWE), 2021, vol. 16, issue 2, 45-57

Abstract: In the big data era, there is a high requirement for data storage and processing. The conventional approach faces a great challenge, and de-duplication is an excellent approach to reduce the storage space and computational time. Many existing approaches take much time to pinpoint the similar data. MapReduce de-duplication system is proposed to attain high duplication ratio. MapReduce is the parallel processing approach that helps to process large number of files in less time. The proposed system uses two threshold two divisor with switch algorithm for chunking. Switch is the average parameter used by TTTD-S to minimize the chunk size variance. Hashing using SHA-3 and fractal tree indexing is used here. In fractal index tree, read and write takes place at the same time. Data size after de-duplication, de-duplication ratio, throughput, hash time, chunk time, and de-duplication time are the parameters used. The performance of the system is tested by college scorecard and ZCTA dataset. The experimental results show that the proposed system can lessen the duplicity and processing time.

Date: 2021
References: Add references at CitEc
Citations:

Downloads: (external link)
http://services.igi-global.com/resolvedoi/resolve. ... 18/IJITWE.2021040103 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:igg:jitwe0:v:16:y:2021:i:2:p:45-57

Access Statistics for this article

International Journal of Information Technology and Web Engineering (IJITWE) is currently edited by Ghazi I. Alkhatib

More articles in International Journal of Information Technology and Web Engineering (IJITWE) from IGI Global
Bibliographic data for series maintained by Journal Editor ().