Efficient small file management in Hadoop distributed file system for enhanced e-government services

Fredrick Ishengoma

Technological Sustainability, 2025, vol. 4, issue 2, 160-180

Abstract: Purpose - This paper introduces the Efficient Small File Management Algorithm (ESFMA) to overcome the challenge of small file inefficiency of Hadoop distributed file system (HDFS) for e-government services. Design/methodology/approach - ESFMA is designed with the following features: hierarchical metadata architecture, caching, block aggregation, prefetching and locality-aware data placement. These are intended to optimize NameNode memory usage, metadata handling, data block management, I/O and network performance. The algorithm was implemented in experiments on HDFS with real e-government small files. Findings - The experiments showed that ESFMA saves 10% of NameNode memory, 12% of metadata requests, 3.8% of data block use, 15% of read latency, 17% of write latency and 10% of network traffic. Practical implications - This study suggests that implementation of ESFMA has the potential to enable better e-government services in HDFS to be run efficiently and effectively. Originality/value - This paper presents an algorithm for small file management in HDFS, filling an important need in improving service efficiency and performance in e-government services.

Keywords: e-Government services; Hadoop distributed file system (HDFS); Small file management; NameNode memory usage; Metadata operations (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.emerald.com/insight/content/doi/10.110 ... d&utm_campaign=repec (text/html)
https://www.emerald.com/insight/content/doi/10.110 ... d&utm_campaign=repec (application/pdf)
Access to full text is restricted to subscribers

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eme:techsp:techs-08-2024-0114

DOI: 10.1108/TECHS-08-2024-0114

Access Statistics for this article

Technological Sustainability is currently edited by Dr Shahla Seifi

More articles in Technological Sustainability from Emerald Group Publishing Limited
Bibliographic data for series maintained by Emerald Support ().