Distributed Index Optimization Techniques for Time-Sensitive Massive Data

Ruohua Yuan

European Journal of AI, Computing & Informatics, 2026, vol. 2, issue 3, 12-20

Abstract: With the continuous deployment of industrial perception networks and smart urban monitoring systems, the volume of temporal stream data has expanded significantly, evolving from terabyte-level to petabyte-scale volumes. The high-throughput and persistent nature of data generation imposes stringent demands on storage and retrieval architectures. Traditional single-machine indexing architectures suffer from limited computational power and storage capacity, making them inadequate for processing ultra-large-scale temporal datasets. Existing distributed indexing solutions predominantly employ fixed sharding mechanisms, often encountering challenges such as uneven node load distribution, high query interaction costs across nodes, and inefficient reuse of cold and hot data indices, thereby failing to simultaneously ensure stable high-concurrency writes and low-latency multi-dimensional queries. Addressing these technical challenges, this paper proposes a dual-layer hybrid distributed indexing architecture that integrates temporal data characteristics with hierarchical storage techniques. The proposed approach compresses index metadata using temporal feature hash summaries, implements dynamic scheduling and partition pruning mechanisms based on data access frequency, and optimizes the underlying logic for log-structured merge tree merging operations. Experimental results across multiple benchmark datasets confirm the effectiveness of the proposed solution in reducing indexing overhead, substantially lowering cross-partition query latency, and achieving balanced cluster workload distribution. The findings provide a viable and scalable technical framework for the efficient distributed retrieval of massive temporal datasets in real-world industrial and urban monitoring applications.

Keywords: time-series data; distributed indexing; data sharding; hierarchical storage; query optimization (search for similar items in EconPapers)
Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://pinnaclepubs.com/index.php/EJACI/article/view/988/947 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:dba:ejacia:v:2:y:2026:i:3:p:12-20

Access Statistics for this article

More articles in European Journal of AI, Computing & Informatics from Pinnacle Academic Press
Bibliographic data for series maintained by Joseph Clark ().