Revealing top-k dominant individuals in incomplete data based on spark environment

Ke Wang (), Binge Cui (), Jerry Chun-Wei Lin () and Jimmy Ming-Tai Wu ()
Additional contact information
Ke Wang: Shandong University of Science and Technology
Binge Cui: Shandong University of Science and Technology
Jerry Chun-Wei Lin: Western Norway University of Applied Sciences
Jimmy Ming-Tai Wu: Shandong University of Science and Technology

Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, 2025, vol. 27, issue 10, No 64, 24837-24857

Abstract: Abstract Incomplete data set is a new type of data set that arises due to various reasons. For example, when performing data transmission, some data are lost due to abnormal signal interruptions; when acquiring gene expression profile data, dust on gene chips and other reasons can also lead to the final acquired data being incomplete. Top-k dominance (TKD) query returns the k data with the largest dominance score in a given dataset. For large scale incomplete datasets with missing data in unknown dimensions, most of the research is based on the Hadoop MapReduce framework, but the algorithm performance is poor because the Hadoop MapReduce computing framework is not good at multi-task iterative computing and has a long start-up time, etc. The Spark framework is a more efficient data processing framework with a rich computational model and in-memory based implementation of data processing. Based on the above analysis, this paper proposes a query algorithm (Spark_TKD) based on Spark framework, which designs a simple object dominating number calculation method, greatly reducing the computational complexity and the interaction of data between cluster nodes, and reducing disk I/O operations. At the end of the paper, comparison experiments are conducted using real and synthetic datasets, and the experimental results show that our proposed algorithm exhibits better performance in terms of time consumption and disk footprint.

Keywords: Incomplete dataset; Top-k dominance query; MapReduce; Spark (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10668-022-02652-5 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:endesu:v:27:y:2025:i:10:d:10.1007_s10668-022-02652-5

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/10668

DOI: 10.1007/s10668-022-02652-5

Access Statistics for this article

Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development is currently edited by Luc Hens

More articles in Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().