A Robust Distributed Clustering of Large Data Sets on a Grid of Commodity Machines

Salah Taamneh, Mo’taz Al-Hami, Hani Bani-Salameh and Alaa E. Abdallah
Additional contact information
Salah Taamneh: Department of Computer Science, The Hashemite University, Zarqa 13133, Jordan
Mo’taz Al-Hami: Department of Computer Information Systems, The Hashemite University, Zarqa 13133, Jordan
Hani Bani-Salameh: Department of Software Engineering, The Hashemite University, Zarqa 13133, Jordan
Alaa E. Abdallah: Department of Computer Science, The Hashemite University, Zarqa 13133, Jordan

Data, 2021, vol. 6, issue 7, 1-23

Abstract: Distributed clustering algorithms have proven to be effective in dramatically reducing execution time. However, distributed environments are characterized by a high rate of failure. Nodes can easily become unreachable. Furthermore, it is not guaranteed that messages are delivered to their destination. As a result, fault tolerance mechanisms are of paramount importance to achieve resiliency and guarantee continuous progress. In this paper, a fault-tolerant distributed k-means algorithm is proposed on a grid of commodity machines. Machines in such an environment are connected in a peer-to-peer fashion and managed by a gossip protocol with the actor model used as the concurrency model. The fact that no synchronization is needed makes it a good fit for parallel processing. Using the passive replication technique for the leader node and the active replication technique for the workers, the system exhibited robustness against failures. The results showed that the distributed k-means algorithm with no fault-tolerant mechanisms achieved up to a 34% improvement over the Hadoop-based k-means algorithm, while the robust one achieved up to a 12% improvement. The experiments also showed that the overhead, using such techniques, was negligible. Moreover, the results indicated that losing up to 10% of the messages had no real impact on the overall performance.

Keywords: k-means clustering; distributed k-means algorithm; actor model; active replication; passive replication; peer-to-peer network (search for similar items in EconPapers)
JEL-codes: C8 C80 C81 C82 C83 (search for similar items in EconPapers)
Date: 2021
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2306-5729/6/7/73/pdf (application/pdf)
https://www.mdpi.com/2306-5729/6/7/73/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jdataj:v:6:y:2021:i:7:p:73-:d:590141

Access Statistics for this article

Data is currently edited by Ms. Cecilia Yang

More articles in Data from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().