An Ontology Based Web Crawler with a Near-Duplicate Detection System to Improve the Performance of a Web Crawler

Ngulamu Daines Walowe (), Dr. Michael Kimwele () and Dr. Ann Kibe ()

International Journal of Technology and Systems, 2024, vol. 9, issue 5, 15 - 28

Abstract: Purpose: The aim of the study is to examine how an ontology-based web crawler with a near-duplicate detection system improves the performance of a web crawler. Methodology: The experiment was carried out using secondary data from a sample web site which was used since crawling is an endless process. Using these two approaches, the ontology web crawler would search for relevant searches according to the search query of the user while the near-duplicate detection system would eliminate redundant data. Findings: It was observed that ontology web crawler performed better and faster than a normal crawler. It takes less execution time to search the web than other web crawlers. This is due to the fact that web documents are being filtered by the ontology web crawler such that only relevant web documents are retrieved according to the search query of the user. The relevant documents are further filtered by a near-duplicate detection system by removing web pages that are duplicates of each other and also remove near-duplicate web documents. This further reduces the number of web pages retrieved by the web crawler. This model saves on storage space because of the reduced number of web pages retrieved as it takes care of irrelevant and redundant web pages searched. Unique Contribution to Theory, Practice and Policy: The study recommends that the model can be improved to be dynamic by adding new relations that is the crawler should search for web pages related to the search even if they don’t contain the keywords searched. Domains and concepts should be added when visiting new web pages. Standardization of weights needs to be done because as of now experts assign weights to terms according to the area of expertise and knowledge.

Keywords: Ontology Based Web Crawler; Near-Duplicate Detection System; Performance (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.iprjb.org/journals/index.php/IJTS/article/view/2984/3509 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bdu:ojijts:v:9:y:2024:i:5:p:15-28:id:2984

Access Statistics for this article

More articles in International Journal of Technology and Systems from IPRJB
Bibliographic data for series maintained by Chief Editor ().