2 Way Crawling: A Review

Mayuri Anantrao Deshmukh
Additional contact information
Mayuri Anantrao Deshmukh: MIT College of Engineering Aurangabad, Pune, India

International Journal of Applied Evolutionary Computation (IJAEC), 2019, vol. 10, issue 3, 34-39

Abstract: As we know that the deep web grows at very fast pace, there has been increased interest in techniques which help efficiently locate and check deep web interfaces. So, it is important to achieve wide coverage and high efficiency on the large volume of web resources. For this we propose a multistage framework, Smart crawler. Smart crawler is a two-stage crawler used to efficiently harvest deep web interfaces. In the first stage, the crawler performs site-based searching for center pages and avoids visiting non-relevant sites. In the second stage, an adaptive link ranking technique is used which helps to searching relevant site by excavating most relevant links. It is important to eliminate bias on visiting highly relevant links which is hidden in web directories, for this a link tree data structure is designed to achieve wider coverage for a website. The proposed framework gives experimental result on different domains and shows the agility and accuracy of the proposed framework, which retrieves deep-web interfaces from a large volume of sites and achieves higher harvest rates than other crawler.

Date: 2019
References: Add references at CitEc
Citations:

Downloads: (external link)
http://services.igi-global.com/resolvedoi/resolve. ... 018/IJAEC.2019070105 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:igg:jaec00:v:10:y:2019:i:3:p:34-39

Access Statistics for this article

International Journal of Applied Evolutionary Computation (IJAEC) is currently edited by Sukhpal Singh Gill

More articles in International Journal of Applied Evolutionary Computation (IJAEC) from IGI Global
Bibliographic data for series maintained by Journal Editor ().