A classification framework for web robots

Derek Doran and Swapna S. Gokhale

Journal of the American Society for Information Science and Technology, 2012, vol. 63, issue 12, 2549-2554

Abstract: The behavior of modern web robots varies widely when they crawl for different purposes. In this article, we present a framework to classify these web robots from two orthogonal perspectives, namely, their functionality and the types of resources they consume. Applying the classification framework to a year‐long access log from the UConn SoE web server, we present trends that point to significant differences in their crawling behavior.

Date: 2012
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/asi.22741

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jamist:v:63:y:2012:i:12:p:2549-2554

Ordering information: This journal article can be ordered from
https://doi.org/10.1002/(ISSN)1532-2890

Access Statistics for this article

More articles in Journal of the American Society for Information Science and Technology from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().