Optimization Issues in Web Search Engines

Zhen Liu () and Philippe Nain ()
Additional contact information
Zhen Liu: IBM Research
Philippe Nain: NRIA

Chapter 34 in Handbook of Optimization in Telecommunications, 2006, pp 981-1015 from Springer

Abstract: Abstract Crawlers are deployed by a Web search engine for collecting information from different Web servers in order to maintain the currency of its data base of Web pages. We present studies on the optimization of Web search engines from different perspectives. We first investigate the number of crawlers to be used by a search engine so as to maximize the currency of the data base without putting an unnecessary load on the network. Both the static setting, where crawlers are always active, and the dynamic setting where, crawlers may be activated/deactivated as a function of the state of the system, are addressed. We then consider the optimal scheduling of the visits of these crawlers to the Web pages assuming these pages are modified at different rates. Finally, we briefly discuss some other optimization issues of Web search engines, including page ranking and system optimization.

Keywords: Web search engines; web crawlers; scheduling; optimal control; queues; Markov decision process (search for similar items in EconPapers)
Date: 2006
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sprchp:978-0-387-30165-5_34

Ordering information: This item can be ordered from
http://www.springer.com/9780387301655

DOI: 10.1007/978-0-387-30165-5_34

Access Statistics for this chapter

More chapters in Springer Books from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().