World Wide Web Robot for Extreme Datamining with Swiss-Tx Supercomputers

A.S.A. Roehrl, M. Frey and R.A. Roehrl
Authors registered in the RePEc Author Service: Richard Alexander Roehrl

Working Papers from International Institute for Applied Systems Analysis

Abstract: This paper discusses the software and hardware issues of designing a highly parallel robot for extreme datamining on the Internet. As a sample application, a World Wide Web server count experiment for Switzerland and Thailand is presented. Our platform of choice is the SwissTx, a supercomputer built from commodity components that runs NT and COMPAQ tru64 UNIX. Hardware and software of this machine are discussed and benchmark results presented. They show that NT is a feasible choice even under the given extreme conditions. Using statistical modelling for optimizing the search process, the inevitable bandwidth problem is reduced to some extent to a computation problem. We suggest that our approach to Web robots is a robust bet for a multitude of future Internet applications which might lead to a large-scale and cost-efficient usage of Web robots.

Date: 1999-06
New Economics Papers: this item is included in nep-cmp
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.iiasa.ac.at/Publications/Documents/IR-99-020.pdf (application/pdf)
Our link check indicates that this URL is bad, the error code is: 404 Not Found (http://www.iiasa.ac.at/Publications/Documents/IR-99-020.pdf [301 Moved Permanently]--> https://www.iiasa.ac.at/Publications/Documents/IR-99-020.pdf [301 Moved Permanently]--> https://iiasa.ac.at//Publications/Documents/IR-99-020.pdf [302 Found]--> https://iiasa.ac.at/Publications/Documents/IR-99-020.pdf)
http://www.iiasa.ac.at/Publications/Documents/IR-99-020.ps (application/postscript)
Our link check indicates that this URL is bad, the error code is: 404 Not Found (http://www.iiasa.ac.at/Publications/Documents/IR-99-020.ps [301 Moved Permanently]--> https://www.iiasa.ac.at/Publications/Documents/IR-99-020.ps [301 Moved Permanently]--> https://iiasa.ac.at//Publications/Documents/IR-99-020.ps [302 Found]--> https://iiasa.ac.at/Publications/Documents/IR-99-020.ps)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wop:iasawp:ir99020

Access Statistics for this paper

More papers in Working Papers from International Institute for Applied Systems Analysis Contact information at EDIRC.
Bibliographic data for series maintained by Thomas Krichel ().