Computing at Glasgow University
Paper ID: 7731
DCS Tech Report Number: TR-2004-174

A Suite of Testbeds for the Realistic Evaluation of Peer-to-Peer Information Retrieval Systems
Klampanos,I.A. Jose,J.M. Poznanski,V. Dickman,P.

Publication Type: Tech Report (internal)
Appeared in: DCS Tech Report
Page Numbers :
Publisher: N/A
Year: 2004

Peer-to-peer (P2P) networking continuously gains popularity among computing science researchers. The problem of information retrieval (IR) over P2P networks is being addressed by various research groups world-wide, that attempt to provide insight on the problem as well as solutions for its successful deployment. All studies that have been published so far on P2P IR solutions have been evaluated by simulation means, using well-known document collections such as ones acquired from TREC. In order to test their systems, researchers, divide a collection by distributing its documents to a number of simulated peers before starting to perform their evaluation. Two major issues arise by these practises: Firstly, there is little justification for the document distribution strategies followed by the relevant studies and secondly, since different studies use different experimental testbeds, there is no common ground for comparing the solutions proposed. In this work, we contribute a number of different document testbeds for evaluating P2P IR systems. Each of these has been deduced from TREC's WT10g collection and corresponds to different potential P2P IR application scenarios. We provide analysis of each methodology and testbed with respect to the document distributions achieved as well as to the location of relevant items within each setting. By this work we hope to provide more realistic evaluation environments for P2P IR systems as well as to create common ground for comparisons of existing or future architectures.

Keywords: Peer-to-Peer, Information Retrieval, P2P IR, Evaluation, Testbeds, Simulation, Content-based

PS/PS.GZ PDF Bibtex entry Endnote XML