Proceedings of the Second Mira Workshop

Edited by Mark Dunlop

University of Padua

Villa Duodo, Monselice, Italy

November 14-15, 1996

Preface

These proceedings are the result of the second workshop of the Mira working group. Mira is a European Union ESPRIT working group which is funded to hold regular meetings to work on evaluation of modern information retrieval systems (including very large scale systems, interactive systems and browsing based systems). The working group is guided by Keith van Rijsbergen and is composed of: The University of Glasgow (prime contractor), City University, Dortmund University, Dublin City University, GMD IPSI Institute , IEI-CNR Pisa, Joseph Fourier University, Nancy University, Padova University, Robert Gordon University, Danish Royal School of Librarianship, Swiss Federal Institute of Technology Zurich, Tampere University. and The Union Bank of Switzerland.

This workshop was held in the very pleasant surroundings of the Villa Duodo. A 16th century villa perched on the hillside above the village of Monselice near Padova (the villa is now in the hands of the University of Padova). This amazing location and social programme (or should that read, eating programme) were arranged by the local organisers: Fabio Crestani and Silvia Gabrielli.

After the first meeting of the working group, in which all participants presented their work and views on evaluation in IR, it was decided to give more time in subsequent meetings for discussion. Steve Draper proposed a format based on short presentations and long group discussions - this was adopted for the workshop (see Attendees and Workshop Format for more details). I implemented this format (even down to a whistle to control the timing) and these proceedings are composed mainly of the reports of the working groups (written up shortly after the workshop).

Together with work group reports and short reports of presenters the proceedings include papers giving an introduction to Thomas Green's Cognitive Dimensions, an overview of hypertext-IR evaluation by Jonathan Furner, an overview of work-place evaluation by Annelise Mark Pejtersen, an introduction to Evaluation-Light by David Harper and David Hendry and concludes with a short paper by Marion Crehange summarising the workshop.

To print these proceedings simply print each section in the contents list below - the proceedings are split into six long HTML pages to make printing easier (or you could print the single page version, which doesn't include page brakes and takes a long time to load but can be printed in one go). When printing pages some amount of reduction is helpful, I find two-to-a-page layout best.

Hypertext links are provided within each page: please make sure a page is fully loaded before using these links (these do not work within the single page version). The symbol will take you to the top of the current page when clicked, from there links lead to each report/presentation within that page and to the contents listing (via the workshop title).

The proceedings are also available as a technical report by writing to "Research Report Request, Computing Science, University of Glasgow, Glasgow G12 8QQ, Scotland". The proceedings may be distributed and printed freely so long as this preface is included - the proceedings must not be charged for (except in printed form by Glasgow University). The proceedings are copyrighted University of Glasgow. The full citation for these proceedings is given below. Further details on the Mira working group are available at http://www.dcs.gla.ac.uk/mira/.

The superb location, good food and wine, and the discussion-based format led to an invigorating meeting and hopefully this comes through in these proceedings.

Mark Dunlop

Contents

Please note that some titles have been shortened to aid formatting of printed version - if citing, please use full title from within the proceedings.

0 Background to Mira and workshop format;

1 Session 1: Combining HCI and IR Evaluation;

• 1.1 Presentation Summaries
• 1.1.1 IR and HCI - Micheline Beaulieau
• 1.1.2 Evaluation in HCI and IR - Steve Draper
• 1.1.3 The problems with precision and recall - Mark Dunlop
• 1.1.4 How many kinds of relevance in IR? - Stefano Mizzaro
• 1.2 An introduction to the Cognitive Dimensions Framework - Thomas Green
• 1.3 Working Group Reports
• 1.3.1 Task measurement and sub-task measurements - Steve Draper
• 1.3.2 User, task and domain - David Harper
• 1.3.3 Three issues for any evaluation - David Hendry
• 1.3.4 Issues in hypermedia system evaluation - Kalervo Jarvelin

2 Session 2: The TREC-5 Interactive Track Experience;

• 2.1 Interactive evaluation at TREC - Micheline Beaulieu
• 2.2 Working Group Reports
• 2.2.1 The positive and negative effects of TREC - Maristella Agosti
• 2.2.2 TREC: A way forward for Mira - David Harper
• 2.2.3 The dynamic nature of information seeking - David Hendry
• 2.2.4 MIRA: the other side of TREC? - Jane Reid
• 2.2.5 Small scale experiments proposed in TREC-6 - Eero Sormunen

3 Session 3: Evaluation of Hypertext/Hypermedia Information Retrieval;

• 3.1 Presentation Reports
• 3.1.1 The evaluation of hypermedia IR systems - Jonathan Furner
• 3.1.2 Experiences from the Irides project - Massimo Melucci
• 3.2 Work place evaluation of complex systems - Annelise Mark Pejtersen
• 3.3 Working group reports
• 3.3.1 WWW search engine evaluation - Nicholas Belkin
• 3.3.2 Benchmark testing hypermedia search engines - Pia Borlund
• 3.3.3 Building a hypermedia test collection - Marion Crehange
• 3.3.4 Testing automatically constructed hypertexts - Kai Grossjohann
• 3.3.5 WWW search engine evaluation (2) - David Hendry

4 Session 4: New directions in the evaluation methodology.

• 4.1 Evaluating IR Interfaces: A Case Study - G Brajnik, S Mizzaro, C Tasso
• 4.2 Evaluation Light - David Harper and David Hendry
• 4.3 Working group reports
• 4.3.1 Evaluation frameworks and methods - Giorgio Brajnik
• 4.3.2 A test-drive for Evaluation Light - Thomas Green
• 4.3.3 Scenario based relevance - Stefano Mizzaro
• 4.4 Short Paper About Models and Abstraction - Marion Crehange

Full Citation

Dunlop, M.D. (editor). Proceedings of the Second Mira Workshop (Monselice, Italy). University of Glasgow Computing Science Research Report TR-1997-2, http://www.dcs.gla.ac.uk/mira/workshops/padua_procs , November 1996.