Fei Liu

Fei Liu

PhD student in NLP at The University of Melbourne

Quit while ahead: Evaluating truncated rankings

Published:

Fei Liu, Alistair Moffat, Timothy Baldwin and Xiuzhen Zhang (2016) Quit while ahead: Evaluating truncated rankings. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy, pp. 953-956.

@inproceedings{Liu+:2016,
  author = {Liu, Fei and Moffat, Alistair and Baldwin, Timothy and Zhang, Xiuzhen},
  title = {Quit While Ahead: Evaluating Truncated Rankings},
  booktitle = {Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2016},
  address = {Pisa, Italy},
  pages = {953--956}
}

Abstract

Many types of search tasks are answered through the computation of a ranked list of suggested answers. We re-examine the usual assumption that answer lists should be as long as possible, and suggest that when the number of matching items is potentially small – perhaps even zero – it may be more helpful to “quit while ahead”, that is, to truncate the answer ranking earlier rather than later. To capture this effect, metrics are required which are attuned to the length of the ranking, and can handle cases in which there are no relevant documents. In this work we explore a generalized approach for representing truncated result sets, and propose modifications to a number of popular evaluation metrics.