Machine Assisted Search Preference Evaluation

Information Retrieval systems are traditionally evaluated using the relevance of web pages to individual queries. Other work on IR evaluation has focused on exploring the use of preference judgments over two search result lists. Unlike traditional query-document evaluation, collecting preference judgments over two search result-lists takes the context of documents, and hence takes the interaction between search results, into consideration. Moreover, preference judgments have been shown to produce more accurate results compared to absolute judgment. On the other hand result list preference judgments have very high annotation cost. In this work, we investigate how machine learned models can assist human judges in order to collect reliable result list preference judgments at large scale with lower judgment-cost. We build novel models that can predict user preference automatically. We investigate the effect of different features on the prediction quality. We focus on predicting preferences with high confidence and show that these models can be effectively used to assist human judges resulting in significant reduction in annotation cost.