Battling Unfair Demons in Peer Review

  • Nihar Shah | Carnegie Mellon University

Peer review is the backbone of scholarly research. It is however faced with a number of challenges (or “demons”) such as subjectivity, bias/mis-calibration, noise, and strategic behavior, which cause unfairness in the process. This talk will present principled and practical approaches to battle these demons in peer review:

(1) Subjectivity: How to ensure that all papers are judged by the same yardstick?

(2) Mis-calibration: How to use ratings in presence of arbitrary or adversarial mis-calibration?

(3) Noise: How to assign reviewers to papers to simultaneously ensure fair and accurate evaluations in the presence of review noise?

(4) Strategic behavior: How to insulate peer review from strategic behavior of author-reviewers?

(5) Bias: How to rigorously test for existence of (gender/fame/race/…) biases in single-blind peer review?

The work uses tools from statistics and learning theory, social choice theory, information theory, game theory and decision theory. No prior knowledge on these topics will be assumed.

[Slides]

    • Portrait of Lin Xiao

      Lin Xiao

      Senior Principal Researcher

  • Research Area

Series: Microsoft Research Talks