The Paradox of Reviewer Accuracy

The presented paradox is based on the assumption that an obviously well-designed and well- conducted hypothetical study reveals experts being unable to accurately interpret the scientific value of research findings outside of their areas of expertise. In case of two or more experts reviewing research off the mainstream there is a certain chance of getting a positive assessment from one reviewer while getting a negative assessment from another reviewer. Amazingly, regardless of opposing assessments from different reviewers, both judgements may support the conclusion of the hypothetical study. A primarily negative assessment leads to rejection of the study but indirectly may confirm the accuracy of the conclusion e.g. by authority bias. A primarily positive assessment may directly confirm accuracy of the study conclusion e.g. by confirmation bias. At the same time, it questions the accuracy of the evaluation taking into account that the reviewer is inexperienced in this issue. This imagined paradox raises serious concern about the assumed objectivity of an independent and unbiased peer review of studies outside mainstream research.

Keywords: Empirical Science, Medical Philosophy, Peer Review, Probability Theory, Scientific Evidence