Don’t blame reviewer #2, blame yourself!

• With the rapid increase of machine learning papers in recent years, we have been putting our reviewing systems under an unprecedented stress test

• Conference organizers have an ever-growing demand for qualified reviewers to match the myriad of submissions

• For example, many major conferences now expect authors of submitted papers to also be reviewers—even if that means that many of those reviewers might have have no or only little experience in reviewing

• Consequently, many people in the field have the impression that the quality of reviews has been gradually decreasing

• This narrative gets reinforced after every reviewing period when disgruntled authors let off steam on Twitter

• Justified or not, many authors rather blame the reviewers before they blame themselves

• Common causes for frustration are that the reviewers “did not get it” or that “what they ask for is literally in the paper”

there is not necessarily a strong correlation between reviewing experience and quality (see ICML? blog post),