Crowdsourcing for Research Paper Reviews (2)

Continued from the previous post

Extend the review process beyond conferences

In a crowdsourcing approach to paper reviews, we would need a proper platform in place. The key element of such a platform is a method to have open calls for paper reviews, such that reviews are extended to the lifetime of the community’s interest in the subject rather than a conference's annual call-for-papers. That is, interest in any submitted paper would peak during the annual conference period, and then it would quickly fade away. The review process is limited only to the submission period, with no other critique afterwards. This annual dance does not lend itself well to the longer term aspect of doing good research.

I would even posit that this is the same reason we are getting a flood of poorer quality papers because authors are becoming obsess with submitting papers. It is partly driven by the academic hiring system which favors the number of papers rather than the quality. Time and again, in my discussions with audiences in both academic and industry, this issue is inherently a top issue. However, no single person could easy change the system easily; especially one that is so entrenched.

Maintain an open review process

Secondly, we want the open community to read other reviews and offer their own technical insides. There is a need to moderate paper reviews for spam and other misuse of crowdsourcing. We need automated ways to remove reviews that are blatant abuses of the process such as self-review and unrelated commentary.

Reward the reviewers

Finally, we want to provide better incentives for reviewers. A recent business week article stated that the Web 2.0 community is working for praise. The paper reviewer community can similarly certainly benefit from this system. I would go further and suggest a point system and other reward mechanisms that easily entice good reviewers.

I speak from experience that getting good reviews is a difficult task. It is becoming a tedious task because it is simply difficult to attract volunteer reviewers. Time spent on searching for good reviewers can be better use to improve other aspects of conference or the publication itself. I believe that a change is in order to make the review process more integrated, and definitely, more rewarding ecosystem.

(fini)