« Workshop: Networks Across Disciplines in Theory and Applications | Main | Nathan Explains Science, welcome to the blogosphere! »
November 04, 2010
Peer review and the meat grinder
ArsTechnica's Chris Lee has a nice (and brief) meditation on peer review, and it's suitability for vetting both research papers and grant proposals. The title gives away a lot "A trip through the peer review sausage grinder", but that should be no surprise to anyone who lives with the peer-review process. The punch line Lee comes to is that peer review works okay at vetting the results of scientific research but fails at vetting potential research, that is, grant proposals. This conclusion seems entirely reasonable to me. [1]
Given some interactions with NSF Program Managers and related folks over the past year, peer review is the one thing that is not up for discussion at NSF. [2] They're happy to hear broad-based appeals for more funding, for suggestions about different types of funding, etc. But they are adamantly attached to sending grant proposals out for review by other scientists and taking the advise they get back seriously. To be honest, I'm not sure how else they could do it. As Lee points out, there are many more scientists now than there is funding, and the fundamental question is how do we allocate money to the projects and people most likely to produce interesting and useful results? This kind of pre-judgement of ultimate quality is fundamentally hard; peer-review frequently fails at doing this for research that is already finished (peer review at journals), and is even worse for research that has not yet been done (peer review for grant proposals). Track-record-based systems are biased against young scholars; shrinking the size of the applicant pool smacks of elitism; and peer-review effectively produces boring, incremental research. Lee suggests a lottery-based system, which is an interesting idea, but it would never fly.
-----
[1] A part of me is proud of the fact that one of my proposals at NSF was criticized both for being too ambitious and for being too incremental.
[2] Here and here are NSF's explanations for why peer review is good and how it works.
posted November 4, 2010 07:55 AM in Simply Academic | permalink