Law Review vs. Peer Review

I have a new essay up on the University of Illinois Online titled "Law Review vs. Peer Review: A Qualified Defense of Student Editors." It begins with a description of the familiar complaints about law reviews, and then discusses the comparable problems in peer reviewed journals.

Here are a few descriptive paragraphs:

I have learned from my recent forays into the social and natural sciences, however, that peer review is attended by deep problems of its own. Most seriously, peer-reviewed journals typically—perhaps even universally—require exclusive submission. An author may send a manuscript to only one journal at a time, and must then wait for a decision before, if rejected, submitting it elsewhere.  That would not be terribly burdensome if academic journals provided relatively quick decisions, but the reality is just the opposite. Turnaround time for peer-reviewed journals is measured in many months and sometimes even years, especially if the initial response is a “revise and resubmit,” meaning that the author must rework the paper with no guarantee of acceptance.

In one extreme, but ultimately unsurprising, example, the sociologist Philip Cohen described what he called a “saga.” The process took 583 days between the initial submission of his article and its ultimate acceptance (it was 776 days until actual publication). Along the way, his article was considered by four journals, subjected to thirteen reviews, and revised multiple times, although the end product, according to Cohen, did not differ significantly from the initial manuscript.

***

The problems with peer review are not unique to psychology, sociology, or the other social sciences. They bedevil medicine and natural science as well. John Ioannidis, of the Stanford Medical School, has opined that “most published research findings are false,” and Richard Horton, editor-in-chief of The Lancet—which is the U.K.’s leading medical journal—has cautioned that “much of the scientific literature, perhaps half, may simply be untrue.”

Ultimately, I point out that peer review is no guarantee of reliability, using the PACE trial and On the Run as examples of obviously flawed research that was not detected by expert reviewers.

The piece concludes with a thought experiment about a situation in which law review editors might actually do a better substantive job than peer reviewers by catching errors that experts have missed.  You can read the essay here.

Comments

[Schraubd]

Good piece. I've also written in qualified defense of law reviews versus peer review before (http://dsadevil.blogspot.com/2015/06/in-defense-of-law-reviews.html), and now having a bit more experience with the peer review world I can't say my view has changed. There are advantages to peer review, but a host of disadvantages as well (one you allude to but which has been particularly frustrating to me is the Frankenstein's Monster effect of having to incorporate multiple review comments into a revised draft. Even when they don't flatly contradict one another, they can convert an integrated essay into a stitched-together patchwork very quickly).

[Al Brophy]

Good work, as always, Steve. One note here -- a lot of peer reviews are fast because editors will reject a submission without sending it out for peer review. In my experience, editors typically only send out a piece for peer review if they think there's a decent likelihood that the piece might end up in the journal.

[Steve Lubet]

Thanks, Al. And yes, you're right, I should have said something about desk rejections (although I've read even they can sometimes take three to six months).

[Steve Lubet]

Thanks also, David. I have read many complaints about responses to R & R reviewers, which can apparently be maddening. Since the article has not yet been accepted, the author is at considerable peril if she disagrees with a reviewer -- so there is great incentive to make the changes for the sake of getting the article published. The consequence of a rejection can be a year of delay.

Also, most social science journals have relatively strict word limits. That should be a real plus for readers -- no bloated 30,000 word articles to wade through -- but it presents a problem when revising an article that is already close to the limit. How to address the reviewers' requirements without adding too many words.

Only about half of pieces are ultimately accepted after an R & R. If rejected, the author has to decide whether to retain the changes (if they were beneficial) or revert to the original piece when submitting to the next journal.

[Ralph Clifford]

"Desk rejections," or as I call them in this context "return address rejections" also occur in non-peer reviewed law journals. If you are a scholar at the wrong school, an article submission can be rejected immediately upon receipt.

[Jeremy Telman]

My defense of student-edited law reviews is less qualified. http://lawprofessors.typepad.com/contractsprof_blog/2013/10/in-defense-of-student-edited-journals.html

[Daniel S. Goldberg]

Having experienced plenty of both systems, my view is fundamentally Churchillian: Peer reviewed systems of scholarship are the worst ones imaginable save all the others ones that have ever been tried.

There are a very large number of problems with peer review. None of these problems -- some of which are very serious -- render it inferior to a system in which second-year law students with little to no content expertise and absolutely no training on how to actually produce scholarship judge the merits of scholarship in a totally unblinded review.

(There are advantages to law reviews and student editing, of course -- I just think they do not nearly overcome the enormous disadvantages).

[[M][a][c][K]]

One of the suggested downsides of 'peer review' in a publish or perish system, it has been alleged to me by people generally knowledgeable about the practices is the way in some fields there is a 'one-hand-washes-the-other' or 'mutual-backscratching' trend in citation and acceptance.

Another is simply the efflorescence at huge cost of journals. An article worth reading on this is:

https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science

Though one must point out that law reviews, which are not peer reviewed are hardly innocent in this regard.

[Jeremy Telman]

@Daniel Goldberg: happily, we, as law professors, do not face Churchill's choice. We can publish in either peer review or student-edited journals (some of which entail something like peer review) and thus live in the best of all possible worlds. And based on my experience with peer review and student-edited journals, I do not think one or the other is clearly superior for all purposes. The review may be more exacting in peer review, but (outside of the hard sciences) the editing is generally far more exacting in student-edited journals. Student editors come cheap and thus can spend a lot more time verifying claims.

[Daniel S. Goldberg]

@Jeremy Telman:

I don't really disagree, but your points about the relative strengths of student-edited law reviews are not responsive to my points about its deficiencies. As I noted above, law reviews do indeed have strengths. In my mind and in my experience, those strengths do not nearly overcome the significant weaknesses.

Others disagree, of course. *shrug*