Search the Lounge


« Bookstore Trivia | Main | Sisk Study of Scholarly Impact, 2012 »

August 08, 2012


Feed You can follow this conversation by subscribing to the comment feed for this post.


You are absolutely right, but the problem is intrinsic to the professoriat and to the legal community generally. The academy, the hiring committees at firms, the judges hiring clerks -- all of them give far too much intrinsic weight to "placement" -- where you went to school, who you clerked for, whether you were on law review, and so on. I have been a lawyer for 20 years and went to a good school and have fine credentials and all that should pale in comparison to the fact that I have done very well in my career, yet when I interview for a job, everyone cares too much about the credentials and not nearly enough about the actual accomplishments.

So absolutely, placement of an article shouldn't be viewed as a mark of its quality. Also, stop caring so darn much about pieces of paper acquired five, ten, twenty years ago.


Publish a Book of Lists. Best all around articles, best in [Insert Subject]. The amazing thing about 21st century technology is that it's a great equalizer. Publish it online, a blog, an online publication. And if it's done well, it could have a positive impact on the selection of articles.


The question is, given the obviousness of this solution to this long-complained of problem, why don't appointments, T/P committees, the broader academic community etc., just read the scholarship? The fact that nearly everyone is dissatisfied with placements as a measure of quality, and the fact that in theory we could all just read the articles and assess the quality for ourselves but we don't, suggests that something else is going on.

That something else probably has to do a tradeoff between the efficiency of evaluating someone's potential going forward and the depth of inquiry. In virtually every field in which Party A is required to make an assessment about how Party B, drawn from a pool of applicants for a job, a loan, etc., is likely to behave in the future, Party A falls back on proxies that are easily observable, thus increasing the efficiency of the screening process. I know of no one that seriously contends that the use of proxies doesn't introduce errors, and that seems unfair. But hiring committees, banks, etc., are not primarily in the business of trying to be fair. They are in the business of trying to assess a large number of applicants for a limited number of positions, loans, etc. They thus have to decide how to use scarce committee resources to assess the applicant pool. Using proxies such as placement or academic pedigree may be perfectly rational. Think of nepotism. The idea of getting a job because you are related to someone (or rather losing out on a job to someone else who gets the job because they are related to someone) is noxious from the point of view merit, but may actually be rational from the standpoint of the hiring person because the family connection conveys information about how the applicant is likely to behave in the future and may also mean that there are interpersonal relationships that will incentivize the applicant to work harder.

The fact of the matter is that even within our fields there is simply too much scholarship to keep up with in any sort of rigorous fashion. The job of appointments and T/P committees is even harder because they aren't assessing people who are necessarily in their field. So we hve to use proxies. Is law review placement a good one, in the sense of being highly correlated with good articles? I share your doubts about it, and Al's data seems to suggest there is something to this intuition. But that doesn't mean there is no information there, and it also doesn't mean that the solution of actually trying to directly assess the quality of all of the many articles published without the use of indicators is a feasible one.

Dan Joyner

Another variable in the equation is that the US is quite unique in the KIND of journals that we use for proxies of quality. By this I mean the use of student edited journals. I write in international law, and often publish in peer reviewed journals in the UK and Europe, where peer reviews are basically the only kinds of academic journals available. Wouldnt a switch to peer reviewed journals as the primary or exclusive type of journals used by US legal academics yield a better proxy for quality?

Now, before everyone jumps on this idea, I know the standard objections. Peer review doesnt necessarily equate to quality review. I get this - believe me. I've had some ridiculous peer reviews of my work.

Another standard objection is that peer reviewed journals require too much time and effort of us faculty, and that inevitably a switch to peer reviews would mean fewer journals overall. I do understand these concerns. So, like all other important things in life, we're not dealing with a clear and easy choice. But would the benefits of a switch to peer reviews outweigh the costs? I personally think it probably would. It would put us on par with all other academic disciplines, in which this calculus has indeed resulted in exclusively peer reviewed journals.

The comments to this entry are closed.


  • StatCounter
Blog powered by Typepad