Search the Lounge


« In Defense of Florida Coastal | Main | Carole King »

May 11, 2018


Feed You can follow this conversation by subscribing to the comment feed for this post.

Patrick S. O'Donnell

In brief, the quest for a quantifiable feature (or combination thereof) as a proxy for article quality is a fool’s errand. Assessing “quality of argument” and “novelty of thesis” are, in the end, matters of judgment (and accounts for why defining ‘article quality’ is incredibly difficult) with both objective and (for better and worse) subjective dimensions . On many topics there are identifiable (so to speak) argument “spaces” and thus often the property of novelty is ascribed merely to an ability to find an “empty” space and “fill,” explore, or (even attempt to) “define” it. I tend to think that, to the extent creativity and insight are distinguishable from novelty, one should likewise rely on these as part of one’s criteria for assessing the quality of an article.

Patrick S. O'Donnell

[In part inspired (albeit indirectly or by free association) by this post, I decided to put together a short list of titles some (especially students or young scholars) might find helpful when it comes to evaluating arguments in various intellectual fora. It is found at both Religious Left Law and Ratio Juris (sans book images at the latter) today should anyone be interested.]

Scott Dodson

I think citations can be more useful if treated with more nuance. The type/form of citation matters. A citation in a string cite for a basic principle of law is practically worthless. But a citation in a paper that really engages and discusses the work is far more indicative of influence and, I think, quality. Perhaps we could measure citations in a different way other than a raw count. Here are two possible options: (1) only count papers citing to a specific work more than three times; (2) only count papers discussing the author or the author's work in the text above the line.

Patrick S. O'Donnell

I should have mentioned that I well appreciate the fact the legal arguments are in some respects distinct from arguments in other fields and there are of course a number of titles devoted to just that subject (by Alexander and Sherwin, Atria, Bix, Brewer, Burton, Dworkin, Golding, Kratochwil, Levi, Lipshaw, MacCormick, Patterson, Stone, Sunstein, Tamanaha, and Weinreb, among others). In my list, the titles by Haack and Toulmin, et al., have some relevant material (the latter with a chapter on legal reasoning).


I agree with Scott: "A citation in a string cite for a basic principle of law is practically worthless."

If an "author" makes long lists of titles of works -- with no accompanying analysis, assessment of content and quality, or discussion specifically directed to each such work -- then a aura of superficiality and almost frivolity arises. This practice would appear to be a form of intellectual hucksterism: engaging in the pretense that making long lists of citations can be deemed useful or scholarly (especially in a context wherein it is likely that literally not one person will use these lists for any purpose).


One can question the value of citations but it is hard to see how an article that is never (or rarely) cited can be influential. I am sure some might be but it seems like it would be quite silly to argue that there are lots of high quality, innovative or influential articles that are never cited. Proxies are by their nature inaccurate but they are often not useless.


Scott, I agree that substantive engagement with the article is much more significant than a mere reference. However, I would bet that many authors substantively engage with work that they do not think is particularly novel or well argued. An unoriginal article might be used as a example of a particular point of view, and a poorly reasoned/researched article might be taken to task. I would think it would be strange to simply refuse to cite a work that is on point because the author thinks it is a "low quality" article, especially if it was well-placed or from a well-known scholar.

MLS, I think influence and article quality are very different things. Citations are strong evidence of influence, but I think very poor evidence of quality. Of course, you might care more about influence than quality in making hiring or promotion decisions. That is really beyond the scope of this post.


Patrick, I agree that there is no quantifiable way to measure article quality. However, I do think that we have a duty to make judgments on article quality in hiring and promotion decisions. In my view, that is the real peer review component of legal academia. We cannot fall into the trap of thinking that all scholarship is equal or that all articles that place well, are cited, or are downloaded are of high quality. Although judgments on quality are difficult and imprecise, they can only be informed decisions if you actually read the stuff.

Enrique Guerra-Pujol

Where is Deep State Special Legal Counsel (or the Captain) on this one?


Let's not forget Sy Abelman and Athletic Second Amendment Supporter too (and who knows how many others)!

A lewd, offensive and entirely irrelevant comment about cars, guns, Illinois, Trump or low-end criminal defense, in response to the post about citations, is missing here!

Deep State Special Legal Counsel

^^^^^Thanks for inviting me to the party. Nice to see my cyber friends. I wasn't going to comment on this one cause I have no experience in this area. But if you insist...and since I am a lawyer and knows everything of course....

What I can relate is that if I cited to a law review article in a motion or in a request for a Bill of Particulars to an over whelmed trial court judge presiding over a traffic room of 250 violators who all want trial believing the coppers and government just want the money even if they were speeding, using a hand held device with no child safety seats would just roll her eyes and think of me as some kind of wack a doo, out of touch attorney. It ain't the law that counts. Got to work the room and know your judge.

Oh, by the way, low end criminal defense pays some of the bills. I am really waiting for a gig like Michael Cohen's---Maybe Sprint can throw some money at me. Can somebody give me a lead?


A couple of points.

First, at least in science there are ways to shall one say, manipulative the citation system. One professor I know had a vast number of citations, most of which were of either more accurate measurements of a value or parameter (more zeros after the decimal point) or confirmatory measurements. Anytime someone wrote an article where the value was required for a purpose - the measurement article would then get cited as proof or confirmation of the parameter/value. He stormed up those citations. Best of all, he could get his grad students to run the experiments.

Second, in some fields there is a bit of "one hand washes the other," i.e., "if I cite you, courtesy means that you cite me...." and on it rolls. I know of a few academics who got heavily cited this way, after the first few cites back and forth "got the ball rolling."

Third, I have cited and used legal academic articles - but primarily in international arbitration and in civil law jurisdictions. In those venues they can be taken pretty seriously - I remember using one on the obligation of an Arbitral Tribunal to reach its own conclusions and not simply adopt those in the report of its own appointed expert (which was both incompetent, chauvinist and frankly nuts, and fell apart on cross.) So, while I would suggest that, certainly outside US appellate courts, legal articles tend not to influence US trial judges much, they can have a pretty strong impact in other fora. One I think is particularly important, though it is not an article is the 2012 UNCITRAL Digest on the Model Law of Arbitration. There is also a Digest I think on the CISG that may be useful.

The comments to this entry are closed.


  • StatCounter
Blog powered by Typepad