Citations are not a Good Proxy for Article Quality

Most people seem to agree that article placement is not a good proxy for the quality of a law review article.  As reported in a recent article about law reviews, Barry Friedman conducted a study that illustrates why (h/t Prawfsblawg).  When Friedman asked law review editors “What factors play a role in your decision to make an offer?” 3 of the 5 most common factors strike me as highly problematic in terms of selecting articles based on merit:  “Author’s High Profile;”  “Faculty Recommendation;” and “Author is a Senior Scholar.”  Like most people, I am deeply skeptical that student reviewers have the time, resources, or expertise to adequately judge the only factors that ranked higher: “Quality of Argument” and “Novelty of Thesis.” 

In the article, Friedman often assumes that citations are a better proxy for article quality. I think many people share this assumption. As I explain after the fold, however, I am very skeptical about this being true.  As someone who has been on a very active appointments committee for the past two years, I wish that a better proxy existed.  However, I think that the only way to judge the quality of an article is to actually read it. 

Of course, defining “article quality” is incredibly difficult.  Broadly speaking, though, I think the editors are right to look for “quality of argument” and “novelty of thesis.”  Stated differently, I think an article is of low quality if: (1) it is merely descriptive of existing literature, i.e., it lacks a novel thesis; or (2) it is poorly researched, intellectually dishonest (misrepresenting the law or other side of the argument), or poorly reasoned, i.e., the quality of the argument is not strong. 

Articles that have these flaws might be cited quite frequently. Descriptive articles in particular are likely to be cited in the background section that is common in most law review articles.  The pressure that law review editors place on authors to cite virtually every sentence of the article may also reward long descriptive pieces. Articles that are poorly reasoned or intellectually dishonest might be frequently cited because they are controversial (like a viral blog post) or an easy target.

It seems very plausible to think that strong articles, however, could be overlooked entirely. Just as institutional bias affects law review editors’ choice of who to publish, an article that was placed in the Yale Law Journal is more likely to be cited.  No one reads everything that is published, and authors could use placement or author affiliation for triage. Moreover, if two articles are both on the same topic, most people will probably cite the article that was placed higher rather than using some independent judge of quality. This makes sense, because a higher placed article will be seen as more authoritative. An author may also give preference to a more recent article to show that their research is up-to-date, even when that means passing over an original article to cite something that was unoriginal when written.  In fact, citing to recent and well-placed articles might help convince student editors that you are writing on a timely and important topic. Citing primarily to older or lower-placed articles could communicate the opposite.

Citations may be a good proxy for influence.  If people are citing an article, that means people are reading it.  Influence, however, is not the same thing as quality.  FOX News is probably the most influential news organization in the country, but I don’t think of it as being particularly high in quality. 

My guess would be that citations are a factor of some combination of the following, plus some other things I can’t think of at the moment:

• Author/institution reputation
• Placement
• Popularity of topic (I have never cited to a tax article. I often cite to articles about constitutional interpretation.)
• Timeliness of article
• Quality of article

Notice that I did put article quality on the list.  Although I think article quality must be correlated to citation count, I would bet that it is a relatively weak correlation.

By the way, SSRN downloads are perhaps the worst proxy of all. Even setting aside the fact that download statistics can be manipulated or skewed for perfectly legitimate reasons (i.e., your students, family, friends, or Twitter followers downloading), an SSRN download indicates nothing more than an interesting abstract.

Comments

[Patrick S. O'Donnell]

In brief, the quest for a quantifiable feature (or combination thereof) as a proxy for article quality is a fool’s errand. Assessing “quality of argument” and “novelty of thesis” are, in the end, matters of judgment (and accounts for why defining ‘article quality’ is incredibly difficult) with both objective and (for better and worse) subjective dimensions . On many topics there are identifiable (so to speak) argument “spaces” and thus often the property of novelty is ascribed merely to an ability to find an “empty” space and “fill,” explore, or (even attempt to) “define” it. I tend to think that, to the extent creativity and insight are distinguishable from novelty, one should likewise rely on these as part of one’s criteria for assessing the quality of an article.

[Patrick S. O'Donnell]

[In part inspired (albeit indirectly or by free association) by this post, I decided to put together a short list of titles some (especially students or young scholars) might find helpful when it comes to evaluating arguments in various intellectual fora. It is found at both Religious Left Law and Ratio Juris (sans book images at the latter) today should anyone be interested.]

[Scott Dodson]

I think citations can be more useful if treated with more nuance. The type/form of citation matters. A citation in a string cite for a basic principle of law is practically worthless. But a citation in a paper that really engages and discusses the work is far more indicative of influence and, I think, quality. Perhaps we could measure citations in a different way other than a raw count. Here are two possible options: (1) only count papers citing to a specific work more than three times; (2) only count papers discussing the author or the author's work in the text above the line.

[Patrick S. O'Donnell]

I should have mentioned that I well appreciate the fact the legal arguments are in some respects distinct from arguments in other fields and there are of course a number of titles devoted to just that subject (by Alexander and Sherwin, Atria, Bix, Brewer, Burton, Dworkin, Golding, Kratochwil, Levi, Lipshaw, MacCormick, Patterson, Stone, Sunstein, Tamanaha, and Weinreb, among others). In my list, the titles by Haack and Toulmin, et al., have some relevant material (the latter with a chapter on legal reasoning).

[anon]

I agree with Scott: "A citation in a string cite for a basic principle of law is practically worthless."

If an "author" makes long lists of titles of works -- with no accompanying analysis, assessment of content and quality, or discussion specifically directed to each such work -- then a aura of superficiality and almost frivolity arises. This practice would appear to be a form of intellectual hucksterism: engaging in the pretense that making long lists of citations can be deemed useful or scholarly (especially in a context wherein it is likely that literally not one person will use these lists for any purpose).

[MLS]

One can question the value of citations but it is hard to see how an article that is never (or rarely) cited can be influential. I am sure some might be but it seems like it would be quite silly to argue that there are lots of high quality, innovative or influential articles that are never cited. Proxies are by their nature inaccurate but they are often not useless.

[Jeff]

Scott, I agree that substantive engagement with the article is much more significant than a mere reference. However, I would bet that many authors substantively engage with work that they do not think is particularly novel or well argued. An unoriginal article might be used as a example of a particular point of view, and a poorly reasoned/researched article might be taken to task. I would think it would be strange to simply refuse to cite a work that is on point because the author thinks it is a "low quality" article, especially if it was well-placed or from a well-known scholar.

MLS, I think influence and article quality are very different things. Citations are strong evidence of influence, but I think very poor evidence of quality. Of course, you might care more about influence than quality in making hiring or promotion decisions. That is really beyond the scope of this post.

[Jeff]

Patrick, I agree that there is no quantifiable way to measure article quality. However, I do think that we have a duty to make judgments on article quality in hiring and promotion decisions. In my view, that is the real peer review component of legal academia. We cannot fall into the trap of thinking that all scholarship is equal or that all articles that place well, are cited, or are downloaded are of high quality. Although judgments on quality are difficult and imprecise, they can only be informed decisions if you actually read the stuff.

[Enrique Guerra-Pujol]

Where is Deep State Special Legal Counsel (or the Captain) on this one?

[anon]

Let's not forget Sy Abelman and Athletic Second Amendment Supporter too (and who knows how many others)!

A lewd, offensive and entirely irrelevant comment about cars, guns, Illinois, Trump or low-end criminal defense, in response to the post about citations, is missing here!

[Deep State Special Legal Counsel]

^^^^^Thanks for inviting me to the party. Nice to see my cyber friends. I wasn't going to comment on this one cause I have no experience in this area. But if you insist...and since I am a lawyer and knows everything of course....

What I can relate is that if I cited to a law review article in a motion or in a request for a Bill of Particulars to an over whelmed trial court judge presiding over a traffic room of 250 violators who all want trial believing the coppers and government just want the money even if they were speeding, using a hand held device with no child safety seats would just roll her eyes and think of me as some kind of wack a doo, out of touch attorney. It ain't the law that counts. Got to work the room and know your judge.

Oh, by the way, low end criminal defense pays some of the bills. I am really waiting for a gig like Michael Cohen's---Maybe Sprint can throw some money at me. Can somebody give me a lead?

[[M][a][c][K]]

A couple of points.

First, at least in science there are ways to shall one say, manipulative the citation system. One professor I know had a vast number of citations, most of which were of either more accurate measurements of a value or parameter (more zeros after the decimal point) or confirmatory measurements. Anytime someone wrote an article where the value was required for a purpose - the measurement article would then get cited as proof or confirmation of the parameter/value. He stormed up those citations. Best of all, he could get his grad students to run the experiments.

Second, in some fields there is a bit of "one hand washes the other," i.e., "if I cite you, courtesy means that you cite me...." and on it rolls. I know of a few academics who got heavily cited this way, after the first few cites back and forth "got the ball rolling."

Third, I have cited and used legal academic articles - but primarily in international arbitration and in civil law jurisdictions. In those venues they can be taken pretty seriously - I remember using one on the obligation of an Arbitral Tribunal to reach its own conclusions and not simply adopt those in the report of its own appointed expert (which was both incompetent, chauvinist and frankly nuts, and fell apart on cross.) So, while I would suggest that, certainly outside US appellate courts, legal articles tend not to influence US trial judges much, they can have a pretty strong impact in other fora. One I think is particularly important, though it is not an article is the 2012 UNCITRAL Digest on the Model Law of Arbitration. There is also a Digest I think on the CISG that may be useful.