Leiter Survey vs. US News

I thought I'd look a little at the differences between Brian Leiter's recent on-line survey of faculty quality and the U.S. News rankings.  First an observation about the changes in the top fifty schools.  There are five schools that appear in Leiter’s 2014 top 50 faculties that are not in the current U.S.News top 50 schools.  They are (and the numbers following the schools are U.S. News rank school and average rank):

University of California, Hastings 55.5 54
Cardozo Law School/Yeshiva Univ 65.5 64
University of San Diego 79.5 79
American University 75 72
Brooklyn Law School 84.5 83

There are also five schools in the current U.S.News top 50 schools that are not in Leiter’s 2014 top 50 faculties.  (And the number following each school is its U.S. News rank):

Wake Forest 33
BYU 37
SMU 42
Maryland 46
Tulane 48
Utah 49

I also looked at the correlations between Leiter's ranks, U.S. News data on peer assessment and overall rank, and Doyle's Washington and Lee Law Review rankings.  The following table reports the correlations between those variables.

	Rank Leiter	Peer Rank	USN Rank	Journals, 2006-2013
Rank Leiter	--	-.94	.84	-.82
Peer 2015	.94	--	-.86	.82
USN Rank avg	.84	-.86	--	-.67
Journals 06-13	-.82	.82	-.67	--

For all correlations, N = 49 and p < .0001

I guess one of the things that really stands out for me is the high correlation between Leiter's ranking and the U.S. News peer assessment scores (.94).  As long I as I'm talking about rankings, I want to say that last summer I posted a paper on ranking law schools using entering student data on LSATs, student employment data, and law review citations (which I use as a proxy for academic reputation).  It's available on ssrn.

Update: I've been surprised at the amount of commentary on this post -- I guess it shows that people care about rankings, or something.  Anyway, I think people might be interested in the differences between the Leiter rank and the U.S. News peer rank for individual.  Here's a table that lists the schools in order of their Leiter rank and compares them to their US News rank.   A negative difference indicates that the school does better on Leiter than U.S. News; a positive indicates that a school's U.S. News peer assessment rank is better than its Leiter rank.   I noticed that my own school (UNC) does substantially less well on the Leiter rank than on the U.S. News peer assessment rank.  Actually, I think UNC has the fourth-largest diffence between Leiter rank and U.S. News peer assessment rank (after Washington and Lee, which does much better on U.S. News than Leiter, the University of Illinois, which does much better on Leiter than U.S. News, and the University of Florida, which does much worse on Leiter than on U.S. News).  I guess I should have voted.  What's the slogan?  "If you don't vote you don't count," or some such.

At the risk of sounding too self-promotional, this paper uses citations as a replacement for U.S. News peer assessment, which -- as some people point out in the comments -- rely on raters to have some knowledge about a bunch of schools.

Comments

[anon]

Paul

As I said above, agreeing with Steve Freedman (at least in part):

"You're right. Many of the persons voted for their alma mater, or for other frivolous reasons.

The persons who "voted" don't know much of anything about the "scholarship" at all these schools. At most, they may have some real knowledge about a few persons writing in "their field," and beyond that, some scant, cursory knowledge of the work of one or two professors at a few of the schools. Nothing upon which anyone would claim an informed judgment. This is like grading papers based on the names of students whom you have never met, or worse (and this is par for the course in legal academia) students about whom one "heard" something."

Thereafter, profs chimed in to validate every one of these observations, in a rather embarrassing way.

So, to answer your questions:

"Anon, is your complaint about rankings[?]"

Yes. As stated above: "supposedly valid "rankings," at the top, [are based on] a long standing pecking order (that preceded USNWR and perpetuates itself by means of faculty hiring, endowments, longevity, etc.) and at the bottom on absolutely nothing at all."

"Anon ... is it about stating any opinion about the scholarly quality of a law school's faculty based on a scholarly interaction with the faculty of that law school"

No. Not any opinion. That seems to be a bit of an overstatement, Paul. I signaled Orin's anecdote about "great comments" as a perfectly representative bit of nonsense that is perpetuated as reliable "fact" in basing a judgment about the "strength of a faculty."

You object to "heat." I'll let that statement stand, rather than going thru these blogs and quoting some of your comments.

[The last Anon]

I'll also put the "heat" matter aside but to the extent Anon is bringing the heat, I feel like many are striking out. The idea that knowledge starts with a set of anecdotes is silly. That's not knowledge unless the knowledge is being able to repeat the anecdotes. One person may say, I got good comments at Georgia State and from that I infer it is better than Mercer. Another may say. I got better comments from Mercer than Charlotte. And then someone says I got better comments at Charlotte than Georgia state. Those anecdotes could all be true but that does that mean we know anything about the quality of law schools. Plus there is the voting for your school or the one from which you graduated issue. Wasn't there a ranking a couple years ago in which most FSU profs rated FSU above all other law schools? Leiter's ranking is simply a report of what people said when asked. Nothing more.

[Paul Horwitz]

Thank you for your response. Just to be clear, I have no objection to criticisms of various forms of ranking, or rankings altogether, for various reasons including the ones you offer here. In particular I agree with some of the reasons you offered, but even if I didn't agree with them, I think this is a perfectly legitimate subject of criticism. And although I do believe commentary should generally be civil, and anonymous commentary in particular should be civil, I don't object to heated replies as some kind of absolute matter, although sometimes I think a fairly innocuous comment ends up getting criticized in a surprisingly and unduly heated manner for various reasons (because the commenter is passionate about that issue in general, for instance).

In this instance, and I am happy to leave it at a polite disagreement with you, I thought Orin's comment was fairly innocuous and, whether you think it representative of bad answers faculty have given in other cases, can't be so described in this instance. In this particular instance, I didn't take Orin to be endorsing rankings in general, endorsing his experience with San Diego as a basis to fill out some actual ballot somewhere, offering advice on which of these two law schools to attend, or anything as significant as that. I took him to be answering a specific question that someone actually posed in seeming hope of receiving an answer: given a particular survey about the quality of faculty, and setting aside the larger questions whether that kind of thing is worth measuring or capable of measure, why might someone rank San Diego higher than Tulane? (Of course, if that's not what Steven Freedman meant, or if he personally read Orin's answer differently than I did or was offended by it, he's welcome to weigh in. He was the person the answer was intended for, after all.) I take it that Orin's answer, which was based on personal interaction with that faculty in the area--scholarly quality--that the questioner was asking about, wasn't meant to do more than offer one answer to that question. It may have been quick and dirty, but I didn't find it particularly overflowing with self-regard or assurances as to absolute reliability. As long as that's all it is, I don't find it terribly outrageous that someone offers such a question based on personal experience. I was thus surprised at what I thought was the vehemence and sarcasm ("me, me, me" and so on) of your answer to the extent it was directed at Orin. I tend to think it's nice, and a pleasant contrast to many online discussions, when people use comments to ask particular questions and others actually offer particular answers. If you felt otherwise about his answer, so be it, I guess.

[The Pentultimate Anon]

This is very interesting. Orin has treated the matter exactly as it should be treated. He had his say; Anon has had his say and neither needs a defense attorney because neither probably cares. But since Paul seems offended on Orin's behalf (or is something else afoot) let's get the facts straight:

Q: Seriously though, I think we all wonder how someone could possibly have a sense of the difference in scholarship quality among such a diverse group of schools. I'd be thrilled to hear someone articulate exactly which factors went into ranking San Diego over Tulane.

A:San Diego has a marvelous faculty with a large number of active an excellent scholars. I've presented papers there before, and I received some of the best comments I have ever received. Tulane is a good school with some fine scholars, to be sure, but I don't think it has nearly the quality of the faculty at San Diego.

So Steve's question is "what factors." Orin's response is a non sequiter. A factor cannot be that he got good comments from one school while at the same time having no responses from the school he ranks second. And then on Tulane; "I don't think is has nearly the quality of the faculty at San Diego." Unless 1)Orin has had comments from every school and 2) comments are the principal measure for ranking, it's not responsive to the question. That leaves us with he does not "think" it has nearly the quality. When Orin gets Lieter's ranking he puts San Diego over Tulane on the basis of comments and a hunch.

I do not what to put words in Anon's comments but I think his beef is that no one answered the original question and the purported answer is irrelevant and hunch.It's like the last Sienfeld episode -- the rankings and this discussion are about nothing but the deep involvement of those who are law professors, who in a bind play the civility card as Paul does, does tell us volumes.

[anon]

Needless to say, I agree with TPA.

And, to respond directly to Paul's comment: "I didn't take Orin to be endorsing his experience with San Diego as a basis to fill out some actual ballot somewhere ... or anything as significant as that."

I did.

And Paul, to bring a little "heat" here, you understood Orin's response the same way, as you say, "I took him to be answering a specific question that someone actually posed in seeming hope of receiving an answer." That question, as TPA points out, was: "I'd be thrilled to hear someone articulate exactly which factors went into ranking San Diego over Tulane." You are dressing up your comments quite a bit here, but the lipstick doesn't really cover the sort of risible nature of the Orin answer.

Is it significant that it was Orin? No. His comment, as repeatedly stated above, has been singled out because it was, as you say, "representative of bad answers faculty have given in other cases." My only disagreement here would be to substitute the word "risible" or "ridiculous" or perhaps even "self referentially offensive" ... Not civil?

I find uncivil arrogance and pomposity. Law profs have a bit too much of it, and too little humility. I believe this hubris is destroying the legal academy. There is little that can be done, but calling it out is a privilege so far afforded here, at least for now.

Of course, the response of any person whose words are blown up is "it was just a little, off hand remark." Think of how many times, however, you have rejected that excuse, society has rejected that excuse. Is Orin personally at issue here? Again, no. But let's not pretend that his comment wasn't noteworthy either.

[Orin Kerr]

Sorry I was away from the thread for a few days; kind of hilarious that my comment drew such a reaction. To be clear, I didn't base my view of San Diego just from presenting a paper there. I based it on following the scholarly work of the San Diego faculty for about 15 years, and also presenting a few papers there. San Diego has a lot of very productive top scholars, and I read their work often; I don't think it's nuts to have an opinion of the work I have personally read over a 15 year period supplemented by the experience of engaging with the faculty in a workshop a few times.

Anyway, sorry I didn't give all of the details of why I think SD has such a good faculty in my comment. It didn't occur to me that anyone cared, or was going to make judgments about the lack of details in the comment. Lesson learned, um, I guess.

[Orin Kerr]

Two more thoughts, in case anyone is still reading:

1) I assume it's obvious that when a person asks to compare two things and say which is better, they are only giving a personal opinion based on their own knowledge and experience. If you ask people to rank the best restaurants in town, or the top college football programs, or the best candidate running for a particular office, the answers they give will always be partial and quirky. I take it that's why such rankings usually ask lots of people and average out the answers or do some assessment of the collective view. So I don't think that expressing a view of whether one thing is better than another based on one's own partial and quirky experience is arrogant.

2) FWIW, I didn't complete the Leiter survey because I don't personally have views of the faculty at most of the schools. I happen to have knowledge of a few schools, and I think I can make broad comparisons between them. But I certainly don't know enough to rank all of them, so I didn't complete the Leiter survey.

[Paul Horwitz]

I must admit that I find incivility uncivil, and arrogance and pomposity arrogant and pompous. However I feel about either quality, I've never felt much of a need to confuse them. And, what with my trying to coexist with other people, who are human, I try to assume that some potentially objectionable remark was innocuous and offhand as often as is reasonably possible, which is an awful lot. Don't you? Otherwise I would go around getting offended all the time, for no especially good reason.

Penultimate, thank you for your comment. Nothing else was afoot other than being saddened and unsettled by what I saw as the commenter's high-dudgeon overreaction. I do appreciate your points and will think about whether I misread the context of the question and answer. I will add three things, though. 1) While I take your point about no one needing a defense attorney, that's not how I saw what I was doing. One might well conclude that I was butting in; if that's so, though, I wasn't alone in all this, since the anonymous person raising the fuss was not the person asking the question that Orin answered in the first place. If we're thinking in terms of people having their say, weren't those the only two relevant people? Wasn't everyone else beside Orin and Stevens just a buttinsky? 2) If you're right in your description, it seems to me a commenter could have written, "Orin, I don't think you answered the question." Wouldn't that have sufficed? 3) I have no problem with criticizing the rankings. But, as I wrote above, I didn't take Orin answering a commenter's question in an offhand fashion as an indication that he had filled out a ranking survey at all, let alone done so on that basis. I kept thinking I'd missed an earlier remark somewhere where he'd written, "Hell yes, I filled out the survey." But, as he confirms above, he didn't. He really was just answering, of failing to answer if you read it that way, an individual question on the comments site of a web site.

[anon]

Orin

You still don't seem to get it, so your attempt to minimize the way you responded to Steve's question is not convincing.

You stated, in response to a question by Steve asking for a valid list of factors that would justify a reason to rank USD over the faculty at Tulane, as follows:

" I've presented papers there before, and I received some of the best comments I have ever received."

Praise is owed for working the word "I" into such a short sentence three times. But, was the fact that YOU received “great comments” when YOU presented a paper there a great response to Steve's question?

Now, you are doubling down. We are supposed to believe that you have some way to compare the entire faculty at USD with the faculty at Tulane because YOU presented more than one paper there! Well, there you are, Orin. Case closed.

Really, Orin, you go on to assert that ranking top college football programs is "always partial and quirky." Again, you just don't seem to get it: ranking football teams is usually the result of, well, let's think a long time about that. Steve was looking for criteria. You were thinking of the fact that you received great comments at USD.

Now (while protesting that you don’t think anyone is reading comments, etc.) you've added: “I've read papers by faculty as USD." Yes, and so? Is that the basis for saying "Tulane is a good school with some fine scholars, to be sure, but I don't think it has nearly the quality of the faculty at San Diego."

What was the basis for that sort of, to use Paul's favorite phrase, uncivil and completely unsupported remark? You have offered nothing worth considering in response to Steve question: only a sort of arrogant assertion backed up by a slam on one faculty and bogus reasons for concluding another is "better quality" (what a joke this all is!).

As several comments above demonstrate, these surveys are nonsense, and aggregating opinions of the type you describe does not validate such surveys, it destroys any putative value associated with them. The fact that you believe that aggregating garbage is valid is so disappointing.

I think most reasonable people would agree, perhaps even you, that the value of the survey under discussion is basically zero. If you really want to pursue the arguments you are making here to support it, then I would say you are casting your lot with it, and the worth of your say so on these matters should be judged right along with the worth of that survey.

The move here, Orin, is not to pretend you just weren't thinking, but to reconsider your stance. YOu are correct, your comment standing alone doesn't mean too much to anyone.

But, your comments reflect an overall group think that should be examined and, hopefully, put in a context that shows surveys of this type to be too flawed to serve any purpose other than the one to which you put this one.

[Orin Kerr]

anon, one of us is seriously misreading the other, so let me approach this a different way with this question: If someone were to ask you what is the best restaurant in the town where you live, how would you answer, and based on what?

[Former Editor]

anon,

Is your objection to ranking at all or to the particular methodology used in this particular survey conducted by Leiter? If it's the former, I'll suggest that you leave those windmills alone. Some kind of ranking is going to exist whether we like it or not. If the latter, do you have a constructive suggestion? i.e. what criteria do you think would be relevant to a ranking of faculty and how would you propose measuring it? There are a lot of ways of "ranking" legal scholars/scholarship. Leiter's own website has used at least four I can think of off the top of my head, most if not all of which have some serious problems.

[Steven Freedman]

Whew. That's a lot of back and forth for my reasonable question and Orin Kerr's perfectly reasonable response.

Truth be told, I don't have any novel thoughts on this issue as I think most people are on the same page here. In a nutshell, I think most people agree that rankings are important to the extent that prospective students, employers, etc. often use them as a tool for comparing schools. Some of them are well constructed enough that they may even have validity to them. At the same time, I think most people, even those relying on the rankings, understand that many (most?) rankings include methods and data best compared to poppycock.

Don't get me wrong. I like Leiter's survey. It reminds me of those college bowl public radio fund drives where you root for your school to beat out your rival. It's the only fund drive I like. But the survey is silly. Other than making it a full-time effort, there is no way for a typical faculty member to keep track of the scholarship quality for 83 separate law schools. It's a popularity and anecdote contest, not a realistic evaluation of faculty quality. I'm not even sure Brian Leiter is suggesting otherwise.

[The Singing Fatman]

"Nothing else was afoot other than being saddened and unsettled by what I saw as the commenter's high-dudgeon overreaction."

Yes, I had to get counseling myself when I read Anon's comment. Fortunately, Orin, rightfully so, does not give a you know what and I think Steven has it right. The Leiter thing may be more of a reflection of those filling it out while knowing they know so little than what they might make of it after filling it out.

[anon]

Orin seems to think that ranking the strength of law school faculties is just like ranking taste in food.

Nuff said. Case closed. Just like Orin's mind.

Oh, and BTW, Orin, for a bit of humor here, watch the criteria used by Gordon Ramsey in his British show to rank the "best restaurants": service (using hidden cameras and tested by experts posing as difficult customers), ability to serve 30 people in two hours start to finish, a fly speck inspection of the kitchen for cleanliness and design, a competition to prepare a meal for the best known critics and consumers in each category, etc., etc., etc.

Funny, nowhere did Ramsey mention or use as a criterion whether anyone made "great comments" about his restaurants. Funny that ... as big an egomaniac as he is, even he knows how ridiculous that would have sounded as a criterion.

As for the "survey" I think there is a consensus that it is worthless. Orin, you apparently will defend it to the last, and others will defend you. Perhaps then, in the final analysis, you personally win the "FL survey" but, of course, lose on the merits. How fitting.

[The Singing Fatter Man]

Anon, you are making so many of us so very sad. Please stop.

[anon]

OK.

[Barry]

In addition, Orin, for prospective students, there are two criteria which matter:

1) Job prospects.
2) Cost of attendance.
(note, these can be modified for regional desires).

Everything else is bullsh*t for a student, whose life can be destroyed by wrong choices.

[Orin Kerr]

Barry, the Leiter survey is just a faculty ranking of the faculty at law schools. It does not ask nor answer how students should decide where to enroll if they are accepted.

[Barry]

OK.