LSAT Scores and Eventual Bar Passage Rates

There's a lot to say about professional responsibility, law school admissions, LSAT scores, and risk of failing the bar exam. Before I return to those broader issues, I want to address a factual question that has emerged in these discussions: Are LSAT scores related to eventual bar-passage rates or just to first-time rates? Most people who raise this question acknowledge that there are serious costs to retaking the bar exam--so first-time rates matter. Still, it is valid to ask whether LSAT scores correlate with eventual bar passage: eventual passage rates matter too.

The first, and most important, response to this question is that each law school knows (or can readily determine) the answer for its own graduates. Bar examiners used to hoard bar passage information in some states, but most now share it readily. My law school knows which of its graduates pass the bar, not only the first time but in subsequent attempts. We also, of course, have a wealth of other data about those students. It's not hard to create both simple correlations and more complex regression analyses identifying factors that predict eventual bar passage or failure.

As an aside, I did this during the late 1990s for my own law school, partly to inform a faculty debate over whether students who took upper-level commercial law courses (e.g., Commercial Paper, Secured Transactions, or Advanced Contracts) were more likely to pass the bar than their classmates. They weren't--although students who took the basic Trusts and Estates course were. LSAT, UGPA, and law school GPA were also significant predictors. [If you're a Trusts and Estates professor, don't rush to your faculty with this news. The most likely explanation for that association is that students who took Trusts and Estates were more conscientious than other students about taking a full slate of bar courses. It wasn't T&E itself that prepped them for the bar; it was their attitude toward course selection.]

In any event, if I could do this in the late 1990s as a solo project without any staff help (or even a blog to post the results on!), any law school can do it today. To act responsibly toward their students, law schools should make these analyses. What variables correlate with eventual bar passage at your school?

National Research

Let's say, though, that your school can't obtain the data for these analyses. Or maybe the credentials of your current students are quite different from those who have already graduated and taken the bar. Is there any existing research that suggests a correlation between LSAT scores and eventual bar passage? 

Sure. The LST report includes some of the relevant research; Robert Anderson and David Frakt cite other sources. The table below summarizes one more dataset that I find illuminating. The first and third columns in the table come from LSAC's landmark study on bar passage. In the course of that study, LSAC reported both the mean LSAT score and eventual bar passage rate for five different racial/ethnic groups. (The study included a total of seven racial/ethnic groups but two of those groups were too small for reliable analyses.)

The table helps visualize the association between LSAT scores and eventual bar passage rates. (The middle column offers a rough translation between the LSAT scores used before 1992 and those used today.)

Eventual Pass Rates: 1991 Matriculants

Mean LSAT Score (Old)	Mean LSAT Score (New) (Est.)	Percentage Passing Bar
29.39	146	77.63
33.05	152	88.44
33.85	154	89.04
36.07	157	91.88
37.50	160	96.68

Three points/caveats about this table. First, the table rather clearly shows that higher LSAT scores predict higher eventual bar passage rates--while lower scores are associated with more risk of failure.

Second, the LSAT scores in each row of the table represent means drawn from a large number of students. Within each group, the overall pattern probably repeats. We cannot, in other words, read the first row in the table to indicate that a student with an LSAT of 146 has a 77.63% chance of passing the bar. The students represented by that mean included some with higher LSATs and some with lower ones. The ones with higher LSATs probably enjoyed (on average) greater success on the bar while those with lower scores enjoyed less success--even after multiple attempts to pass.

The means, however, still offer some sobering insights. The students in the LSAC study with an LSAT score of 39-40 (about 146 on today's scale) were unlikely to enjoy a higher probability of passing the bar than the 77.63% identified with that mean. Similarly, students with LSAT scores below 39 (146) almost certainly faced a lower probability than 77.63%--again, even after repeated attempts to pass the bar.

Finally, insights from this table are somewhat limited by the fact that the groups in each row represent different racial/ethnic groups. To the extent that those groups vary in their law school performance (after controlling for LSAT), persistence in taking the bar, or other factors, the bar-passage probabilities reflect some of those differences. The table, in other words, does not reflect 5 groups randomly drawn from the population. I will return to this issue in an upcoming discussion of race/ethnicity, law school admission, and bar passage.

Associations and Causation

The data offered above, along with data already publicized, support an association between LSAT scores and bar passage--whether one examines first-time bar passage or eventual passage. Association, of course, is not causation. The relationship suggests that LSAT scores can predict an applicant's eventual success in passing the bar. The score doesn't cause bar passage or failure, any more than an LSAT score causes a student to get particular law school grades. Predictors, however, can offer useful insights when making decisions.

Predictors, on the other hand, are imperfect. Some students perform better in law school than their LSAT score predicts; others perform worse. Sometimes we can predict those variations because of other information about the student; e.g., a student with a higher UGPA likely will perform better with one who has the same LSAT score but a lower UGPA. Other times, the performance surprises us.

The same is true of bar passage predictions based on LSAT scores. LSAT scores are an imperfect predictor, but they offer information that a responsible decision maker would consider; in fact, we already do consider that information in law school admissions. Next up, I'll consider the appropriate way to use LSAT scores to consider whether we--or other law schools--are fulfilling our obligation to put student interests above self-interest.

Comments

[twbb]

Tammy:

"I am new to this. If a student wants to invest 100,000 in going to law school and has a 70% chance of passing the bar, what is the ethical principle that requires them to be told no?"

In addition to all of the other ethical issues described above, another problem is they're not investing THEIR 100,000; they're investing the taxpayers' 100,000.

[David Frakt]

Just to be clear, the admissions debate is not about students who have a 70% chance of passing the bar, it is about schools admitting students with a 20-30% chance of completing a J.D. and passing the bar the first time, and a less than 50% chance of ever passing the bar. These are realistic numbers for students in the 130s and low 140s on their LSATs with correspondingly low grades. For confirmation, look at the bar pass statistics for non-ABA accredited law schools, back when students with these kind of LSATs and grades couldn't get into ABA accredited schools.

[antiro]

"If a student wants to invest 100,000 in going to law school and has a 70% chance of passing the bar, what is the ethical principle that requires them to be told no? If there is one, why does it not apply to going into debt to buy a car or house that may appear risky to others. I mean it's just a product and, as long as there is no fraud or misrepresentation, what is the beef?"

Let's analogize this to an attorney-client representation. Say you are hired to defend someone for a DUI. They were drunk, refused FST's and the breathalyzer, but the dashcam shows some swerving on a straightaway, there is the odor of alcohol and bloodshot eyes, and the client handed over his library card when asked for license and insurance. Further more, your client says some nonsense followed by slurred words: "I wan' see mah lawyer, I not drunk, *hiccup*." The officer is unimpeachable, and the judge has not ruled favorably in similar cases.

The client really doesn't want to get his 2nd DUI (he had his first plead down to reckless driving, and his second he blew a .015 at the station).

What do you tell your client about his chances at trial? Do you tell him he can spend an extra 2, 3, or 4 grand to do a bench trial with the adequate preparation, or do you tell him the best you can do is a straight plea with a high fine?

Compare this common scenario to that of a marginal law student who sort of knows the ropes with the seasoned law school admissions team who has been in the business for decades who can predict with a reasonable degree of accuracy which type of student will be successful and what type of student will be unsuccessful.

This post is getting a bit long, but the point is, if you are the attorney for the client dead set on the DUI trial where you are sure to lose, you strenuously advise your client they are risking a worse sentence before taking the extra few thousand and preparing for trial. I argue that the law school admissions should do something similar.

[salesman]

Antiro, the analogy is of limited value. Students are not clients. Admissions teams are not fiduciaries.

A more apt analogy is probably used car salesman.

[[M][@][c][K]]

Salesman:

"A more apt analogy is probably used car salesman."

No, a buyer from a used car salesman is protected by "truth in lending laws," state and federal, lemon laws, etc. that do not apply to education and specifically exclude Federally backed student loans from their remit; oh and the buyer can go bankrupt or hand the car back.

[Jojo]

Tammy,

In the very recent past, some institutions of higher legal learning used to lie about employment outcomes to get the marks in the door. While all but one of the lawsuits from such practices have failed, at least one court found that at least one school made public misrepresentations about employment outcomes.

I disagree with the caveat emptor defense in the educational context. These are not sophisticated consumers and there is informational assymmetry. State governments give law schools special exclusive status as the vehicle to enter the legal profession.

Also you got the dollars wrong, Tammy. It's more like $150,000 or $180,000. If the student goes on IBR, then the tax paying public takes the risk. If the student goes on government benefits because of the squandered human capital, that's on the public again.

But if you find some dim, independently wealthy mark who always wanted to go to law school, and you tell him the odds, he's fair game. Sell away.

[Deborah Merritt]

Antiro's analogy is excellent analogy--it underscores the point that law professors are professionals who are expected to act differently from sales people.

Here, we can take Antiro's analogy one step further. Suppose that through 2013, the DUI lawyer had a "success" rate of 65% for her clients. In this context, "success" would mean keeping the client out of jail; the client either was acquitted or paid only a fine. But then the lawyer started accepting a different type of client. To pay her bills, she started taking clients who had refused the breathalyzer. Given both the law and the attitude of local judges, the lawyer knew that it would be much harder to keep these clients out of jail.

Could the lawyer give these clients the same advice (about trial versus plea) that she gave to previous clients? Of course not; these clients are different. Could the client post on her website success rates for previous years, without disclosing how changes in the underlying facts would affect those success rates? Not unless she's eager to see the lawyer disciplinary system firsthand.

These are the type of activities that some law schools practice--and that we collectively allow. Indeed, we facilitate some of this behavior through the way we structure reporting of bar passage rates.

Why is it so hard to get academics, who love to offer their opinions and knowledge on many subjects, to offer better information to prospective law students? And to counsel those students in a more individualized way, when necessary, about their prospects for success?

[Jeb]

Deborah - Sometimes it is expected to be worse in the future, sometimes better. Once you get beyond objective results data should you counsel students that the data may look worse than they can expect? People will ultimately look at employment (and bar passage) figures with a 2 year lag when applying. I would think, with respect to employment, the numbers they are looking at are probably a bit worse than they could expect if they apply now. Should there be a similar caveat with the employment numbers to say "we expect our numbers to be better in the future because there are fewer grads and a slightly better employment market and (if applicable) we are doing a better job placing people"?

[Jeff Harrison]

These numbers perhaps could be viewed in a slightly different way. They remind me of going to a doctor who says, your chance of a heart attacks is 5% higher if you eat eggs. Then you ask what is your chance of a heart attack and she says 5%. So my chance is 5% higher than the norm of 5%. Yes she says, your chance is 5.025% So if bar passage is 96% for the highest level and 77% for the lowest then I think, depending on what you use as a base, the low score person is 19% less likely to pass the bar or 81% as likely to pass as the high score person. It is worse if you use 77% as the base. This is pretty significant but also, to me at least, means LSAT scores must be weighed carefully with other factors.

As a prior commentator noted, the numbers are in a sense misleading. For example, if your doctor says you have a 50% chance of survival, it's really not relevant. You will or you will not. There are no 50% survivors just as there are no 80% bar passers. To me this means, again, that the LSAT tells applicants and admissions officials very little.

[Tammy]

I appreciate the responses to my question. There seem to be two principles at work. One is that law professors have a duty not to just be another salesperson. The other is that there is an absence of disclosure by law schools. Also, as I understand it, (although perhaps not a principle) taxpayers pay for the loans that are defaulted on but, by then, law professors have been paid. I must admit, I find the last one most troubling since the first two could perhaps be solved by posting bar passage rates, employment rates, and salaries by LSAT/GPA. is that impossible or just resisted?

[twbb]

Jeff, while the LSAT might be an ambiguous predictor at certain boundaries it is not a completely abstract test unconnected from professional study. It tests for basic comprehension of written English and some level of logical thinking, and I don't see how anyone can legitimately argue that ability to meet that test on the LSAT is unrelated to law school, the bar exam, or practice.

As a law professor what do you think you can teach someone who scores at the 5th percentile on the test? Can someone who just can't decipher the meaning of an easy written passage on the LSAT to the extent that they can answer a multiple choice question, then be taught in three years to master far more complicated language and make fine distinctions between factual and legal situations? At some level isn't someone just not prepared for the study of law?

[Jeff Harrison]

twbb, actually I am not a LSAT detractor although it may seem that way and not an expert. It's just that in the ranges in the original post I was surprised that it was not more indicative of bar passage. I most definitely do not want to teach anyone who is in the 5th percentile although sometimes I think I am doing just that. My thought is that as schools go to lower scores -- low 150/high 140 -- perhaps other variables are more important. As a side note, my school took an LSAT tumble this year and I teach contracts to ILs. Just finished grading and found no papers that I would say were written by people who did not deserve a opportunity. On the other hand, and maybe I am imagining this, to me the lower part of the class was lower than usual.

[anon]

Jeff,

It looks like your school fell one (1) LSAT point at the median (from 158 to 157) and 1 point at the 75th percentile (from 161 to 160). The 25th percentile looks to be unchanged.

That translates to a drop at your median from the 73rd percentile last year to the 70th percentile this year, and from the 82nd percentile to the 80th percentile at your 75th. That alone hardly seems significant enough to be noticeable in the classroom or to really be described as an kind of real "tumble."

Now, there is, of course, a lot of variance within the 75th and the median, so perhaps the bands extend further this year than last year. That is, it's hypothetically possible that the median student this year could have had a 157, but then the next lowest score was a 150 or something, with everyone else below that. But that seems unlikely, or at least the degree of variation within the bands at any given school seems unlikely to vary that much year-over-year.

My point is that while it's obviously better to go up than down, many people (both faculty and outside observers) may tend to overestimate how much movement (up or down) a single LSAT point actually represents in reality, at least in these middle/medium-high ranges of the LSAT distribution.

[Jeff Harrison]

Thanks anon. Thanks for the clarificaion. There was near panic at the law school about the fall in the LSAT in terms of the rankings. As I recall now, it was more concern that we had fallen by maintaining enrollment levels while other schools, including our chief in in-state rival, had decreased admission in order to maintain LSAT scores and national ranking. Nevertheless I am sure we will go out an offer scholarships to people who do not need them in order to recover proving, I guess, that almost anything can be bought. I actually don't care except to the extent it affects student employment.

[Former Editor]

I think that it would be easier to believe that law schools in the "extreme risk" 25th percentile band are living up to their obligation to protect the integrity of the profession if admissions requirements other than UGPA and LSAT at those schools were also visibly being adjusted. If a school is assessing the chances of students with very different metrics than the students it has assessed in the past, it's hard to see how it wouldn't need additional or different information from the new group of students in order to do so competently.

For example, the ability to engage in analytical writing is a critical skill for success in law school and the profession. So, it would make sense to me for a school to admit a low LSAT/UGPA student if the student had otherwise demonstrated the ability to engage in above average analytical writing.

Figuring out if an applicant has that skill should be easy enough, too. Most law schools don't ask for a serious writing sample, such as a college term paper, but nothing prevents a school from doing so. Yet, not a single one of the schools in the 25th percentile "extreme risk" band requires (or even indicates on its website that it will accept as a supplemental material) any more substantive writing sample than a 4 page "personal statement" (which is not analytical writing) and the laughable, ungraded writing portion of the LSAT. About half of the schools do not even require a personal statement, and most that do restrict its length to even less than four pages.

Oh, and whatever happened to Prof. Lyke's response to Prof. Frakt?

[Eric Rasmusen]

I encourage anyone in a law school who can get access to the data--- usually kept very secret to prevent embarassment and knowledge of illegal affirmative action liability--- to do the simple regression. What I'd also like to see, but is more difficult, is to see *which* undergrad grades predict law school performance. Look at specific courses--freshman English or math. They probably predict bar passage a lot better than overall GPA. It's like first-year law school GPA being better than overall law school GPA. In fact, maybe first-year civil procedure is better than first year GPA. Also look at undergrad major.