LSAT Scores and Eventual Bar Passage Rates

There's a lot to say about professional responsibility, law school admissions, LSAT scores, and risk of failing the bar exam. Before I return to those broader issues, I want to address a factual question that has emerged in these discussions: Are LSAT scores related to eventual bar-passage rates or just to first-time rates? Most people who raise this question acknowledge that there are serious costs to retaking the bar exam--so first-time rates matter. Still, it is valid to ask whether LSAT scores correlate with eventual bar passage: eventual passage rates matter too.

The first, and most important, response to this question is that each law school knows (or can readily determine) the answer for its own graduates. Bar examiners used to hoard bar passage information in some states, but most now share it readily. My law school knows which of its graduates pass the bar, not only the first time but in subsequent attempts. We also, of course, have a wealth of other data about those students. It's not hard to create both simple correlations and more complex regression analyses identifying factors that predict eventual bar passage or failure.

As an aside, I did this during the late 1990s for my own law school, partly to inform a faculty debate over whether students who took upper-level commercial law courses (e.g., Commercial Paper, Secured Transactions, or Advanced Contracts) were more likely to pass the bar than their classmates. They weren't--although students who took the basic Trusts and Estates course were. LSAT, UGPA, and law school GPA were also significant predictors. [If you're a Trusts and Estates professor, don't rush to your faculty with this news. The most likely explanation for that association is that students who took Trusts and Estates were more conscientious than other students about taking a full slate of bar courses. It wasn't T&E itself that prepped them for the bar; it was their attitude toward course selection.]

In any event, if I could do this in the late 1990s as a solo project without any staff help (or even a blog to post the results on!), any law school can do it today. To act responsibly toward their students, law schools should make these analyses. What variables correlate with eventual bar passage at your school?

National Research

Let's say, though, that your school can't obtain the data for these analyses. Or maybe the credentials of your current students are quite different from those who have already graduated and taken the bar. Is there any existing research that suggests a correlation between LSAT scores and eventual bar passage? 

Sure. The LST report includes some of the relevant research; Robert Anderson and David Frakt cite other sources. The table below summarizes one more dataset that I find illuminating. The first and third columns in the table come from LSAC's landmark study on bar passage. In the course of that study, LSAC reported both the mean LSAT score and eventual bar passage rate for five different racial/ethnic groups. (The study included a total of seven racial/ethnic groups but two of those groups were too small for reliable analyses.)

The table helps visualize the association between LSAT scores and eventual bar passage rates. (The middle column offers a rough translation between the LSAT scores used before 1992 and those used today.)

Eventual Pass Rates: 1991 Matriculants

Mean LSAT Score (Old)	Mean LSAT Score (New) (Est.)	Percentage Passing Bar
29.39	146	77.63
33.05	152	88.44
33.85	154	89.04
36.07	157	91.88
37.50	160	96.68

Three points/caveats about this table. First, the table rather clearly shows that higher LSAT scores predict higher eventual bar passage rates--while lower scores are associated with more risk of failure.

Second, the LSAT scores in each row of the table represent means drawn from a large number of students. Within each group, the overall pattern probably repeats. We cannot, in other words, read the first row in the table to indicate that a student with an LSAT of 146 has a 77.63% chance of passing the bar. The students represented by that mean included some with higher LSATs and some with lower ones. The ones with higher LSATs probably enjoyed (on average) greater success on the bar while those with lower scores enjoyed less success--even after multiple attempts to pass.

The means, however, still offer some sobering insights. The students in the LSAC study with an LSAT score of 39-40 (about 146 on today's scale) were unlikely to enjoy a higher probability of passing the bar than the 77.63% identified with that mean. Similarly, students with LSAT scores below 39 (146) almost certainly faced a lower probability than 77.63%--again, even after repeated attempts to pass the bar.

Finally, insights from this table are somewhat limited by the fact that the groups in each row represent different racial/ethnic groups. To the extent that those groups vary in their law school performance (after controlling for LSAT), persistence in taking the bar, or other factors, the bar-passage probabilities reflect some of those differences. The table, in other words, does not reflect 5 groups randomly drawn from the population. I will return to this issue in an upcoming discussion of race/ethnicity, law school admission, and bar passage.

Associations and Causation

The data offered above, along with data already publicized, support an association between LSAT scores and bar passage--whether one examines first-time bar passage or eventual passage. Association, of course, is not causation. The relationship suggests that LSAT scores can predict an applicant's eventual success in passing the bar. The score doesn't cause bar passage or failure, any more than an LSAT score causes a student to get particular law school grades. Predictors, however, can offer useful insights when making decisions.

Predictors, on the other hand, are imperfect. Some students perform better in law school than their LSAT score predicts; others perform worse. Sometimes we can predict those variations because of other information about the student; e.g., a student with a higher UGPA likely will perform better with one who has the same LSAT score but a lower UGPA. Other times, the performance surprises us.

The same is true of bar passage predictions based on LSAT scores. LSAT scores are an imperfect predictor, but they offer information that a responsible decision maker would consider; in fact, we already do consider that information in law school admissions. Next up, I'll consider the appropriate way to use LSAT scores to consider whether we--or other law schools--are fulfilling our obligation to put student interests above self-interest.

Comments

[anon]

You seem to be attacking another straw man. Of course someone who performs better on the LSAT, a logic-based multiple choice test, is more likely to eventually pass the bar. I can't imagine how anyone could disagree with this post.

I can only assume that your next post will argue that it is unethical to admit a student who has a 77.63% chance of eventually passing the bar.

[David Frakt]

Another caveat - these eventual bar passage rates only take into account people who actually got their J.D. and took the bar exam, not the people who failed.

[jDsec]

Amongst the US News top 50 schools, Ohio State has among the lowest median LSAT (159) and 25th (156).

[Jojo]

Two comments:

1. Until about six months ago, I had never heard anyone discuss "eventual bar pass rates." I'd heard discussions of bar passage, bar pass rates, bar failure, and occasional discussions of repeater scores on the current exam, but never "eventual bar pass rates." I knew that they were tracked per the ABA standard, but they were never mentioned in public among polite company.

It says a lot about the current state of affairs that a nontrivial number of law schools invoke an eventual bar pass argument. "Well, I think this criticism about our open admission criteria ignores the fact that if they stick with it and scrounge up tens of thousands more dollars, eventually after 3 or 4 six month stints, our little achievers can eventually pass the bar. Then they can enter the glutted marketplace and the sky's the limit." Who reasons like that?

2. I would love - LOVE - one post here or on prawfs wherein all faculty post (anonymously, of course) and just admit that they are talking their book and love the pre-2010 law school model and fear losing it. (Perhaps we could do it on December 23 for Festivus)

Here are the facts: first, we now have zero exclusivity in law. If you can graduate from college, have no major criminal issues, and you can sign a loan form, you can enroll at an ABA law school; second, as the practices and outcomes of law school became more transparent, demand fell like a stone. Think hard about that one; third, it is a daily battle out there for young Small law lawyers just to remain in the game. It is so glutted that there are hundreds of applications for every infrequent government law job that posts. Small Law firms are seeking associates - and getting them - in major markets for under $40 grand with no benefits. Doc review rates have been cut to $25 per hour. Last week a suburban justice court judge told me that he just had a DUI trial with a lawyer who charged $350 for a DUI trial! (I know it means nothing to faculty, but that rate is bad -- like a quarter of what it used to be ten years ago). Student quality is down because the profession is down. It is just that simple.

[[M][@][c][K]]

I really find the projection of so many of the anon's here almost amusing - their current trope is accusing people of raising straw-men, while busily stuffing their own.

There are two big strawmen being floated by the defenders - first, that LST has proposed the LSAT alone as an admission criterion - cf anon at 10:22 PM (by the way, is ti too much to ask the anon's to pick panhandle so we can keep straight who is shilling here.) Second, that LST proposed an LSAT cutoff, let alone one of 150. To that is added another implicit straw man, the bottom-feeding law school that is in fact considering factors beyond LSAT and GPA, and not carry very much about those two anyway. It is transparently obvious that for many of these schools, they are not looking at the "other factors" the anon's happily prattle about - the only other factor those schools care about is the ability to borrow Federally backed loans.

And here is another demonstration from anon at 10:22 PM: "I can only assume that your next post will argue that it is unethical to admit a student who has a 77.63% chance of eventually passing the bar."

anon at 10:22 PM knows that is not the point, it's a straw-man, because this anon knows the other parts of the question:

• how much is it ethical to charge in tuition someone who has only a 77.63% chance of passing the ba?
• how ethical is it to admit them if they have a less than 50% chance of post-law school employment where their JD is even useful;
• how ethical is it to rent-seek $40-$50,000 p.a., in tuition, $150,000 from someone who may be the first person in their family to go to law school, who could do much more for their family and community by not going to law school;
• How ethical is it to admit students who could have above average earnings when the majority of graduates from your law school will earn less than $50-60,000 p.a..
• How ethical is it to take that tuition knowing that your graduate will be impoverished by that law school debt and end up in a program like IBR?

Because anon knows that is the real question - the outcomes as a whole from schools like Whittier, Florida Coastal, etc. and the rent seeking that admitting students who are very unlikely to benefit (though anon obviously does) from their attending law school.

[Derek Tokaz]

It may help to be a bit more specific with what we mean by something like "a 77.63% chance of passing the bar."

For most students going into the bar exam, odds of passing are going to hover near 100% or 0%, with not a lot in the middle.

This will be a bit simplified, but I think it'll demonstrate the point. There 200 questions on the MBE, and let's say you're going to need to get 140 of those correct. If you go in knowing the answers to 100 of them, your odds of guessing at least 40 of the remaining 100 correct are just 0.07%. If you know 110 and need 30/90, it's still only a 4.7% chance. If you know 120 and need 20/80, your odds jump up to 54%. Meanwhile, if you go in knowing 130 and need to get 10 of the remaining 70 right, your odds are 99%.

So, what do we mean by someone with a 78% chance of passing? Do we mean someone who goes in knowing how to answer about 125/200 MBE questions correctly? I don't really think so, because we're not really talking about individuals. We're talking about aggregates. I suspect it's much more likely that if we take 4 students with 146 LSATs, 3 of them will go in with near 100% chances of passing the bar, and 1 will go in without ever having a chance. There will be some people for whom luck makes or breaks the test, either because they guessed a few more right than normal or, more likely, they got lucky on topic selection.

For most people though, it'll be the ol' Ron Burgundy "60% of the time, it works every time." If this is the case, it puts the ethics question in a whole new light. I don't think there's something terribly unethical about admitting a student with only a 78% chance of ever passing the bar. We should aim higher, of course, but we're maybe not in definitively unethical territory either. But, if schools are instead accepting 3 students who will almost certainly pass and 1 who will almost certainly fail, it's a different story because 1 of those students should not be attending at all.

Schools won't have perfect knowledge about who will end up nearly certain to pass and who will almost certainly never pass, but I bet if they did the type of analysis DJM is describing above, they'd have a really good idea. And, I'm not sure there's an important ethical distinction here between what a school actually knows and what it ought to know.

[DifferentAnon]

Here's the thing that gets me... I could *maybe* give some credit to the arguments proffered by the bottom-tier, for-profit law schools about serving underrepresented communities etc. IF they gave special services to those students after admitting them to HELP them overcome the odds against passing the bar. For example, special bar prep classes throughout the 2nd and 3rd year with more personal one-on-one help. Or pre-selected courses in smaller sections to cover the bar material and really track whether those students are mastering it. Or even one intensive summer of pre-bar prep prep with JUST those students in a smaller class with the very best teachers. IF all that was happening, and there was fair disclosure on the front end-- then I guess I personally don't have a problem admitting students whose metrics seemed "off" because the law school was making an active choice to give special attention to those students to raise their performance. But that's not happening at MANY of the schools. That's the real rub for me. If your real motivation is serving minority students-- then SERVE them. Don't just admit them, hang them to dry, and then act surprised when they perform the way the predictors suggested they would.

[Michael Risch]

Derek explains it quite well and is one of the most cogent posts on this topic I've seen. And you can be sure that schools - even those that are admitting low scoring folks - are doing this analysis even if they aren't sharing it with the public. They know what students are passing or not, and I'm sure they have people looking at those numbers.

That said, I'm certain that many schools are making cuts of the data in a way that folks commenting here wouldn't agree with. But I can tell you having seen the bar data at my own school that a singular focus on low LSAT is not the way to go. Yeah, I know LST has a bunch of caveats that says it's a starting place, and maybe it is. But it is most definitely not the ending place.

[Kyle McEntee]

Michael, the biggest problem the other indicators is that they're not available to schools until after one probably two semesters -- maybe more. These indicators are useful for attrition and additional academic or bar support, but the decision to let students continue their studies is a different one than letting them attend in the first place.

In a perfect world, perhaps we let everyone try their hand at law school so we can find the people best suited for a legal career. But in our world, we owe a duty to ask the questions m*** asked just above. His questions really were on point.

I've said this over and over. If we shift more of the burden of risk to schools or students who are close-to-assured of passing the bar, we probably end up with a fairer structure. That's the best way to prove a school's commitment to opportunity for students, as opposed to an opportunity for revenue.

Overall, I think most of the best arguments advanced against (against isn't quite the right word...) our report are ones that apply to schools we're not really that concerned about in terms of risk of bar failure (whether first-time or eventual). In terms of composite bar passage rates used for Standard 316, 15 schools for 2013 graduates were below 65%. For 2014 grads, it's 27, and we're four more years of significant aggregate decline away from when those 2014 grads started school (2011).

And if it's not incoming students with worse predictors like LSAT and GPA, then what is it that explains the bar exam declines the last two years? Will any law school admit the education is worse? Did BarBri, Kaplan, and Themis change their programs? All the hand waving in the world (and this is not directed at you) won't change the basic facts on the ground.

[Matthew Bruckner]

Deborah suggests that the following factors were relevant to bar passage at her law school in the late 1990s: (i) LSAT; (ii) uGPA; (iii) lGPA; and (iv) taking T&E. She also finds that a commercial bar prep course was not relevant. As I understand matters, my law school has found 1L GPA relevant and active participation in a commercial bar prep course are highly relevant. (I don't mean to suggest that the other factors are not relevant, just that I haven't seen any data on our students suggesting that it is relevant.)

One of the relevant factors for us can only be assessed AFTER 3 years of law school, and the other after a full year. Maybe we're behind the times on data crunching and/or the sophistication of our modeling, but I find the suggestion troubling that my school is knowingly admitting people that our data tells us will almost surely fail the bar. Maybe my colleagues are so parochial as to protect their perceived sinecure over our students' well-being, but I am generally more optimistic about human nature than that. Though, reading the posts on this blog make me think I'm in the minority...

In any case, I've taught at three law schools in the last ~5 years. Two of them hired a full-time academic support person while I was at the school. One didn't have one when I left. It would seem, therefore, that none of these schools have a rich trove of historic data on how much academic support can move the needle on bar passage and how interventions should be targeted. Isn't this what our schools should be doing? If the data eventually suggests that our students are likely to fail the bar despite our current set of interventions, shouldn't schools try something else? Or should they just stop admitted at-risk students?

Maybe continuing to admit at-risk students is fraudulent, or a violation of the ABA's "non-exploitation" rules, but it doesn't seem obviously so to me. Law schools are operating in a dynamic market and are under a lot of stress as the pool of candidates has shrunk. They're adjusting to the "new normal" on the fly. To me, they appear to be doing what other distressed businesses do. Maybe law schools should be help to a different standard because they provide a public good, because they enjoy a public subsidy, or otherwise, but it's not obvious to me that this is true.

[El E]

The question that you might want to ask is - should all schools be decreasing class size? Here is a chart of those who have and have not since 2011: http://associatesmind.com/2015/12/15/2015-aba-509-disclosures-fifty-shades-of-grey/

Notably, the University of Colorado has increased class size by 25% over this period, the third largest increase. Meanwhile, one of their outspoken professors -- a hero to LST and the scambloggers -- is on his blog attacking another school yesterday for having the audacity to increase its class size (and then targeting one of its professors -- an enemy of the scambloggers -- for criticism).

Perhaps if the scamblogs attacked the problems rather than targeting and defaming individuals it might make more sense. The scamblogs seem to spend an inordinate amount of time targeting a University of Chicago professor - do we really think that school is a problem or is churning out too many lawyers? They single out junior professors who write articles they do not think is worthy -- why should we care?

[twbb]

I just wanted to note that while we keep talking about whether schools should admit sub-150 scores, at this point admissions standards have moved to the point where several schools are admitting significant numbers of people with sub-140 scores. I'd be curious as to whether those defending letting people in with 146s maintain that argument when talking about 136s...

[Deborah Merritt]

Matthew, thanks for your comments. One clarification: My reference to "commercial law courses" means upperlevel law courses like commercial paper and secured transactions--not commercial bar prep courses. I did the analysis because some colleagues repeatedly asserted that they "just knew" that their commercial law courses were the key to bar success. (I have trouble resisting an empirical challenge.) I didn't have access to data on bar prep courses, so didn't include that in my analysis.

On your deeper points: I'm relatively upbeat about human nature, but I think it's hard (and dangerous) to dismiss the influence of self-interest. Humans make some of their biggest mistakes when they overlook the possibility that self-interest is tainting their judgment. As professionals, we should be especially wary of the role of self-interest because we enjoy the privilege of self-regulation.

Because of our professional status, I disagree strongly with the idea that law schools are free to act like other distressed businesses. Professionals have privileges that other business people don't enjoy. E.g, as faculty we enjoy the privilege of tenure (or the ability to earn that status), along with considerable autonomy in deciding what to teach or research. As legal educators, we benefit from the legal profession's exclusive right to practice law; our status as gatekeepers to that limited profession allows us to charge a premium for legal education.

In return for privileges like these, professionals promise to put the interests of others (students, clients, patients) ahead of their self-interest. That's a greater obligation than a failing shoe store or video rental booth owes to its customers. We may differ on how to implement that duty (or on how our profession should collectively check itself on compliance), but I hope we can agree that law faculty are professionals--and that professionals owe different duties than other business people.

[Matthew Bruckner]

Even that you frame it as a "mistake" because of the taint of self-interest, suggests that you're more moderate than some. Reading the comments on this blog, one gets the sense that many believe that law professors are, on the whole, a completely self-serving bunch who would knowingly lie to potential students to preserve their jobs/status/etc.

I think the latter part of your post asks a provocative question and my initial response is that I disagree with you. Let's separate out for the moment the question of whether law school faculty SHOULD have different duties. On the question of whether they DO have different duties, I think the answer is that they do not. And one problem with many of the scambloggers arguments, to my mind, is that they suggest that a different set of legal duties does exist. But law faculty are not fiduciaries, are they? Law students, particularly potential law students, are not particularly analogous to clients, are they? I think the assumption that a duty does exist when it may not and, even if it does, is not well-defined can create a host of its own problems, such as accusations that these duties are being violated when the alleged violators may not even acknowledge that a duty exists.

On the former question, of whether they should, I think that you raise interesting points. I take it that you've made two arguments. First, that the privilege of tenure suggests that faculty have greater duties (of course, this shouldn't be just limited to law faculty, then, should it?). Second, because law is something of a guild, existing members of the guild benefit and therefore owe duties to potential new entrants, particularly to people who are accepted into the our training program (i.e law school). They're interesting and I'll think about them.

[terry malloy]

"Reading the comments on this blog, one gets the sense that many believe that law professors are, on the whole, a completely self-serving bunch who would
knowingly lie to potential students to preserve their jobs/status/etc."

Very few outside law school administration are actively shilling for the fleecing of the unwise in the vein of Dean Allard from Brooklyn. Law professors are more like mob wives. They know where the money comes from and they don't rock the boat. That is immoral.

Up until recently, law professors had the opacity of law student outcomes and the graveyard problem to assist in their rationalizations. Now, with LST and a few 'Upton Sinclair' explanations from law professors of how the student loan sausage gets made, there is little room for a law professor to claim the status quo is harmless and look him or herself in the mirror.

[Jojo]

"one gets the sense that many believe that law professors are, on the whole, a completely self-serving bunch who would knowingly lie to potential students to preserve their jobs/status/etc."

The mob wives analogy is apt.

We all know what's going on here, but why say anything. First they came for the sub-150 LSATs, and I did not speak out because I was not a sub-150 LSAT.

[Tammy]

I am new to this. If a student wants to invest 100,000 in going to law school and has a 70% chance of passing the bar, what is the ethical principle that requires them to be told no? If there is one, why does it not apply to going into debt to buy a car or house that may appear risky to others. I mean it's just a product and, as long as there is no fraud or misrepresentation, what is the beef?

[anon]

Tammy

I think what the whole argument comes down to here is the degree of disclosure required to avoid "misrepresentation" and, at what point a law school's cognizance of a significant barrier to entry into the legal profession is relevant to its admission decisions.

The latter question is more focused these days because so many law schools have been lowering their admissions criteria to fill classes.

[terry malloy]

Tammy, if a person sold you a car for $100,000 that they knew had a 70% chance of driving off the lot, and your car does not drive off the lot, would you feel cheated?

What if that person was a someone that culture has taught you to trust, like doctor or a policeman, and that person used that cultural bias to separate you from your $100,000.

It is also interesting how you try and distance the direct pecuniary interest of law professors in allowing the student to take on serious risk at serious cost.

"why does it not apply to going into debt to buy a car or house that may appear risky to others."

Professors are not a passive third party. The law school collects the debt funded tuition as revenue, and this revenue is paid as cost to professors on their W-2 in the form of wages.

[Cent Rieker]

Tammy,
Are you conceding that law school pitchmen are no different than either real estate agents or car salesmen? If so, then you should know for far too long law faculty maintained an information asymmetry, which enabled a lot of matriculants to incur debt for degrees at schools not well-regarded by either the legal profession or the legal academy. Once the information on how integral a law school's pedigreee became known, applications, enrollments and admissions standards all plummeted.

Now going back to cars and houses, both require certain standards. Car manufacturers cannot sell cars that explode, and home builders need to build homes to code. There are no comparable rules for higher education. Instead, the federal government provides unconditional for people to attend such fine institutions like Whittier, Thomas Jefferson, and a whole slew of for-proft law schools (basically educational "lemons." The only beneficiaries are well credentialed law professors who want six figure salaries (at schools they wouldn't be caught dead ever attending), but not have to deal with 2000 + annual billable hours.

For more on information asymmetry, check out the work done by Nobel winning economists George Akerloff and Joseph Stiglitz. It is almost as impressive as economic research done by Seton Hall Law Professor Simokvic.