Important New Report From Law School Transparency on Law School Admissions

An important new report entitled: 2015 State of Legal Education: An in-depth look into law school admissions choices” has just been released by Law School Transparency (LST).  (Full disclosure – I had a minor role in editing some sections of the report.)  The report has already received attention in the press (see this editorial in today’s New York Times) and is likely to receive more in the coming days.  It is also likely to be a topic of conversation among current and prospective law students, many of whom rely heavily on LST for reliable information about law schools.  I recommend that all law professors and law school administrators read it in its entirety.  In the meantime, I offer below a summary of what I consider to be some of the most important findings and recommendations in the report.  

As the title implies, the report focuses on the questionable admissions policies practiced by many law schools during the enrollment crisis from 2011 to the present (although there are several other important subjects covered as well).  Using a risk profile that I developed based on LSAT scores (first reported here on TFL)  the report identifies 74 schools which admitted classes consisting of at least 25% at risk students in the fall of 2014 (statistics are not yet available for the 2015 entering classes).  Fully half of these schools, 37, admitted classes consisting of at least 50% at risk students (149 LSAT and below).  Most disturbingly, 18 schools admitted classes where over half of the students were categorized as being at very high or extreme risk of failing out of law school or failing the bar exam. (See the list of schools here.)

The report acknowledges that LSAT scores are imperfect predictors, and that the risk presented by admitting a student with a low LSAT score might be offset by requiring that the student have a correspondingly strong undergraduate GPA.  Through sophisticated statistical modeling, the report proves that this is not what is happening.  (See the section headed “Serious risk law schools did not take enough students with better undergraduate GPAs” in the Analysis section of the report.) 

To confirm the reliability of my risk-model, LST managed to acquire data regarding bar passage from two law schools. A school which chose to remain anonymous provided 7 years’ worth of data, and the University of Denver provided 14 years’ worth of data.  The data confirms that my risk categories are valid.  At the anonymous school, only 16% of students in the extreme risk category (144 LSAT or lower) passed the bar on their first attempt.  At the University of Denver, students in the extreme risk category passed on their first attempt at the slightly higher rate of 27%.   In the very high risk category (LSAT of 145 or 146), 19% passed the first time at the anonymous school, and 39% passed the first time at Denver.  In the high risk category (147-149 LSAT), 23% passed the first time at the anonymous school and 57% passed at Denver.  

   

The difference in bar pass rates between the anonymous school and Denver is largely attributable to the fact that the anonymous school is located in a state with a more difficult bar exam than Colorado.  While minimal risk students (those with LSATs in the top third of LSAT takers -- 156 or higher) performed almost identically at the two schools -- 88% at the unidentified school and 89% at Denver – all other groups faired much worse at the anonymous school.  For example, low risk students at the anonymous school (those with LSAT from 153-155) performed much more poorly than those at Denver, achieved a 63% first time pass rate compared to 84% at Denver.  Modest risk students (LSAT of 150-152) at the school in the tough bar state had a 50% first time par pass rate, compared to 71% at Denver.

Of course, there could be other factors at play. It is possible that Denver did a better job of preparing its students for the bar exam than the anonymous school. It is also possible that Denver admitted students with low LSATs only when they had better than average UGPAs.  It is also possible that Denver more aggressively attrited low performing students to weed out probable bar failers.  Unfortunately, with the data that is publicly available, it is not possible to determine which of these factors, if any, contributed to the difference.  But what these data sets reinforce is that LSAT scores have a very strong predictive value of bar passage. 

It is important to keep in mind that these numbers represent only those students in the various risk categories who managed to graduate and sit for the bar.  Presumably, a significant percentage of students in these high risk categories never graduated due to academic or voluntary attrition.  So the real chance for achieving the dream of becoming a practicing lawyer is lower, probably much lower, than even these dire bar pass statistics would suggest. 

The important question raised by this report is whether the schools that are admitting droves of very high and extremely high risk students are simply exploiting those students for tuition, knowing full well that such an applicant has a very low probability of success.  If so, it would appear that these schools are at least potentially running afoul of ABA Standard 501(b) which states that “A law school shall not admit an applicant who does not appear capable of satisfactorily completing its program of legal education and being admitted to the bar.”  While the ABA does not quantify what chance of success qualifies as “appearing capable”, I would argue that no ABA-accredited law school should admit any student who has less than a 50/50 chance of becoming a lawyer, at least not without specifically advising the student, in writing, of their predicted success rate based on the experience of students with similar LSAT and UGPA profiles at the same law school over the previous 5 years.  Schools desiring to offer “opportunities” to deserving students with low predictors (less than a 50% chance) should be required to offer the students at least a 50% refund if they fail within the first year, to prove that the school is not economically exploiting these students. 

Unfortunately, the ABA has never seriously enforced Standard 501(b).  Rather than closely scrutinizing admissions, ABA accreditors have primarily focused on back-end results. Thus, if a school is in compliance with Standard 316 on bar passage, it is presumed to be in compliance with Standard 501(b).  Acceptable bar passage rates may have been a reasonably proxy for appropriate admission practices in an earlier era. When law schools had stable or increasing law school applications, so that the entrance credentials of each succeeding class were equal or superior to their predecessors, then past success on the bar exam by that school’s students may have been a reasonable indication that the school was engaging in responsible admissions.  But in an era of declining applications, when each succeeding law school class is weaker than the last, looking at historic bar pass rates is all but useless in analyzing the soundness of current admissions practices.

We have already seen the results of the decision by many law schools in 2011 to start lowering their admission standards.  The graduating class of 2014 performed abysmally on the bar exam, and the graduating class of 2015 performed even worse (even without any ExamSoft debacle to blame).  The entering classes of 2013 and 2014 were significantly weaker than the 2011 and 2012 classes, suggesting that even more disastrous bar examination results are in the offing. Although complete statistics are not yet available for the 2015 entering classes, there is every reason to believe that the picture has gotten even bleaker this fall, as the number of applicants continued to fall in 2015, although not quite as precipitously as in the years 2011-14.

Although bar passage statistics at many high-risk schools have reached historic lows, few, if any, schools have yet run afoul of the weak and loophole-ridden ABA Standard 316 which sets acceptable bar passage rates.  Because so many law schools lowered their admissions standards at the same time, law schools have largely been able to stay within 15% of the state average despite plummeting bar pass rates.  The LST report recommends strengthening Standard 316 to close these loopholes for the future, while vigorously enforcing Standard 501(b) now.  (See Action Items and this letter to the ABA)

The report recommends such common-sense actions as requiring law schools with high risk admissions policies to justify those policies by providing detailed empirical data to the ABA Section of Legal Education.   Those schools unable to justify their admissions policies (and there are likely to be quite a few) should immediately be placed on probation.

Many of the schools on LST’s  serious risk list are not only ABA-accredited, but are members of the Association of American Law Schools. The AALS also requires that school select “students based upon intellectual ability and personal potential for success in the study and practice of law.”  The AALS should also closely evaluate the admission practices of member schools  that are admitting large numbers of high risk students and take appropriate steps to censure or expel those schools who are violating the core values of the association.

Law faculties should independently demand that the administration at their law school provide data on law school completion and bar passage by LSAT and UGPA profile, and then insist that no students be admitted who do not have a decent chance of success.  If law professors are to maintain any self-respect and live up to the high ethical standards of the legal profession, they must not continue to turn a blind eye to the exploitation of high risk students upon which their livelihood currently depends.

Comments

[Matt]

You claim to have confirmed the reliability of your model by checking how frequently people pass the bar on the first try. That seems irrelevant. Many great lawyers, including supreme court justices and governors, failed the bar on the first try.

[Jess]

I do no think being a governor is related to being a lawyer, indeed a lot of governors are not lawyers. I had not heard Supreme Court justices failed on the first try -- did you mean US Supreme Court, and if so, which ones?

[David Frakt]

First-time bar passage rate is widely considered to be a useful benchmark, and that is what the ABA requires that law schools report. So, for most schools, that is the only data that is available. It may also be useful for prospective law students to consider an "ultimate" or "eventual" bar passage rate, but considering the devastating consequences of failing the bar, financially and emotionally, I believe the focus is appropriately on first-time pass rates. Also, in my opinion, the more times a graduate takes the bar before passing, the less that passing the bar should be credited to the law school. The LST Report does include "eventual passage" rate for the anonymous school, but not for Denver. For those in the extreme and very high risk groups at the anonymous school, the eventual pass rate was 36% and 38%, respectively. Not very good. And schools are admitting students who are far weaker than the students who achieved these abysmal results. No law school has a 100% pass rate. Some people who are undeniably capable of passing the bar fail it, either due to a case of nerves, or being ill on the day of the exam, or, more commonly, not studying hard enough. The issue is whether schools should admit students that will likely never pass the bar no matter how hard they study.

[Mark Edwards]

Here's a question: what should the bar passage rate be? It's tempting to say 100%, of course, but they can't be right -- that would mean graduates were being challenged enough. The passage rate for the CPA exams is generally much lower than the average bar exam passage rate (https://media.nasba.org/files/2015/01/2014_Overall.pdf). An average performing university CPA program would probably be listed as an extreme risk under the metric used here.

On the other hand, medical board exam passage rates look an awful lot like bar exam passage rates, including the precipitous decline in the last five years. (http://knowledgeplus.nejm.org/abim-pass-rates-behind-declines/). Obviously, that can't be explained by lower entering LSAT scores, so I wonder if there is a common cause we haven't discovered yet.

Food for thought, anyway.

[Mark Edwards]

Here's a question: what should the bar passage rate be? It's tempting to say 100%, of course, but they can't be right -- that would mean graduates were being challenged enough. The passage rate for the CPA exams is generally much lower than the average bar exam passage rate. An average performing university CPA program would probably be listed as an extreme risk under the metric used here. On the other hand, medical board exam passage rates look an awful lot like bar exam passage rates, including the precipitous decline in the last five years. Obviously, that can't be explained by lower entering LSAT scores, so I wonder if there is a common cause we haven't discovered yet.

[David Frakt]

Mark -

Thanks for your questions. I don't know anything about medical exam boards. I know a little about CPA exams because my wife is a CPA. I don't think the CPA exam is a good comparison to the bar exam because people can't sit for the CPA exam until they have significant experience in accounting, and CPA candidates generally take the exam while they are already working full time. The CPA exam is broken into parts which are taken at different times. And while failing a section is common, it is also not a big deal. You don't lose your job. You just sign up and take it again a few weeks later. It costs a few hundred bucks, but is not a life changing event like failing the bar exam often is.

As for the question of what the bar passage rate should be, the LST report recommends changing the ABA standard so that a law school must have an 85% percent pass rate within two years (four administrations of the bar exam) after a student graduates. (According to the LSAC National Longitudinal Bar Passage Study, 99.9% of people who pass the bar exam have done so by the fourth attempt. After three attempts, the figure is 99.3%.) I believe this is a reasonable standard. In addition, I believe that ABA-accredited schools should be required to be within 15% of the state average, or have a 60% first-time bar pass rate, whichever is higher. Schools that drop below this standard for a calendar year (combining the February and July bar results) should automatically be placed on probation, absent extraordinary circumstances (like a large number of students from the top of the class transferring to other schools where they passed the bar on the first attempt). I do not consider it acceptable for an ABA-accredited law school to have a bar pass rate below 60%, even in jurisdictions with tougher standards, but there are currently several schools with rates in the 50s and even 40s that are still meeting the current ABA standard. Many of the students that failed the bar at these schools never should have been admitted in the first place because there was no evidence that they had reasonable aptitude for the study of law.

[Jojo]

I'm among the harshest critics of law school, but I think 85 percent is too high. I'd do 75 though. If 3 out of 4 cannot pass the bar in 2 years, your school is doing something wrong. Either you aren't teaching them the nuts and bolts of law or else you are admitting too many students who you know to be at risk (i.e. Scamming). 75 percent gives schools enough cushion to take a risk and admit a few possible diamonds in the rough each year without risking accreditation.

I'd also make my cutoffs have teeth. After 1 year you are on probation. If you get 2 more strikes in the subsequent 3 years, you lose accreditation.

[Anonymous]

Another option is reform the attorney licensure system. The main skill set necessary for passing the bar exam is the ability to cram for exams. As many have noted, lawyering never involves free writing for six hours at time without the opportunity to look up statues or laws. Similarly, practice never presents attorneys with multiple choice exams. Yet these are the two activities from which the bar exam derives its scores and determines the fate of thousands. Does that really make sense?

If anyone can cite any studies that show a correlation between effective lawyering and bar exam scores, that would be something. Instead, we rely on a century old process that is just as likely to reward an idiot savant as it is to punish someone without test-taking skills but who has the empathy and ability to be a damned good lawyer.

Note: this isn't sour grapes, I passed two bar exams. I just think the bar exam is an illogical barrier to practice.

[anonymous]

No study seems needed. Being a competent lawyer requires being a lawyer.

[RQA]

The LST report focuses, as does this discussion, on private and especially for-profit schools. But two categories of public schools stand out as problematic: historically black universities (the schools with the riskiest admissions) and universities in remote rural areas (viz., the Dakotas). I'm not sure how to think about the issues those schools present for this analysis: do they affect at-risk populations particularly in need of protection, or should we assume the schools' admissions policies have more benign motivations and should be given the benefit of the doubt? Either way, I'd want a policy placing schools on probation to consider which schools would seemingly be most likely to be involved or explain why and how they'd be treated differently.

[Paul Campos]

"The main skill set necessary for passing the bar exam is the ability to cram for exams. As many have noted, lawyering never involves free writing for six hours at time without the opportunity to look up statues or laws. Similarly, practice never presents attorneys with multiple choice exams. Yet these are the two activities from which the bar exam derives its scores and determines the fate of thousands. Does that really make sense?

If anyone can cite any studies that show a correlation between effective lawyering and bar exam scores, that would be something. Instead, we rely on a century old process that is just as likely to reward an idiot savant as it is to punish someone without test-taking skills but who has the empathy and ability to be a damned good lawyer."

Hmmm, I wonder if there's any other part of the required process for becoming a lawyer to which this very same criticism applies?

[David Frakt]

I would not endorse creating a separate standard for historically black universities. The admirable goal of promoting greater diversity in the profession is not an excuse for admitting students with exceedingly dim prospects of becoming lawyers. While it may be slightly less morally reprehensible to admit unqualified students for the purpose of keeping a worthy institution with an important mission open, as opposed to admitting unqualified students for the purpose of lining the pockets of the owners, it is still fundamentally wrong to exploit such students.

[Barry]

Matt: "You claim to have confirmed the reliability of your model by checking how frequently people pass the bar on the first try. That seems irrelevant. Many great lawyers, including supreme court justices and governors, failed the bar on the first try."

Many people have survived horrific battles.
Many people have survived falling from great heights.
Many people have survived being bitten by poisonous snakes.
Many people have [insert her]

[Mark Edwards]

It seems like many people agree that the bar exam is not a very useful measure of ability to practice, for fairly obvious reasons -- the exam format in no way resembles the practice of law.

But if that's true, then can the answer to this problem really be to only admit students with high LSAT scores -- high scores on an exam whose format in no way resembles the practice of law?

David Frakt may be correct that there is a high correlation between LSAT score and likelihood of bar exam passage. But that suggests that both the LSAT and the bar exam are measuring the same thing -- the ability to perform well on an exam with a format that in no way resembles the actual practice of law.

Many law schools and profs are implementing new forms of assessment that at least bear a passing resemblance to practice. That seems very wise to me. So that shouldn't change.

It seems like we've stumbled into a 4 step process for becoming a good lawyer that makes no logical sense:

1. Do well on a standardized exam that in no way resembles the practice of law.
2. Either (a) attend a classically Langdellian law school, and do well on exams that in no way resemble the actual practice of law; or (b) attend a school with a more practically-oriented means of assessment, and do well on those assessments.
3. Do well on a standardized exam exam that in no way resembles the practice of law.
4. Practice law well.

The best solution to this problem is clearly NOT to get rid of institutions described in 2(b) above. But, ironically, because such institutions tend to be lower prestige, they are the very ones that are composed of students with low LSAT scores. And it is true, as David Frakt points out, that low LSAT score is correlated with low bar passage rate.

If we do as many people suggest, and close down the institutions with low LSAT scores and low bar passage, then we compound the massive gap between law school and practice of law. If, on the other hand, we de-emphasize LSAT score and seriously reform assessment for purposes of bar admittance, we close the gap between law and practice. In either case, we are likely to have fewer people attending law school who discover that after all their investment of time and money they are unable to practice law.

But I would very much prefer we alter steps 1 and 3, rather than shut down institutions described in 2(b). I have a strong sense that the practicing bar would prefer that as well.

[Mark Edwards]

I'm hoping my previous comment shows up, because this is an addendum to it . . . .

In the meantime, it would be very interesting to compare actual bar passage rates to the predicted rates using David Frakt's metric for all law schools. We could then compile a ranking of schools based on the extent to which their graduates exceed bar passage rate predictions (or not) based on their LSAT scores. If we are still going to use the LSAT and the bar exam, it would be useful to know which schools are the most transformative for their students. Under the current system, arguably, those schools are doing the most effective teaching.

For example, it's possible that there are 2 schools, one with an expected bar passage rate of 40% that is producing 80% passage, and another with an expected bar passage rate of 95% that is producing 85% passage. Under the LST proposal as I understand it, the first would lose accreditation and the second would not. That seems like a very bad result to me, as we would be losing effective teaching and keeping ineffective teaching.

[RQA]

Perhaps the ultimate answer should be the same for law schools at historically black universities as for for-profit law schools, but I think the analysis has to be significantly different. Were every school on the LST list of 18 with the 50th percentile at "extreme" or "very high" risk to close, we'd lose only three for-profit schools, but every (all four) historically black school outside D.C. I'm inclined to believe that prospective students who then wouldn't be able to attend Arizona Summit, Florida Coastal or Charlotte would either have other decent choices or shouldn't have been going to law school in the first place. I'm not so sure about the prospective students who wouldn't be able to enroll at North Carolina Central, Florida A&M, Southern University or Texas Southern. I recognize that some of the students who'd be displaced should not be in law school at all. But I'm not sure the students who might stand to benefit from law school would be equally well-served by the majority-white options they'd be left with. (I know the three for-profits aren't exactly majority-white either, but they're a far cry from the historically black schools.) I'm harping on this because it seems to me LST is ignoring the impact of its recommendations on historically black schools. And, to reiterate, I'm not sure those schools shouldn't be drastically reformed or closed -- but I think the costs associated with that outcome would need to be weighed very carefully and quite separately from the effect on for-profit schools.

[Anon]

I appreciate efforts to collect and analyze data relating to law school. More information is a very good thing so kudos on that. But on your "reliability" test, I do not understand why you would draw such generalizations on reliability from a sample of TWO schools. It's just not representative and your attempt to discuss potential variables is very weak and incomplete. Why can't you just note the data and let the reader decide what to make of it? I can think of other school couples that you could have used where the results would have differed significantly. I think you weaken your argument by going a few steps too far.

[David Frakt]

Anon -
I am not drawing generalizations based only on two schools. Keep in mind that I developed my model based on my review of several years of data at Western State University College of Law, and by reviewing the available research. I also taught at Barry Law School for two years where I observed similar results. My father was formerly Dean of Loyola Los Angeles and Widener University Law School and he observed similar results. The data from these two schools confirms what I already knew, but which some of my critics, such as Jay Conison at Charlotte Law School, have vehemently denied. What is significant is that this is the first hard data that we have from any law schools that has been made publicly available. If other law schools are able to produce better outcomes with students with similarly low entrance credentials, we would expect those schools to want to publicize that information, and it would be very useful for prospective students to have that information. The main thing that I, and LST, are calling for, is greater transparency and enforcement of existing standards. Schools should provide empirical data to the accrediting agency (the ABA) justifying their admissions policies. If they are admitting students that they know to have a 10 or 20% chance of completing law school and passing the bar, then they should be forced to publicly acknowledge and defend that practice (if they can). Whatever your view of the efficacy or value of the bar exam, it seems highly unlikely that it is going to go away or be radically reformed anytime soon. The students being admitted to law school are almost certainly going to have to take a bar exam very similar to the one that droves of students just failed in July. And, unless states dramatically lower the score required to pass, then record percentages of graduates in 2016, 2017, and 2018 are going to fail. I view this is a large-scale, but completely preventable, human tragedy. LST's proposals would not necessarily result in the closure of any law schools, but the proposals would require law schools which are currently exploiting unqualified students to stop doing so. This would likely mean getting dramatically smaller and laying off some employees until law school applications rebound. If a school can't stay in business without exploiting unqualified law students, then it is no great loss to society if that school is forced to close.

[twbb]

Anon, in statistical terms it's not a sample of two schools, because the inference is not being drawn about the schools but about the students.

[anon]

The fact that so many who run and teach at law schools have never been or wanted to be lawyers is proof enough that the system is a crooked one. This needs to change. The ones losing here are students who seek nothing more than a move up the socioeconomic ladder and have a fulfilling career.