Last week, the ABA released a new product, a spreadsheet which compiles data on bar passage from all ABA-accredited law schools. The spreadsheet is supposed to provide more timely information for consumers, and make it easier for prospective law students to compare the performance of different law schools. In this series of columns, I will provide some analysis of the data and the lessons that can be gleaned from it. In this first column, I will provide an overview of the information contained in the various pages of the Spreadsheet. In subsequent columns, I will do a deeper dive into the data.
Before I start explaining the Spreadsheet, let me make one general comment. The raw data on bar passage tells us very little about the quality of the education provided at any particular law school, or how well that school prepares its students for the bar. One should not draw the conclusion that a school with an 80% first-time bar passage rate is “better” than a school with a 60% first-time bar pass rate. It is critically important to have additional information to put the bar passage rate into context. First, you must know the jurisdiction(s) where the majority of the graduates take the bar, what the average bar pass rate there is, and how strong the other schools are that provide most of the test-takers in the jurisdiction. Second, in order to make a meaningful comparison between or among law schools, you must know the entrance credentials (LSAT scores and GPA) of the students from each of the schools you are comparing (which you can get by looking at the Standard 509 reports from three years earlier). To illustrate this point, consider the example of Brooklyn Law School and the University of San Diego. In 2017, these schools had a nearly identical first-time bar pass rate: 77.36% for USD and 77.37% for Brooklyn. Virtually all of USD’s grads took the California Bar, while virtually all of Brooklyn’s grads took the New York Bar. The California Bar is arguably the toughest in the country, with a 66.19% pass rate in 2017. (Arizona’s rate was lower at 63.99%, but this was primarily a result of the extraordinary incompetence of Arizona Summit’s graduates, and does not reflect a higher cut score in Arizona.) The New York Bar is among the easiest in the country, with an overall first-time pass rate of 83.92% in 2017. (Only New Mexico, Missouri, Iowa and Oklahoma had higher first-time pass rates in 2017.) So, the University of San Diego performed far above average, +10.51% on the 2017 California bar, while Brooklyn performed below average, -5.18%. So, USD students really outperformed Brooklyn students, right? Not so fast. Remember, it is important to also compare the caliber of their students. USD’s entering class of 2014 was very strong with LSAT 75/50/25 of 161/159/155 and UGPA of 3.66/3.50/3.23. In comparison, Brooklyn’s entering class of 2014, although above average, was not nearly as strong as USD’s, with LSATs of 159/156/153 and UGPA of 3.53/3.31/3.05. The extra 2.33 LSAT points and .17 GPA points per student is statistically significant. To truly understand how these schools performed, we have to compare their performance to other schools in their home state. Of the 15 New York law schools, only one law school with weaker entering students (and only slightly weaker), City University of New York, beat Brooklyn’s pass rate, while Brooklyn’s rate surpassed seven schools with weaker students. Similarly, of the 20 California law schools, only one law school with weaker credentials (again, just barely weaker), Santa Clara University, beat San Diego’s pass rate. USD’s pass rate surpassed all of the other California law schools with weaker or comparable students, and it beat one, UC Davis, that had slightly stronger entrance credentials, by one percent. So, in essence, both USD and Brooklyn performed pretty much as expected, and we can't really conclude much about the quality of the legal education that these schools provide based on their bar performance in 2017. Another important lesson is not to judge a law school based on one year’s performance. For example, Brooklyn’s pass rate in 2016 was 80.66% while San Diego was at 72.45% in 2016. Swings of 5 to 6% from year to year are not unusual and not necessarily indicative of a trend.
Now, on to the Spreadsheet.
The inaugural spreadsheet (it is a Microsoft Excel file) includes five different sheets or pages.
The first page, entitled “Ultimate Bar Passage 2015” provides data on the cumulative bar performance over the last two years of students who graduated in 2015. So, for example, if a law school had 100 graduates in May 2015, and they all took the bar, and 60 passed in July 2015, then 10 more passed in February 2016, then 4 more passed in July 2016, then 1 more passed in February 2017, the school would have an “Ultimate Bar Passage 2015” rate of 75%. This data is very useful because many law schools with low first-time bar passage rates have asserted that a high percentage of their students eventually pass the bar, but the information to verify (or debunk) these assertions was never publicly available. The two year time-frame was chosen because over 99% of those who pass the bar do so within two years of graduation, and because if it takes more than two years for a graduate to pass, the law school probably doesn’t deserve much, if any credit, for that accomplishment. In addition to the actual “UBP” rate, this page provides some interesting new statistics about the number and percentage of graduates who never take the bar. The ABA Council has proposed a new standard, which I support, see here and here. Under this Standard, law schools must have a 75% ultimate bar pass rate within two years. By including this page on the spreadsheet, we are able to see what the impact would be if this proposed Standard is adopted.
The second page is “First time Bar Passage 2017” which is self-explanatory. This page also includes interesting information about the number of graduates who did not immediately take the bar, and the number of graduates from prior years who took the bar for the first time in 2017. This page also includes the average state pass percentage and the average pass difference, so you can compare the bar performance of one law schools’ graduates to the performance of all ABA law school grads in the same state. The fourth page, “First Time Bar Passage 2016” presents the same date for 2016. (Note - First Time Bar passage for 2015 is available on each law schools’ Standard 509 Report for 2016.)
Under the current ABA Standard 316, a law school can be in compliance if it is within 15% of the state average in 3 out of the last 5 years, or if it has a 75% bar pass rate in 3 out of the last 5 years, or if it has a 75% ultimate bar pass rate within five years of graduation. Under the current standard, a school must only report on enough jurisdictions to account for 70% of its graduates. This has enabled some law schools to hide the true bar passage rate of their graduates, by not reporting the results from states, like California, where their graduates did very poorly. The new spreadsheet is more reliable because it includes data on all students for whom data is available, and provides the number of students for whom information is missing.
The third and fifth spreadsheets “Jurisdictions 2017” and “Jurisdictions 2016” reports the passing statistics for each school in all states where at least 10 of their graduates took the bar for the first time in those years, and provides the state pass differential. Sorting this sheet by jurisdiction enables an easy side-by-side comparison of the performance of all law schools in that state. For example, by sorting for California, you can see that American University grads did worse there than any other school in 2017, with 7 of 28 passing (25%). American was slightly better in 2016, with 13 of 34 passing, for a 38.24% rate. Tulane was the worst performer in California (with at least ten takers) in 2016, with just 2 of 17 passing for an 11.76% rate. Tulane fared a bit better in 2017 with 5 of 18 passing, for a 27.78% rate. Because these numbers represent a fairly small percentage of the graduates of these schools (about 8%), these results would not have been included in the school’s bar passage rate under the old reporting system, even though they are statistically significant for the overall bar pass rate. On their 2016 ABA 509 Report, American reported results from only New York, Maryland and Virginia for 2013-15, and Tulane reported results from only Louisiana, New York and Texas for 2013-15. This demonstrates how under the old reporting rules, schools could report inflated bar passage results, while still complying with the rules.
Comments