In my last post, I discussed the need to consider both sides of the ratio when examining bar passage rates. Looking only at a particular law school’s passage without examining the overall bar passage numbers can lead to incorrect conclusions. In this post, I am going to challenge the belief that law schools can easily predict which students will do well in law school and the bar. This is based partially on my years of experience being on the Admissions Committee for my school (not a current assignment, fortunately, as it is a really hard job) and on the statistics that have been developed about applicants.
When an individual applies to be admitted to a law school, a collection of objective and subjective information is provided to the admissions committee from which a decision can be made. The key objective information is the undergraduate grade point average (UGPA) and the LSAT score. The subjective information includes letters of recommendation, essays, c.v., and even the student’s transcript. The admissions committee and staff stir this information around and offer some students a seat. It seems that much of the discussion about admissions standards revolves around the objective indicators, so the focus of this post will be with those.
Without a doubt, the objective information drives the process at most schools. The subjective information is used, too, but UGPA/LSAT control most of the decision-making. Typically, an applicant must be defined objective standards first before any subjective information is considered. Indeed, much of the current debate about appropriate admissions standards rests on this assumption and focuses fairly exclusively on law school’s alleged failure to accord these objective measurements their driving force. The essence of the argument is that some students with low objective indicators should never (or at least hardly ever) be admitted to law school. See David Frakt’s LSAT Score Risk Bands. The major problem with this argument is that it ignores what the statistics tells us about the LSAT and UGPA.
According to the LSAC (the publishers of the LSAT), each of the objective data points are correlated with first-year law school performance. See LSAT Scores as Predictors of Law School Performance. For the LSAT, calculated on a school-by-school basis, the relationship correlates at between .19 and .56; for UGPA it is between .06 and .43.
Looking first at UGPA, at the bottom end, a correlation of .06 is statistically meaningless unless there are at least 1,000 students in the data set. See Table of critical values for Pearson correlation. According to the ABA, the largest admitted class last year (2016 data has not yet been released) was 576 at Georgetown. See ABA 2015 1L Data. The only other school above 500 was Harvard. See id. For these two schools, the correlation needs to be greater than .07 to have any validity; for the rest of us, a much higher correlation is needed. For my school, U.Mass., with an entering class in 2015 of 71, the correlation must be above about .19 to be valid. Even the highest correlation found, .43, is considered to be a statistically weak relationship. See How to Interpret a Correlation Coefficient r.
The correlation between LSAT and the first year’s performance is stronger than the one for UGPA; indeed, the highest correlation found, .56, is considered a moderately strong correlation. Further, with the lowest correlation at .19, schools with more than about 70 matriculates (all but seven ABA schools) would have a statistically valid result, although that correlation is considered weak.
There is a problem with these relationships, however. If a law school is supposed to recognize which students will ultimately pass the bar examination based on pre-admission data (and thus not be a “bottom feeder”), it must have data that predicts bar passage. Neither of these objective values do this.
To begin with, the relationship tested by LSAC is between UGPA/LSAT and first-year law school performance, not between the indicators and bar passage. The LSAC is not measuring the relationship with graduating law school or with passing the bar examination. Measuring either of these is challenging. If, for example, an individual drops out of law school or transfers to a different school, does that mean that he/she flunked the bar?
More importantly, the power of prediction of the objective measures is, even at their best, weak. Statistically, to obtain an estimate of the proportion of influence the correlated value has, you calculate the coefficient of determination. See Coefficient of Determination. This allows you to approximate the percentage effect that the correlate has. It is a simple calculation as all you have to do is square the correlation obtained. Thus, for UGPA, the coefficient runs from 0.4% to 18.5%, with a median of 6.8%. Likewise for LSAT, the coefficient ranges from 3.6% to 31.4% with a median of 14.4%. In other words, for the median school, the LSAT captures about 15% and UGPA captures about 7% of the student’s ultimate probability of succeeding in their first year of law school, leaving about 80% to other factors. My favorite example here is a student from years ago who had an UGPA of 2.00 but graduated near the top of our class. When I asked him about it, he indicated that he had specialized in a different kind of bar in college and was now on the wagon.
The conclusion that has to be reached is that there is no simple way to identify a potential student as one who should absolutely not be admitted to law school. LSAT and UGPA can not be ignored in the process, but their use is of limited value. A school with a low LSAT or UGPA spread in its entering class might be admitting anyone with a pulse or they might be successfully using the other 80% to find a valid class. To find a “bottom feeder” — particularly one who is in violation of ABA Standard 501(b) — a full analysis is needed that examines how the admission decision is being made and by whom, what factors are being considered in the decision-making, and how well the school is educating its class including its students’ attrition rate and success with the bar. To do less that the full analysis is unsupported by the statistics.