The LSAT-free Illusion

I'd like to thank Al Brophy for inviting me to be a guest in the Faculty Lounge.  As Al has already indicated, I am particularly interested in the proposal before the ABA Standards Review Committee to change the minimum Bar passage benchmarks in ABA Interpretation 301-6 (Report of the Subcommittee on Bar Passage for April 2011 meeting). 

I'd like to begin by discussing the proposed elimination of Standard 503, which requires law schools to use admissions tests.

Standard 503. ADMISSION TEST

A law school shall require each applicant for admission as a first year J.D. student to take a valid and reliable admission test to assist the school and the applicant in assessing the applicant’s capability of satisfactorily completing the school’s educational program. * * *  (p. 2).

The argument for eliminating Standard 503 is that LSAT scores are at best an imperfect predictor of the academic success (see the Statement in Support of Eliminating the Requirement of an Admissions by the Society of American Law Teachers, March 2011).  Although the Subcommittee on Attracting and Matriculating Students proposes to eliminate Standard 503, its Report for the April 2, 2011 meeting noted that the "LSAT can and does provide a fair measure of first-year law school performance and correlates well with the final law school grade-point average, rank in class, and performance on bar examinations" (p.4, emphasis added).

So, why the disconnect between these two views of the LSAT?  I believe that it grows out of the difference between predicting the probable success of individuals and the overall performance of groups.  Consider the Bar passage rates of persons with the same LSAT.  The following chart shows first-time and cumulative (after multiple attempts) Bar passage rates of participants in the LSAC National Longitudinal Bar Passage Study: 
Both first-time and cumulative Bar passage rates of persons with the same LSAT fell as LSAT scores fell.  (Note:  the Bar Passage Study focused on the entering classes of 1991, when the LSAT used a scale of 10-48, rather than the current 120-180 scale.)

How about the other measure used in law-school admissions, the undergraduate GPA ("UGPA")?  The next chart shows first-time and cumulative Bar passage rates of students with the same UGPA, rounded to the nearest tenth of a point:

 
While Bar passage rates generally fell as UGPA fell, they were not as sensitive to changes in UGPA as they were to changes in LSAT.  Over the range of LSAT scores, cumulative Bar passage rates changed from about 20% to about 100%.  Over the range of UGPA scores, cumulative Bar Passage rates changed from about 67% to about 100%, and were relatively flat between 2.0 and 2.5.

 So, even for law-school graduates, the LSAT is a better measure of the overall risk than is the UGPA. Still, admissions committees consider applicants individually, where the LSAT is less predictive of future success.  Admissions committees can, and do, try to identify applicants that will perform better than their academic credentials might indicate.  But, in crafting an entering class, admissions committees also have to take into account the overall LSAT profile of the class, especially given other parts of the Standards.  But more about that later.

 

Comments

[Howard Wasserman]

Plus, I have heard of studies that suggest bar passage is influence by how a student does in law school compared with a student at a different school with the same indicators. So take two students with the same LSAT: A goes to a higher-ranked school and finishes in the bottom quartile, B goes to a lower-ranked school and finishes in the top quartile--B is more likely to pass the bar. It seems to me that finding furthers the individual/group distinction you are suggesting here.

[Joe]

Is this really so surprising? Most states have a multiple choice section that accounts for 40%+ of the final bar result. Some states don't even have a minimum score for the non-multiple choice portions, which means that test takers can really make up a lot of ground if they knock the multiple choice section out of the park.

So at the end of the day, doesn't this really just amount to "people whi do well on standardized multiple choice tests do well three years later on standardized multiple choice tests"?

[TJ]

Joe, it may be as you say, that the correlation arises from better abilities to take standardized tests and not something else like "analytical ability" (however defined). But even then the correlation is important, because passing the bar is important in and of itself to the law school and to the student.