The Overlooked Bar Passage Denominator

    As I discussed in an earlier post, to understand bar passage problems requires more than a consideration of the preparation of the applicants for admission. If the exam becomes more difficult or its grading becomes stricter, the bar passage rate will decrease even if the quality of the applicants stays the same. Even David Frakt acknowledges this. See Some Thoughts on the California Bar Exam. Between the two of us, we have raised concern that both Florida and California have done exactly that—made the bar examination more difficult.

    The purpose of this post is to add at least one additional state to this list: Colorado. Professor Scott Johns has posted an article on SSRN that compares the results students from his school, the University of Denver, have seen on the Colorado bar exam over the last years. See Scott Johns, Testing the Testers: The National Conference of Bar Examiners' LSAT Claim and a Roller Coaster Bar Exam Ride. After a particularly strong presentation about the statistical methodology he used in his study, he showed that the results on the Colorado bar demonstrate that it has become harder. The basic fact established in his study is that the students entering the University of Denver have become stronger on the objective measurements (particularly the LSAT) over the last years, but their bar passage rates have nevertheless declined. As he states:

        As a law school that serves in a U[niform] B[ar] E[xamination] jurisdiction, we have a unique opportunity to evaluate the N[ational] C[onfrence of] B[ar] E[xaminers]’s primary claim that recent lower first-time bar passage rates are the consequence of declines in LSAT scores in contrast to more successful bar passage years. Based on empirical evidence, we do not find statistical support for the NCBE’s claim with respect to LSAT scores because those scores have remained relatively consistent (or slightly better) since 2008 (and since advent of the UBE in Colorado in 2012), and, yet, first-time bar passage rates have fallen for our law school. Simply put, the NCBE’s claim is unfounded, based on our empirical evidence.

Id. at 1.

    The fact that Colorado is a Uniform Bar Examination state is significant. It is not an unfair assumption that the declining success among applicants taking the UBE around the country (now, half of the states plus D.C. use the UBE) has a source that is in addition to changes in the student population being admitted to U.S. law schools. Some of the declining percentage of success on bar exams must be attributed to the denominator changing as bar exams have become more difficult to pass.

Comments

[Captain Hruska Carswell, Continuance King]

Unfortunately, it is rather simple. Bar passage rates are down because the profession is no longer attracting the best and the brightest. Who wants to join a profession when there are billboards shilling Traffic Ticket Defense for $49.00? "Don't Pay that Ticket." Or when there are legions of currently underemployed and unemployed attorneys struggling along with huge amounts of non-dischargeable student loan debt. It's not a secret.

[David Frakt]

One problem with trying to assess whether the bar exam has gotten harder or whether the students are less capable, is that we don't have enough data to compare classes from law schools. It might seem like a school had admitted identical classes because the LSAT and UGPA at the 25/50/75 look the same, but that doesn't mean that the two groups are comparable. The top 24% might have gotten much weaker, and the bottom 24% might have gotten much weaker as well. Even the middle two quartiles might be weaker. For example, a school could have a median of 150 and a 75th of 155, but all the students from the 50th to the 74th percentile could have a 150. Or the 25th could be 145 and all the students from the 25th to the 49th percentile could have a 145.

That is why I advocate that law schools should release much more granular data that would allow meaningful comparisons between schools and between graduating classes at the same school to see if a school is improving or declining.

[anon]

" both Florida and California have done exactly that—made the bar examination more difficult."

at least with respect to California, this statement is simply false. The recent low pass rate replicates the pass rate in the mid 80s. Low pass rates don't mean that the bar is "harder" ... this is an incredibly simplistic conclusion.

Broken link to avoid scam filter

paste

abovethelaw[dot]com/2016/11/

(replace dot with period)

with this attached

californias-bar-exam-passage-rate-reaches-32-year-low/

[Derek Muller]

It continues to surprise me that there remain unsubstantiated claims that the California, or Florida, or any state's bar is "harder" than previous years (absent an express change in the passing score or a change in the scoring method, which state bars publicize). The passing score in California continues to be 1440. The MBE continues to be equated and scaled. The essays continue to be scaled to the MBE. Nothing has changed. Unless some audit of the NCBE discloses a systemic error that has manifested itself in the last few years and continues to push pass rates down, these claims are, I think, without evidence.

[Ralph Clifford]

David,

If you haven't looked at Scott's paper, you might want to. He describes his data set on page 8. It includes data on 1743 first-time takers of the Colorado Bar over eight years, so it includes the kind of granular data you are talking about.

That being said, I agree with you completely that obtaining granular data is critical. How about all of the information about everyone in every state who has taken the bar over the last ten years released as a publicly accessible database? Well, it's nice to have dreams.

[Ralph Clifford]

Derek,

Yes, the MBE is equated and scaled. What about the essay questions? If a grader of the essay exam decides that an answer that would have gotten a "5" last year should only get a "4" this year, passing the bar exam has just become "harder."

[terry malloy]

it couldn't have anything to do with schools taking money from anyone who can fog a mirror, could it?

[anon]

Essay grading is normalized. The statement "If a grader of the essay exam decides that an answer that would have gotten a "5" last year should only get a "4" this year, passing the bar exam has just become "harder."" seems to ignore the actual process by which the graders are trained.

The pass rate on the California Bar Exam this past administration is exactly what it was 32 years ago. It is an improper and unsupported inference that the Bar Exam gets "harder" and then "easier" and then "harder" based on pass rates that vary from year to year.

[Derek Muller]

Ralph, that's not how scoring on the essays works. Essays are also scaled in most jurisdictions, including California and Florida. It doesn't matter if the graders decide to get stingy. It wouldn't affect the scores or the pass rates.

[Ralph Clifford]

Derek,

On what basis can the essays be scaled? It's a fairly easy job normalizing a 200 question multiple choice exam as you can reuse a sufficient number of questions on each exam that will allow you to equate the performance on the two different tests. This technique doesn't work for an essay exam as there are too few questions to normalize.

More importantly, essay questions themselves are significantly different from multiple choice. Grading multiple choice questions is without any shades of gray to consider. Essays require the grader to make an evaluation of quality. Even with training, there is no way to remove the potential of evaluation bias. A compounding factor here may be the universe from which graders are hired: practicing attorneys. To overcome some of this problem, I'm sure that most states use multiple graders on each question with some form of average grade being used for the final score, but unless the number of graders is large, a single grader's decision to drop a point can have a measurable effect on the final score (think of the scoring of figure skating at the Olympics).

[anon]

Ralph

ONe wishes so much that you would actually familiarize yourself with the way the Calfornia Bar Exam essays are actually graded.

"To overcome some of this problem, I'm sure that most states use multiple graders on each question with some form of average grade being used for the final score." You are WAY off base. YOu seem to think that the bar exam essays are scored the way that law school essays are scored.

PLEASE! Before you post, learn. Before you opine, PLEASE, get some facts. Your entire presentation about the grading of the essays on the California Bar Exam is simply and plainly wrong.

ANd, to speak of evaluation bias in this context adds insult to injury, as you clearly are working from a premise reached without evidence.

BTW, Ralph deleted a prior comment along these lines. LEt's see if the Emperor can tolerate an observation about his wardrobe.

[anon]

Hint

Ralph: Familiarize yourself with the calibration sessions, etc. The normalization process starts before the exams are even graded initially.

Your premise - that the bar exam gets "harder" and "less hard" and then "harder" based on the decision by an essay grader to give an "answer that would have gotten a "5" last year should only get a "4" this year" is risible.

Your premise - that recent bar exam results are unprecedented in recent memory is false.

Your big goal -- to show that entrance credentials, law school proliferation, faculty incompetence, etc. have less to do with rising and falling bar exam pass rates than does the subjective whim of bar essay graders or some nefarious plan to make the exam "harder" -- is just beyond belief.

You really need to read the exams (which are posted). You are making arguments here, but, in a very unconvincing way, because your arguments don't track facts.

[Twbb]

I don't know enough about the precise methodology used to equate LSAT scores to speak authoritatively, but since I believe it is equated to be about a 150 every year a testing population that steadily decreased in ability every year would still show roughly the same scores, at least on measures of central tendency.

Maintaining median GPA is easy; just admit from grade inflating poorer colleges (such as the for-profit ones that have cropped up so much in recent decades).

[Ralph Clifford]

Anon 10:01,

Yes I deleted an earlier comment because it contained only personal attacks. Rather than just stating that I am "WAY off base," why don't you provide the missing information on the differences? Educate us. (By the way, "evaluation bias" is a term of art for a significant statistical problem when you deal with subjectively-based data rather than an accusation of misconduct.)

Anon 10:13,

My "big goal" is to have a reasoned discussion about how law schools can do a better job. I have never stated that bar exams are "harder" now than they have ever been, but the year-to-year variation in the bar results is significant if it is to be used as the baseline to measure how a law school is doing.

[terry malloy]

"My "big goal" is to have a reasoned discussion about how law schools can do a better job."

A discussion of the difficulty of the bar exam without discussing the quality deterioration of the incoming classes is absurd.

Law school transparency>national>charts. it's aba data put in graphical form.

Other than that Ms. Lincoln how was the show?

[Ralph Clifford]

Terry,

I agree with you. My recent posts on the topic argue exactly that. The discussion before had been exclusively arguing that law school admission practices are the sole cause of the problem; my posts are to suggest that there are multiple sources that need to be considered together: law school admission standards, law school education practices, and bar examination procedures. If you don't include any of these, you are unlikely to understand the problem.

[anon]

"Some of the declining percentage of success on bar exams must be attributed to the denominator changing as bar exams have become more difficult to pass."

THis is your assertion, Ralph. It is for you to support it.

It is fair to say that above it is demonstrated that your comments about the MBE and the grading of essays demonstrate that you are not sufficiently familiar with the process to make any such broad assertions about "making the bar exams more difficult to pass."

In fact, you are wrong, as you implicitly concede ("I have never stated that bar exams are "harder" now than they have ever been ...").

Your bias is showing, Ralph. You don't want the bar pass rate to measure law school adherence to standards ("year-to-year variation in the bar results [should be understood to be a function of more difficult to pass bar exams] if it is to be used as the baseline to measure how a law school is doing.")

But, the ABA has standards, Ralph, based on the long-standing, even more valid conclusion today, that certain law schools are admitting persons unlikely to pass the bar, who fail to pass in a sufficient number to call into question the ethics of allowing such an institution to persist with the ABA's accreditation.

Schools that fail to prepare grads for the profession they have studied to join have been subjected to standards for a long, long time. Law schools should be no different.