Conison guest post: Black Boxes and Halos

The following is a guest post by Dean Jay Conison of Charlotte School of Law.

We all want simplicity. The world is complex and we cannot deal with complexity all the time. We instinctively use heuristics to get through day-to-day life. Simplifying our thinking enables us to tell stories and frame theories. And as Daniel Kahneman noted (Thinking Fast and Slow, p. 201), “it is easier to construct a coherent story when you know very little.”

But not all simplifications are innate and not all are benign. We have a sorry history of replacing the complexity of an individual person by a single characteristic (Black, Gay, Woman, and so forth) and acting as if that is all we need to know. One of our great advances in recent times is in getting past this harmful form of oversimplification—not because such characteristics are irrelevant to anything at all, but because they can be used to hide rich and complex stories of persons or groups under a distorting cloak.

Simplification abounds in the world of legal education. We are living through a radical transformation brought about by a network of severe stresses and challenges. These include realignments in the legal services market, an increasingly anti-regulatory culture, a dramatic shift in the nature of preparation of individuals for law school, and much more. It is all very complicated and reduction to a simpler narrative is tempting. In the early days of this transformation, the main oversimplification was the search for perpetrators whom one could blame: Selfish faculty! Corporate mindset deans! A nitpicking and craven accreditor! And so forth.

By now, we are largely beyond this reductive obsession with blame. Yet, there are two broad oversimplifications that, as concerns law schools and legal education, get in the way of understanding, problem solving, and sometimes reasoned discourse.

One is the black-box view of law schools. By this I mean the tendency to see a law school as a machine for converting certain kinds of inputs (mainly LSAT scores) into certain kinds of output (mainly career and bar outcomes). On this view, a few inputs and outputs are all one needs to know about a school; specifically, to judge whether it is a “good” school or a “bad” school, and just how good or bad it is. What goes on inside the box is not a factor and is not of interest.

The poster child for black-box thinking is U.S. News. It notoriously judges law schools mainly on a limited number of inputs and outputs, and ignores what goes on within the school, except— bizarrely—how many dollars it spends. But U.S. News is not alone. A goodly number of bloggers and writers look only at inputs and outputs to draw sweeping conclusions about value and motive, usually negative (Bad school! Shut it down!). On this way of thinking, the character and quality of the education in the box is immaterial.

One problem with black box thinking is that it assumes the input and output data are good measures of something important. But it does not take much to see that the few measures commonly used are slippery and do not necessarily have an unambiguous meaning. Take LSAT profile. In an earlier post, I noted two factors (transfers in and out, and conditional admit programs) that suggest incoming LSAT profile does not have as clear a meaning as we like to think, and that we must take care in making inter-school comparisons. Here are three additional factors: (a) LSAT distribution is affected by incoming student diversity. Different groups have different LSAT distributions and so schools similarly situated can have different LSAT distributions because of different levels of student diversity. (b) Although LSAT is correlated with first-year academic performance, the strength of the correlation varies from school to school. Thus, two students, each with LSAT score of X, may have different likelihood of certain academic or performance outcomes, depending on the school they attend. (c) There are firms in the business of delivering increases in the LSAT scores of applicants. My understanding is that these increases can be substantial. But, to my knowledge, there is no data indicating whether these substantial increases also increase academic outcomes for the students—that is to say, whether the increase really increases the chance of success or just makes the student a better LSAT taker.

Similar points can be made about outcomes such as placement data and bar passage. Yet, even if we could have meaningful and unequivocal measures of inputs and outputs, a deeper problem would remain: that black-box thinking ignores most of what law school is about. Law school is all about educating students and transforming them into professionals. A law school is a complex educational enterprise delivering a wide range of educational services. Different schools have different strengths and weaknesses. Some are very strong in clinical education, others very strong in public policy. Some position themselves to provide opportunity; others position themselves to develop large firm lawyers. Some schools put much effort into developing professionalism and other non-academic competencies; others emphasize building scholarly competencies and future law professors. Understanding what law schools do inside the box is critical to understanding and evaluating them. That is why ABA site inspections involve not only mounds of paper, but two to three days of comprehensive on-site examination, to enable a close and careful look at what law schools do. We can never fully understand and assess a school if we restrict our attention just to a few imperfect measures of what goes in and what comes out.

Black-box thinking is given aid and comfort by another type of oversimplification, a cognitive bias, known as the halo effect. The halo effect is a tendency to attribute an evaluative characteristic (such as goodness or badness) to all aspects of a person or organization, based on a judgment or impression about one salient characteristic. This effect underlies first impressions about people (I like X, so he must be intelligent, so I will hire him) and the tendency to think businesses that are currently successful must be doing everything right (the premise of so many business books sold in airport bookstores).

The halo effect easily links with black-box thinking about law schools. For example, incoming LSAT profile strongly affects judgments about the school: high profile means good school, and thus means good program, good faculty, good teaching, good other things inside the box. But the halo effect in legal education goes beyond LSAT profile and other elements of the black-box model. For example, schools actively seek to build halos by hiring high-visibility faculty, on the quite logical theory that visibly good faculty translate into an overall good impression of the school, and through the halo effect support the conclusion that everything else about the school is good as well. Conversely, certain characteristics create a negative halo. A century ago, such a characteristic was being a part-time evening school, particularly one that served immigrant populations. Today, a negative halo is created by a law school’s taxable status: a school that is taxed is not liked within the law school world. Just as with other dislikes, this one triggers an assumption that anything else about the school must be bad, and that that is all one needs to know.

It is a little surprising to see black boxes and halos so widely used in discussions of law schools and legal education. As lawyers and educators, we are trained to ask questions, to appreciate the complexity of situations, to be able to see the multiplicity of perspectives, and to appreciate that there are few simple answers to hard questions. Black boxes and haloes are attractive because they allow us to tell a simple story with good guys and villains. But they impede our understanding and prevent us from seriously addressing the hard questions about law schools and legal education.

Law schools today do a wealth of different things and will likely do even more and different things in the years to come. Each school, in its own way, is dealing with stresses and changes; is trying to adapt and innovate; is trying to serve students and society in ways consonant with mission, strengths, and vision. There is an enormous amount we can learn about law schools, and an enormous amount we learn from the their experiments and experiences. Black box thinking and halos are comforting, but they disserve us by wishing away so much of what we very much need to know.

Comments

[No, breh]

"In other words it has been true for many years that only half of all JDs practice law."

"it does undermine the claim that law schools have been over-producing licensed lawyers."

I'm guessing we won't agree on the second point, and, based on the first point, I suspect that's because we define "over-produce" quite differently.

[Barry]

Posted by: Just saying...

"Interesting that the Dean discounts quantitative measures. I know that Infilaw is big into Six Sigma, or at least was. Some of the deans at other Infilaw schools were applying it to their operations. I would assume that was a corporate mandate."

Stages of bullsh*tting. First, fake the numbers (e.g., count everybody for percentage employed, but only the good jobs for salaries).

Second, call the people pointing out the fraud liars.

Third, invent jobs ('JD Advantage').

Fourth, claim that people want those 'JD Advantage' jobs.

Fifth, scream at the people pointing out the lies and fraud.

Sixth, disparage the whole idea of numbers.

Seventh - recycle as needed.

[John Thompson]

@Anon/6:19 p.m.:

"Even if one does not accept the conclusions about the value of a JD demonstrated by Simkovic and McIntyre (whose research has been vetted by many scholars no doubt more qualified than "no,breh"), it seems a big stretch to argue that law schools have been fooling people for decades about where JDs end up working."

Before the Internet, I'm guessing that it was harder to find a group of underemployed JDs willing to talk about their experiences in sufficient detail to learn that one's own career disappointments were not unique. It also probably helped that law school used to be affordable enough that the resulting debt could be serviced by other kinds of white-collar work while allowing a typical middle-class American lifestyle. People just assumed that their failures were theirs alone, and quietly moved on with their lives.

Law schools were positioned to observe all this, with desperate graduates coming back to the career development offices to beg for help finding legal work months after graduation. They knew that one's chances of first being hired as an attorney got worse over time. They might not have done any investigating into the lives of their less successful graduates, but they would have known that a surprising number of JDs were struggling to enter the profession. None of that made the brochures or law school rankings guides that were most people's introduction to the life of a law student before the Internet.

[Barry]

Anon: "..., there have been more people working as lawyers every year for more than the last decade at least and earning more income every year even in 2008."

That doesn't mean much at all, and this has been covered again and again and again on the 'scamblogs'.

[Anon]

It may not mean much to you but it explains why law schools will not shut down because they are not over producing licensed lawyers.

[Nathan A]

"It may not mean much to you but it explains why law schools will not shut down because they are not over producing licensed lawyers."

This is pretty ridiculous. Businesses shutdown when their revenue streams dry up. Even if EVERY law graduate in America ended up unemployed, law schools would remain open so long they could find people willing to pay $250k over three years. Law schools won't shutdown for the same reason that for-profit scams like Kaplan College and Bridgepoint Education won't shutdown - a surplus of unsuspecting victims with easy access to loan money.

[Barry]

Anon: " (Again, please demonstrate how the administrators and faculty are paid less at "non profit" law schools before drawing these negative inferences.)"

Bullsh*t. The thing which people are complaining about is that the tuition is far, far too high for the outcomes.

Nobody here is complaining about Harvard's tuition.

[Anon]

Nathan you display an astonishing naivete about the business of higher education. Many universities are stepping up and supporting the law schools because they know based on long term data and the current ongoing recovery that the law schools will be profitable again. In fact many of them probably are already thanks to the over-reaction of undergrads who have been moving away from applying in larger numbers. That last trend will of course begin to reverse in the next year or two. And then the university investments will have paid off handsomely. The only schools really at serious risk are the stand-alones and the for profits.

[Barry]

Anon's got to be a law professor.

The lack of logic, blithe assertion of non-facts, inability to do math,...

[Just saying...]

Anon: Many of the mid and lower tiered law schools that still receive "support" from their universities have had to cut to the bone. The support is minimal. The universities support them, in large part, because no U. wants to be the first to close its law school.

[AProf]

"The only schools really at serious risk are the stand-alones and the for profits."

Anon, tell that to the dental schools that closed, some of them highly-ranked and connected to prestigious universities.

[MacK]

Anon-

The reality is that no university wants to be first to close its law school, but several will settle for fifth, tenth and fifteenth. Once the first few schools close, the stigma will shrink

[Nathan A]

Anon @ 11:07 AM.

Not naive. I just remember what happened to dental schools in the 1980s. I'm not the only person who has been thinking about the demise of dental schools - (http://lawprofessors.typepad.com/law_deans/2013/11/a-lesson-from-a-different-profession.html).

I also find it abominable that any university would subsidize its law school. Law professors are paid more than professors in science/engineering/social science even though professors in science/engineering/social science have to (a) find outside support for their research, and (b) have their work subjected to a higher standard (peer review).

The higher salaries for law professors were fine when law school tuition dollars subsidized "legal scholarship." Since that is no longer the case its time for a pay cut or for law professors to demonstrate how valuable their scholarship is by bringing in grant money.