Conison guest post: Black Boxes and Halos

The following is a guest post by Dean Jay Conison of Charlotte School of Law.

We all want simplicity. The world is complex and we cannot deal with complexity all the time. We instinctively use heuristics to get through day-to-day life. Simplifying our thinking enables us to tell stories and frame theories. And as Daniel Kahneman noted (Thinking Fast and Slow, p. 201), “it is easier to construct a coherent story when you know very little.”

But not all simplifications are innate and not all are benign. We have a sorry history of replacing the complexity of an individual person by a single characteristic (Black, Gay, Woman, and so forth) and acting as if that is all we need to know. One of our great advances in recent times is in getting past this harmful form of oversimplification—not because such characteristics are irrelevant to anything at all, but because they can be used to hide rich and complex stories of persons or groups under a distorting cloak.

Simplification abounds in the world of legal education. We are living through a radical transformation brought about by a network of severe stresses and challenges. These include realignments in the legal services market, an increasingly anti-regulatory culture, a dramatic shift in the nature of preparation of individuals for law school, and much more. It is all very complicated and reduction to a simpler narrative is tempting. In the early days of this transformation, the main oversimplification was the search for perpetrators whom one could blame: Selfish faculty! Corporate mindset deans! A nitpicking and craven accreditor! And so forth.

By now, we are largely beyond this reductive obsession with blame. Yet, there are two broad oversimplifications that, as concerns law schools and legal education, get in the way of understanding, problem solving, and sometimes reasoned discourse.

One is the black-box view of law schools. By this I mean the tendency to see a law school as a machine for converting certain kinds of inputs (mainly LSAT scores) into certain kinds of output (mainly career and bar outcomes). On this view, a few inputs and outputs are all one needs to know about a school; specifically, to judge whether it is a “good” school or a “bad” school, and just how good or bad it is. What goes on inside the box is not a factor and is not of interest.

The poster child for black-box thinking is U.S. News. It notoriously judges law schools mainly on a limited number of inputs and outputs, and ignores what goes on within the school, except— bizarrely—how many dollars it spends. But U.S. News is not alone. A goodly number of bloggers and writers look only at inputs and outputs to draw sweeping conclusions about value and motive, usually negative (Bad school! Shut it down!). On this way of thinking, the character and quality of the education in the box is immaterial.

One problem with black box thinking is that it assumes the input and output data are good measures of something important. But it does not take much to see that the few measures commonly used are slippery and do not necessarily have an unambiguous meaning. Take LSAT profile. In an earlier post, I noted two factors (transfers in and out, and conditional admit programs) that suggest incoming LSAT profile does not have as clear a meaning as we like to think, and that we must take care in making inter-school comparisons. Here are three additional factors: (a) LSAT distribution is affected by incoming student diversity. Different groups have different LSAT distributions and so schools similarly situated can have different LSAT distributions because of different levels of student diversity. (b) Although LSAT is correlated with first-year academic performance, the strength of the correlation varies from school to school. Thus, two students, each with LSAT score of X, may have different likelihood of certain academic or performance outcomes, depending on the school they attend. (c) There are firms in the business of delivering increases in the LSAT scores of applicants. My understanding is that these increases can be substantial. But, to my knowledge, there is no data indicating whether these substantial increases also increase academic outcomes for the students—that is to say, whether the increase really increases the chance of success or just makes the student a better LSAT taker.

Similar points can be made about outcomes such as placement data and bar passage. Yet, even if we could have meaningful and unequivocal measures of inputs and outputs, a deeper problem would remain: that black-box thinking ignores most of what law school is about. Law school is all about educating students and transforming them into professionals. A law school is a complex educational enterprise delivering a wide range of educational services. Different schools have different strengths and weaknesses. Some are very strong in clinical education, others very strong in public policy. Some position themselves to provide opportunity; others position themselves to develop large firm lawyers. Some schools put much effort into developing professionalism and other non-academic competencies; others emphasize building scholarly competencies and future law professors. Understanding what law schools do inside the box is critical to understanding and evaluating them. That is why ABA site inspections involve not only mounds of paper, but two to three days of comprehensive on-site examination, to enable a close and careful look at what law schools do. We can never fully understand and assess a school if we restrict our attention just to a few imperfect measures of what goes in and what comes out.

Black-box thinking is given aid and comfort by another type of oversimplification, a cognitive bias, known as the halo effect. The halo effect is a tendency to attribute an evaluative characteristic (such as goodness or badness) to all aspects of a person or organization, based on a judgment or impression about one salient characteristic. This effect underlies first impressions about people (I like X, so he must be intelligent, so I will hire him) and the tendency to think businesses that are currently successful must be doing everything right (the premise of so many business books sold in airport bookstores).

The halo effect easily links with black-box thinking about law schools. For example, incoming LSAT profile strongly affects judgments about the school: high profile means good school, and thus means good program, good faculty, good teaching, good other things inside the box. But the halo effect in legal education goes beyond LSAT profile and other elements of the black-box model. For example, schools actively seek to build halos by hiring high-visibility faculty, on the quite logical theory that visibly good faculty translate into an overall good impression of the school, and through the halo effect support the conclusion that everything else about the school is good as well. Conversely, certain characteristics create a negative halo. A century ago, such a characteristic was being a part-time evening school, particularly one that served immigrant populations. Today, a negative halo is created by a law school’s taxable status: a school that is taxed is not liked within the law school world. Just as with other dislikes, this one triggers an assumption that anything else about the school must be bad, and that that is all one needs to know.

It is a little surprising to see black boxes and halos so widely used in discussions of law schools and legal education. As lawyers and educators, we are trained to ask questions, to appreciate the complexity of situations, to be able to see the multiplicity of perspectives, and to appreciate that there are few simple answers to hard questions. Black boxes and haloes are attractive because they allow us to tell a simple story with good guys and villains. But they impede our understanding and prevent us from seriously addressing the hard questions about law schools and legal education.

Law schools today do a wealth of different things and will likely do even more and different things in the years to come. Each school, in its own way, is dealing with stresses and changes; is trying to adapt and innovate; is trying to serve students and society in ways consonant with mission, strengths, and vision. There is an enormous amount we can learn about law schools, and an enormous amount we learn from the their experiments and experiences. Black box thinking and halos are comforting, but they disserve us by wishing away so much of what we very much need to know.

Comments

[Nathan A]

"There is an enormous amount we can learn about law schools, and an enormous amount we learn from the their experiments and experiences."

Care to offer us any data on Charlotte's experience with educating entering classes with record low LSAT scores? Maybe you guys are getting sub 145s to pass the test a rate significantly better than your peers. I'd love to know.

Until then, let's be adults and recognize that Infilaw is in the business of generating money for vulture capitalists. That's the mission. The end.

[No, breh.]

Principal: Mr. Madison, the Industrial Revolution changed the face of the modern novel forever. Discuss, citing specific examples.

[Billy clears his throat several times]

Billy Madison: Uh... Okay. The Industrial Revolution to me is just like a story I know called "The Puppy Who Lost His Way." The world was changing, and the puppy was getting... bigger.

[Later]

Billy Madison: So, you see, the puppy was like industry. In that, they were both lost in the woods. And nobody, especially the little boy - "society" - knew where to find 'em. Except that the puppy was a dog. But the industry, my friends, that was a revolution.

[Long pause]

Billy Madison: Knibb High football rules!

[the crowd erupts into cheers]

Principal: Mr. Madison, what you have just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.

Billy Madison: Okay, a simple "wrong" would've done just fine.

[anon]

This reminds me of "Dear Officer Krupke" ...

This is the cogent proof that the legal academy, writ large, is subject to the most bizarre self delusions, based on a proven ability to spin and spin and spin until nothing is true and nothing is false.

It is all so complex ... isn't it? Unfathomable, really.

Thus, we don't really need to focus on "certain kinds of output (mainly career and bar outcomes)."

EXACTLY! Why, what a brilliant piece! For the vast majority of students, especially at schools such as the one in question, the goal is to become an attorney and practice law.

Why focus on such narrow criteria, like career and bar outcomes? Please ... the bar review courses inside the box are much more important than the bar pass rate, and the illusion of becoming an attorney is far more important than actually practicing law.

But, one need be as brilliant as the author to appreciate these nuances.

[Kevin Jon Heller]

I agree with Dean Conison. Surely the students at Charlotte Law School aren't paying $40,000 per year to pass the bar and get jobs. It's about becoming educated and being transformed into the professionals they would need to be if they could pass the bar and get jobs.

[MacK]

"The world is complex and we cannot deal with complexity all the time."

Funny practicing law we deal with complexity all the time, but know how to reduce it to simple questions. Simp,e questions have been put to you Dean Connison - what proportion of the students admitted to your institution with poor GPAs and weak LSAT scores via your special admissions program pass the bar? The question has been put to you by many observers. Frankly, this post is mostly fell-good nonsense that never addresses the basic issue. Why not?

It is quite simply a pile of silly catchphrases, management marketing speak jargon which goes no where, a chain of clichés: "we all want simplify" "the world is complex" "heuristic"" "Black, Gay, Woman" "halo effect" "rich and complex stories" "realignments" "narratives" "black-box thinking" "negative haloes" But of course "VISION."

If I asked a computer program to come out with a trite contentless heap of piffle - it would read quite like what you just wrote. If a witness, an expert witness, vomited up a stream of contentless verbiage in any court of tribunal, he'd be luck not to be held in contempt.

Where is the content in this post - there is no there there - no facet, not substance, just a pile of clichés, I'm embarrassed for you.

[Barry]

Seconding No, breh, here. Conason's post was pretty close to 100% bull sh*t; the only part which wasn't was the straw man.

[confused by your post]

Your post is what passes for sophistry these days Dean Conison? It's a sad day for snake oil salesmen everywhere.

The best written defense you can come up with for (in my opinion) the terrible and harmful practices of Infilaw is this? It's complicated is your defense?

Why even post something like this? People here who don't know anything about Infilaw and what it is doing aren't going to read your post and come away thinking you're honestly or effectively addressing critics. They'll just think you're being dishonest.

[anon]

" black-box thinking ignores most of what law school is about. Law school is all about educating students and transforming them into professionals."

" outcomes such as placement data and bar passage ... are slippery and do not necessarily have an unambiguous meaning."

That persuasive, isn't it? Ignore the failure rate on the bar (such that persons cannot practice law) and the fact that graduates can't find JD related work, because these measures are not "unambiguous."

The "profit" issue in my view is irrelevant (is there a big salary differential at the "non profits"?) What I want to know is: what makes a good law school dean?

It seems to me that, although trashing this post for what it is (bs), most of the faculty reading this website support the role of dean as bs'er in chief, and support a version of this claptrap in their own schools to justify every policy, every action, every effort to obfuscate and hide negative information, and every self interested effort to feather their own nests and justify their own positions.

That makes faculty jumping all over "Infilaw" as hypocritical as they can be.

[AnonProf]

This post is just pathetic...

[terry malloy]

Your law school is a black box that entices rubes to hand you money that they loaned from the government.

This tap dance is the best defense you can muster?

Your schools are no better than the sham medical training programs that prey on the working class.

It is shameful how you earn your living.

[unirritant]

Ironically The Charlotte School of Law just moved from it's "campus" http://www.freemorewest.com/assets/img/uploads/locator/Charlotte%20School%20of%20Law1.jpg to an actual black box in downtown Charlotte.
http://www.emporis.com/images/show/392902-Large.jpg A move that has, at some level, at least a poetic sort of justice I suppose. Though I'm sure I speak for all of us when I say that for true poetry nothing beats "contentless heap of piffle".

[Sam Browning]

In a court room when a witness on the stand so clearly avoids answering the question that was posed, one of the standard responses is as follows.

Your honor, will you please order the witness to answer for the jury's benefit, the question that was asked rather the question that he wish I had asked.*

*hat tip, to Attorney Kimbell Hunt of Connecticut

[BoredJD]

This is a great post. In fact, it's such a great post that I'm sure you'd be willing to tell every prospective student at Charlotte that the school cannot be judged on such "simplistic metrics" as bar passage rates, job rates, starting salaries, or "percentage of graduates who have enough to eat."

But you can take this one step farther. Head over to Infilaw HQ. Tell the Board that despite everything they learned in business school, for-profit educational institutions can't accurately be judged by simplistic metrics like "number of enrollees" "revenues" "expenditures" "profit margins" or "return on investment." That these broad oversimplifications are not really measures of anything important, just comfortable heuristics that business owners use in light of being able to seriously address the hard questions about their businesses.

Do that, and you might find you and Professor Frakt have at least one thing in common.

[MacK]

I'm just waiting for Brian Leiter, Steve Diamond or one of that crew to pop up, explain the the Faculty Lounge has been hacked and that Paul Campos, BoredJD, Dybbuk and I actually wrote this parody-post in an effort to discredit Dean Conison - and that we have been forging legal letters to various sundry philosophers too....

[Sam Browning]

Bored JD, does this mean that Charlotte has now entered the ranks of the post modern law schools? Will there be a Foucault professorship?

[thirtyyearcharlottelawyer]

Is this guy really trying to convince us that the Charlotte School of Law produces graduates ready to...what? Practice law? What a joke. I'm in the trenches in this town and I promise you that very few CSL graduates show promise. The vast majority, 95%, shouldn't be lawyers. They don't even understand the basics - like showing up on time dressed in courtroom attire.

I'm embarrassed for the young man in a business suit with no socks on who shows up 30 minutes late with his I-phone in his hand wondering why the judge is pissed. Or the young lady in rubber flip flops. Or the guy who files a lawsuit at the last minute and then serves the defendant by first class mail, but can't afford malpractice insurance. Or the woman who insists to the point of hostility to an exasperated judge that she doesn't need a supporting affidavit in her Motion For Summary Judgment - in a lawsuit she filed against her client who is refusing to pay her because she's incompetent. I could go on.

One other thing - in addition to ruining the futures of these student victims, CSL is also destroying the small law firm business model, at least in Charlotte. Even if only half the 400 graduates a year pass the bar, nobody's hiring these people so they have no choice but to hang out a shingle and hope for the best. Our local legal directory is full of strange, ever-changing names with residence addresses or P.O. Box addresses and cellphone numbers only.

They've saturated this market to such a degree, and provide (shoddy) legal services for next to nothing, that the rest of the local Bar is feeling the pinch. Spare me your retorts about healthy competition. Can I handle a traffic ticket for $50? Not if I want to pay my staff. Is the minority or immigrant community served by this influx of lawyers who work for next to nothing? Not if they don't know what they're doing. State Bar grievance filings? Through the roof.

Graduates forever buried in debt. Clients with incompetent and uninsured lawyers. Experienced lawyers and firms with ever-increasing revenue shortfalls. Exasperated judges and court personnel. Overworked State Bar investigators. Everyone loses except Jay Conison and the vulture capitalists at Infilaw's parent hedge fund.

I agree with Paul Campos. If it can't go on, it won't go on. Conison's gibberish nothwithstanding, the real question is how much damage Infilaw will do before he and they leave town.

[Jojo]

What can I say about this post that hasn't already been said about North Korea? It's irrelevant, paranoid, defensive, deceptive, and ego-maniacal.

You are the dean of a professional school that teaches students who already have graduated from a four year college. You aren't expanding their minds or teaching critical thought. That was done - if at all - before they arrived. They're borrowing a total of 4 years' worth of per capita GDP so that they can enter a profession. If you can't get them over the modest hurdle of the bar exam, your law school's reason d'être does not exist.

[scanlon]

Hard to follow "thirtyyearcharlottelawyer"; would love to see Dean Conison's response to that, but I suspect we won't.

Anyway:

Dean Conison,
You are correct in that black box, quantitatively-focused analysis should not be the only way to look at an institution. The problem is your law school, and Infilaw as a whole, also fail when you apply qualitative, more nuanced analyses at any level.

In my opinion, Infilaw is a hedge fund creation designed with the primary goal of exploiting a poorly-designed federal loan system that provides little oversight. Its primary goal is profit; it will charge as much as it can for as long as it can then cash out if necessary. The lower-performing students are admitted because the federal system does not discriminate between lower-performing students and higher-performing, and they are more likely to purchase your product.

[anon]

After reading these comments, it is clear that the points Dean Conison attempted to explain have been met with, shall we say, less than glowing reviews.

Given the uniformity of the condemnation, one suspects the Dean will simply conclude that he is the victim of disgruntled "scam bloggers" who have no other purpose other than an agenda to "shut them all down." Or, he will at least conclude, rightly perhaps, that an unfair prejudice exists against "for profit" law schools (Again, please demonstrate how the administrators and faculty are paid less at "non profit" law schools before drawing these negative inferences.)

To the Dean I would offer only this: until and unless you demonstrate that your admissions policies, at each level of admission criteria (LSAT and GPA), are producing graduates who are more likely than not to pass the bar and obtain JD required employment, no one will hear anything else you have to say.

In other words, if persons admitted with an LSAT score below 145 have a slim chance of passing the bar and obtaining JD required employment, for your own sake Dean, admit it!

Either meet the challenge or, if you are wise, stop posting and subjecting yourself to this humiliation.

[antiro]

There is some merit in the point that some of what law schools do is not appreciated due to the relentless focus on student debt, bar pass rates, and employment prospects.

But when it comes to law schools, the focus FIRST should be on student debt, bar pass rates, and employment prospects. When people are entrusting three years or more of their lives, you owe them the basics.

The basics are that they graduate from your institution, can pass the bar, and obtain a job requiring your degree that allows you to pay your debt and save some at a level that your financial straits have improved due to attending said school.

If you're not meeting the basics, Mr. Conison, you can't be charging a premium. According to a quick search, your school has a yearly cost of attendance of $41,000.

If your law school had good job prospects, good bar pass rates, and were giving students careers that were paying them well enough to manage the debt, save, and become contributing consumers, then we could seriously discuss what goes on in the "black box."

But such talk about black boxes when your graduates are faring so poorly is premature.