Heat vs. Light II: It Matters Who Counts and Who Cares

This is the second in what I intend to be a short series of posts on the importance and limitations of Mike Simkovic’s and Frank McIntyre’s recent contributions to the literature on the purpose and value of a legal education.  (The first is here.)

I’ll devote this post to setting aside what I think is a peripheral issue that has occupied some of the resurgent debate spawned by Mike Simkovic’s series of posts on Brian Leiter’s blog.  In recent exchanges, this issue has emerged as a series of accusations and assumptions about what employment and earnings data ought generally to be presented, and whether the way such data are generally understood makes their presentation subject to misinterpretation.  A lot of the recent controversy has focused on whether, when a law school reports its graduates’ employment numbers, “employed” should or should not include part-time, short-term or non-law-related employment; and whether “unemployed” should include only those actively seeking work, excluding the category the ABA collects as “unemployed—not seeking.”  Similar disagreements persist over how to count school-funded positions, and what qualifies as law-related employment, though these are not front-and-center in the recent exchanges I’m writing about here.

Mike’s recent posts have taken the strong and categorical view that law schools, NALP and the ABA ought to report law-graduate employment the same way the U.S. government reports on employment generally, and that any other view is ignorant or misinformed.  Board of Labor Statistics and Census data (among others) report people as “employed” if they have any kind of work at all, including work that is part-time, short-term, or (in the case of law-school graduates) entirely unrelated to their legal education; and as "unemployed" only those who are actively looking for work.  The widely articulated criticism “that law schools behaved unethically or even committed fraud . . . by presenting their employment statistics in a misleading way,” says Simkovic, “comes down to this:  The law schools used the same standard method of reporting data as the U.S. Government.”  Any contrary view, he argues, is “based on an incorrect belief that law school only benefits the subset of graduates who practice law . . . ,” a point he not infrequently expresses as an impatience for those who would be influenced by near-term results resembling the ones they might desire for themselves rather than more dispassionately considering lifelong outcomes of the JD population as a whole to assess their prospects.

Not so fast.  It matters who counts, and who cares.  As for who counts, there is a very significant issue in equating the accuracy and reliability of statistics gathered and disseminated by disinterested government employees with those gathered and disseminated by self-reporting law schools, especially in an environment in which both rankings and prospective law student decisions essential to the welfare of the reporting school are often based on what it reports.   I have no doubt that many schools are meticulous and unstintingly honest in their self-reporting.  But we already have several documented incidents of motivated or flatly dishonest reporting, and my regretful suspicion is that such discrepancies will prove to be more common than anyone would hope as the ABA starts auditing.

As for who cares, my modest proposal is that people should get the information they want in the terms they are most likely to understand.  I would guess that a libertarian economist like my friend Mike Simkovic agrees, in which case his arguments on these issues don’t square with our apparently shared principles in at least two important respects. 

First, with respect to the proposition that people should get information in the form they are most likely to understand:  It is demeaning and more importantly inaccurate to argue that everyone understands—or any “reasonable” person should—that law schools reported employment statistics the same way the government does.  In fact many if not most people relying on US News rankings and law-school statistics (among others, tens of thousands of prospective law students and their families each year) didn’t understand that at all, which is why the ABA stepped in to change the reporting requirements in 2010.  I personally was stunned when I first learned that law schools’ “employment” rates reported before 2011 included part-time, short-term and non-law-related work.  I am not, as I have previously cautioned all of you, a social scientist, but I daresay I’m no more stupid or ignorant than the next guy.  Countless others have expressed the same surprise, and even greater surprise at the discrepancies between “employment” as reported before 2011 and the more granular information reported afterwards.  We needed more detailed and explicit reporting for the same reason that just about every state has consumer-protection legislation governing the form and content of disclosures regarding common consumer transactions:  You take your audience as you find them. 

Second, with respect to giving people the information they want:  Simkovic and McIntyre are free to argue (as they do) that any prospective law student’s interest in the nature or duration of near-term post-graduate employment is “based on an incorrect belief that law school only benefits the subset of graduates who practice law . . . .” And they have in fact presented significant empirical evidence suggesting that it is possible for some people to overemphasize these near-term details at the expense of best-informed decisionmaking.  As I will discuss in future posts, that doesn’t mean that these conclusions should immediately be uncritically accepted as immutable laws of the universe, and it especially doesn’t mean that they are true for all of the people all of the time.  And I assume that, even if they are fully convinced of the truth and immutability of their conclusions (a conviction I suggest is premature), they are not arguing we should withhold information from people so that they can make decisions that the authors consider “better.”

Just as importantly, Simkovic and McIntyre’s studies attempt to quantify the effect of a JD degree on career earnings, and offer suggested courses of action some might choose to take if their goal is to make as much money as they can over a lifetime.  No one could claim that this is an irrelevant perspective, but it is hardly the only one we see:  While it certainly isn’t true of everyone, many people actually go to law school because they want to be lawyers.  Some crave the drama of the courtroom; some have specific social or political purposes in mind; some (and there is at least one study suggesting that this may be particularly true of some who attend lower-ranked law schools with less robust employment prospects and are among the first in their families to seek higher education) enjoy satisfaction from the prestige of being a professional.  Even the most doctrinaire neoclassical economist would not presume to argue with a consumer’s utility preferences—which is just econo-speak for there’s no accounting for taste.  For the population—and I will suggest that it is a significant one—that specifically wants to be lawyers, statistics about any old job held by JDs are not very helpful. And if you’re still thinking that this doesn’t reflect the real thoughts of real people in the real world, consult the recent exhortation posted by the dean of Cooley Law School—an institution that, with all respect, has an uninspiring recent track record in placing its graduates in lawyer jobs—proclaiming “Now Is the Time to Fulfill Your Dream of Becoming a Lawyer!”

So let’s talk about what information people might want, and let’s talk about how they ought to consider using it.  I think Mike Simkovic’s recent posts on the Leiter Blog focus predominantly on just that, and hats off to him for doing so.  Let the discussion continue. 

But I do want to close by taking issue with one of those posts, which I view as a cautionary tale about the temptations of partisanship.  On March 31, a Berkeley Law professor published an op-ed in the New York Times entitled “Law Schools and Industry Show Signs of Life, Despite Forecasts of Doom.”  The first paragraph poses the question “is now a good time to go to law school?” and the balance of the op-ed argues hell yes, focusing on (among other things) the employment prospects of recent graduates of Georgetown Law, and on the growth of “quite profitable” large law firms and in-house law departments to argue future need for “lawyers, and plenty of them.”  Kyle McEntee (of Law School Transparency) and Steven Harper (of Belly of the Beast) both took strong issue with these arguments on a number of grounds, principally that they were inexcusably sloppy and cherry-picked statistics misleadingly—for example choosing one of the top law schools’ employment outcomes for an example when scores of them have much worse results; suggesting that large firms are hiring aggressively when in fact they are hiring about 30% fewer new lawyers than they did seven years ago; and using part-time, short-term and non-law-related employment outcomes to argue that we could anticipate lots of future demand for lawyers.

One of Mike’s posts not only defended the op-ed uncritically, but did so in terms and in tone that (I hope he’ll forgive me for saying) fall well below the standards I have come to know him for.  “Mr. McEntee’s problem,” Mike wrote, “is not that The New York Times got the facts wrong.  His problem is that The New York Times got too many of the facts right.  Mr. McEntee simply dislikes the facts.”  With all respect to Mike Simkovic, that’s really quite unfair.  In my own view, McEntee and Harper both nailed it.  The op-ed author used cherry-picked statistics including non-lawyer employment and lawyer underemployment to predict (among other things) the future demand for lawyers.  That’s misleading.  And the entire thrust of the op-ed—“is now a good time to go to law school?” and can those who apply to law school now expect to work as lawyers?—is antithetical to Simkovic’s and McIntyre’s hypotheses, which are that it’s a bad idea to ask whether now is a good time to go to law school (because you can’t know, and any one time is as a matter of prediction as good as any other), and an even worse idea to worry about whether you’re going to be a lawyer when you’re done (because that could distract you from the long-term earnings enhancements that they believe JDs offer most comers).  Mike should have lambasted the editorial for citing his work while missing his point.

Next time we’ll talk more about Mike’s points, including both their valuable insights, and those insights’ potential limits.

--Bernie

 

Comments

[[M][@][c][K]]

And we know that few people read the wiki talk pages or indeed check the IP addresses of who edits the pages to discover places like Fordham, the Univeristity of Texas - followed by a change to Chicago, etc.

Seriously,shouldn't the Wikipedia page be honest? Should prospective law students have to go to the talk pages? Why would you suggest that....oh!

[anon]

FE wrote above "Let's not go down the path of looking to identify the anonymous commentators here."

I second that. Let's not name someone who prefers to use a pseudonym.

Here in the FL, many anonymous comments decry anonymous comments. Such is life.

But, let's not be "outing" people. That reminds me of a certain very vicious and inexcusable move a few years ago.

[[M][@][c][K]]

Who BDG is evidently something well known at Prawfs and I thought it was Brian Galle - but his middle initial is not on the blogger list.

Anon at 9:21 - if you are who it has been suggested you are - you may not want to raise that issue - you know why. Your prior actions have been - well let's put it this way, I don't have to announce "my wife is a lawyer" to have legal credibility....

[[M][@][c][K]]

Voodoo94

I don't think anyone should be making comparisons to the late Dan Markel. Prawfsblawg is a forum, like this one, that names those running threads as bloggers. BDG as a handle runs into the problem that Brian Galle does not use his middle initial in his profile - which made me uncertain as to whether it was him.

I do find it pretty shocking at this point in the debate, that anyone would be saying that 22 year old potential law students should be googling for underlying methodology. The caveat emptor argument is disgraceful and does reflect on any institution that it emanates from - BC applicants should know that this view is held there to the extent that it would openly asserted as a principle by Galle.

[anon]

Everyone has a "principled" reason for wanting to expose the identity of a person posting an anonymous comment with which one so passionately disagrees.

However, we have seen the damage done by this sort of self-justification. Until the FL posts a comment policy regarding anonymity, let's all resolve to respect the anonymity of those who wish to post in that fashion, and, to, as requested in this thread, address the arguments and points raised by addressing the arguments or points raised, and not by attacking the person making those arguments or points.

Some comments are clearly not serious or genuine or are designed to provoke just for the sake of provocation. It is probably ok to say so, because saying an argument is frivolous would seem in bounds.

I'm not making the rules here, but just expressing a pov on this issue.

[anon]

And, btw, if there is proof that a named author has engaged in efforts to stifle debate or discussion that is critical of that author's pov, I think it is fair to at least inquire about the circumstances and give the author an opportunity to respond.

Unfortunately, there isn't really a good way to do this in blog comments on another's thread and S posts in a way (that I've seen) that doesn't allow a direct discussion with him.

All in all, his work doesn't really prove much, and doesn't merit, IMHO, all the attention it has received. This attention is mainly because discussion supposedly about S's work has been used as a proxy for a bigger and broader debate.

The proof? Look at these threads. No one really cares to debate S's work, or blog posts, because the anodyne nature of the sort of watered down conclusions and observations of the obvious just aren't really all that interesting or important.

[[M][@][c][K]]

There is a basic issue here - BDG - is a named person. Who he is was not a secret - but it was unclear that it was Brian Galle, though it certainly looked that way. Considering the caveat emptor argument being made, and how appalled people are, I wanted to know wig fulminations were fairly directed. I would point out that BDG was highly critical of anonymous and pseudonymous commenting in his own thread, so we can assume, unless he is a hypocrit, that he expects readers to know who BDG is.

[[M][@][c][K]]

Returning to the debate at issue - how should data be presented.

I'm going to look at it from a litigation perspective - how would a judge or arbitrator react if I presented data that was in anyway misleading. I've recently seen this in action in an arbitration, where my opponent made a habit of making statements of fact, which though literally true (or could be read so as to be true) were very misleading. In effect the statements gave an impression that was contrary to reality. They got caught, again and again - and it destroyed their case, since the tribunal reached the point where it would believe nothing they said. Hilariously, they coached a witness to give one such circumlocution when answering a scripted question, which was then shown on cross to be misleading and contrary to his own witness statement (made for another purpose.)

The reality is that a judge or arbitrator would hand me my neatly severed head if I was lucky. No good litigator I know wants to be seen by the court or tribunal to be shading the data, or to have to hunt around for a standard created by another party to be able to argue - well they do it too. If you are going to present it you make very sure that any issues with its interpretation are presented by you rather than caught by opposing counsel.

I think the fact that Simkovic had secure sponsorship from interested parties, NALP and Access is important, because it indicates the likely use that the study would be put to. It was inevitably going to be presented as fact by law school admissions departments, I.e., the marketers, as well as the deans of certain schools - and presented to an audience of 22 year old recent college graduates. To suggest that the ultimate audience for "the Million Dollar Law Degree" was only going to be labor economists and law professors is to be delusional.

Given what has transpired, the false data peddled by so many law schools, the ABA's belated efforts to clean up data reporting, there may not be a legal duty of care to someone writing an article in this space, but there is a moral one, and a fair question of professional one too. Simkovic's refusal to accept the idea that he has some obligations to potential law students is sad.

[Jojo]

M@cc,

It is crazy that law schools blend employment and unemployment data so casually and defend the data as technically true because BLS! That's such a bad argument that it wouldn't even make the brief in litigation.

America's unemployment rate is 5.5 percent. I guess I'd accept a glossy marketing pitch touting employment rates if they assume full labor force participation (unlike BLS data) and if they compared to the overall rate of the USA. "Ten months out, 93 percent of our graduates are employed! That's only 2 points worse than America overall."

[twbb]

@JoJo:
It's a little funny that the BLS is the absolute gold standard for everything, except when it projects an oversupply of lawyers, in which case it doesn't know what it's doing...

@[M][@][c][K]:
Definitely a huge credibility gap. I've seen the same sort of weird behavior in Federal court, where experienced attorneys from prestigious law firms would try to get the most misleading statements past the judge and just crash and burn because of it. As far as I'm concerned, the NALP -- which makes the incredible claim that the median salary for first year associates at firms of any size is $130,000 -- doesn't have much saved in the credibility bank.

For the sake of my blood pressure, I haven't really jumped into the details of Simkovic's argument, but now that I have I am a little surprised at some of his defenses of the NALP figures, which really highlights the credibility issues you mention. I didn't agree with his overall conclusion before, but I assumed he was trying to at least do economics.

For example, Merritt and McEntee criticized the NALP figures for (quite reasonably) having a nonresponse bias. His response is a lot of handwaving and dodgy arguments (from his April 11 response on Leiter's blog):

(I know I'm jumping into the debate late, and a lot of this stuff has been discussed, but I think my blood pressure will only get worse if I don't put my two cents in.)

1. "Nonresponse is not the same thing as nonresponse bias. Law school critics do not seem to understand this distinction. A problem only arises if the individuals who respond are systematically different from those who do not respond along the dimensions being measured."

This just seems dishonest strawmanning. The argument is clearly that response rates differ based on salary (or firm size, which functions as a proxy for salary).

2. "Law school critics claim that law school graduates who do not respond to questions about income are likely to have lower incomes than those who do respond. This claim is not consistent with the evidence."

Ok, this is just handwaving. How is it not consistent with the evidence? Response rates for large firms are higher than response rates for small firms. Large firm salaries are higher than small firms. The NALP itself tells us that.

3. "To the contrary, high-income individuals often value privacy and are reluctant to share details about their finances.***"

Meaningless handwaving and I'm starting to wonder that maybe McIntyre really doesn't want his name on these arguments, because other economists would find this stuff dodgy. "Often value privacy"? How often? Is it true for all high-income individuals or by field? How about by age? Career status? Where's your support for this claim?

4. "*** After the JD researchers noted that young law graduates working in large urban markets—presumably a relatively high-income group—were particularly reluctant to respond to the survey."

THAT'S your sole citation to support number 3 above? To "presum[e]" young law graduates working in large urban markets are a high-income group without any further evidence? You don't get to legitimately do that. A professional labor economist wouldn't offer that presumption without support, and I'm certainly not going to take it from a non-economist. In any event, hordes of temporary, low-paid document reviewers would probably disagree with you.

Furthermore, as Deborah Merritt has pointed out, law schools have access to first year salaries for the largest, best-paying firms, and a strong motivation to count as many high salaries and as few low salaries as possible. Unless I missed something, Simkovic has never addressed that.

5. "Other biases such as systematic under-reporting of incomes by highly educated individuals,*** under-reporting of bonuses and outside income, and the like should be taken into account.****

Again, handwaving. "[S]ystematic under-reporting of incomes by highly educated individuals," even if true, is irrelevant here since just about your entire study population has a JD. If "under-reporting of bonuses and outside income, and the like" should be taken into account then it's your job to do so.

6. "To the extent that these biases cut in opposite directions, they can offset each other."

And, they could not. You're the one trying to prove your hypothesis. You don't do that by recounting the rich possibilities inherent in the world.

7. "If Professor Merritt’s view is that differences between NALP, ABA, and U.S. Census Bureau data collection and reporting conventions make law school-collected data more difficult to compare to other data sources and make law school data less useful, then I am glad to see Professor Merritt coming around to a point I have made repeatedly."

Her point was that law school-collected data underestimates employment and overestimates median salaries, not some vague, overarching difficulty in comparing them. She is not supporting your argument in any way. Furthermore, if you agree that the data collections and reporting conventions introduce these difficulties, then why do you rely on this data so much? This reminds me of a contract case I was on once where we argued something to the effect that the opposing side breached the contract and made it impossible for us to perform. Their response was to characterize that argument as our admission that we "breached the contract." Speaking as a litigator, if you're going to try to reframe your opponents argument to make it seem like they support you in some way, you've got to do a better job than this.

[Barry]

twbb: "It's a little funny that the BLS is the absolute gold standard for everything, except when it projects an oversupply of lawyers, in which case it doesn't know what it's doing..."

Incorrect. Please try again.

[Barry]

"twbb: "It's a little funny that the BLS is the absolute gold standard for everything, except when it projects an oversupply of lawyers, in which case it doesn't know what it's doing..."

Incorrect. Please try again."

I apologize; I didn't read with my actual brain turned on, and thought that you meant the opposite.

[[M][@][c][K]]

twbb:

vis-à-vis "After the JD researchers noted that young law graduates working in large urban markets—presumably a relatively high-income group—were particularly reluctant to respond to the survey" and ""It's a little funny that the BLS is the absolute gold standard for everything, except when it projects an oversupply of lawyers, in which case it doesn't know what it's doing..."

A long time ago I started to correspond with Paul Campos, before he was so well known about law school reporting of incomes in the Washington DC market. One of the topics was about the discrepancy of the salaries that all of the DC law schools were claiming along with the employment rate as compared to the BLS data for average lawyer salaries in D.C.

At that time pretty well every DC area law school - so Georgetown, GW, AU, Howard, Catholic, George Mason, UVa and I think Baltimore and UMD were claiming 90%+ employment and a median of $160k, but the BLS showed a DC median of around $100k (I forget the exact number) for all private practice attorneys, including those with years of experience. The point I made was that this could not possibly make sense - how could the median for 1st year associates conceivably be higher than the median for experienced attorneys? Campos as I recalled told me that this was curiously pretty well the case in every large urban market - the law schools in those markets all claimed their median matched the reputed BigLaw salaries being paid and substantial exceeded the BLS median for all private practice lawyers and indeed all attorneys. Someone was not telling the truth.

Now consider for a moment the impact that telling your graduates that if they are not making $160k p.a., they are are well below average, when the reality is that this number is grossly exaggerated? Lawyers and law students are very competitive - the schools were telling them in effect that they were losers.

With respect to the point about federal court - it may be a little dated, but there was a patent litigator in San Jose - who had to appear frequently before the NDCa - who once was cuttingly described as "being like Joe Isuzu without the sincerity." He did bear a certain resemblance to a tall David Leisure (I've heard this again by the way in reference to Rick Perry.) It became a problem - his tendency to get caught. I tend not to be before the same court very often, but you become aware over time that credibility matters.

There used to be a joke that "a diplomat is someone sent abroad to lie for his country." My father used to answer (and often referenced BN) that a diplomat in fact "starts his career with one lie, and he guards that lie, and polishes it, and feeds it with small and large truths - and it becomes a bigger and bigger lie, every more important, and the chances are he retires with the lie unused, because it was too valuable to spend except for something very very important."

Law schools need to realise that they have done huge damage to their reputations with the games they played on employment data for the 15-20+ years up to 2010-12. There is not way to escape it - they lied to their customers, to the people who were effectively their clients. Law schools that taught ethics to generations of students - that made the classes mandatory, were simultaneously lying to those students. The law schools won the law suits not because the courts did not think they lied, but by invoking caveat emptor. Interestingly if you read the Wikipedia page, law schools have edited it so as to say "In 2011, several law schools were sued for fraud and for misleading job placement statistics. Most of these suits have been dismissed on the merits." The implication of this statement is that it was found that the statistics were not misleading or indeed false - in short the law schools are making statements that are literally true, but (yet again) intentionally misleading.

There has been no mea culpa from law schools. Instead various jackasses (and that is the kindest word for at least one or two) belligerently deny that there was any deception at all, that any of the data was "cooked," see for example the discussion of Santa Clara over at Prawfs (which has a fine example of the misleading statement from someone who cannot seem to work the Wayback Machine, but presents himself as a techie.) It is time for the broad legal education community to get ahead of this mess - to recognise that their schools were dishonest and to denounce what was unethical and immoral behaviour. Trying to say - "ah well, the BLS definitions...." or "well we managed to get the cases dismissed" is not good enough. Law schools need to get their credibility back - but you they only do that by being rigorously honest, not by trying the shade there data some more and claim that nothing untoward happened.

[twbb]

[M][@][c][K]:

Good points, all. The sheer advocacy is what really surprised me. We shouldn't have to point out errors; if it was properly done, Simkovic should have done that himself. Self-criticism is an important part of proper social science research, and I just don't see it here.

"Law schools need to realise that they have done huge damage to their reputations with the games they played on employment data for the 15-20+ years up to 2010-12."

I think that many don't realize that they will have to face the consequences of that, and those likely consequences are not going to be gentle, gradual budgetary adjustments over the next decade that only impacts the next crop of law professors and administrators.