The Faculty Lounge

New Study on the State of the Entry-Level Law-Jobs Market and its Implications (Part I: Where the Job Market for New Law Graduates is Today)

I’ve posted on SSRN a draft of my new study of the entry-level Law-Jobs market and its implications.  You can find the draft here.  It’s a followup to my earlier study on the job market, published in the middle of the Great Recession’s aftermath in 2014.  I presented an earlier draft of the new paper last spring at Florida International’s Summit on the Future of Legal Education and Entry to the Profession.  That conference was ably organized by Scott Norberg and generously co-sponsored by LSAC.  The paper will appear later this year in the Symposium Issue of the FIU Law Review devoted to the Summit.

Rather than imposing one long post on those interested, I thought I would break a discussion of the findings into four parts, posted over the next week or two.  This Part will discuss the current state of the entry-level Law-Jobs market.  Subsequent posts will discuss (ii) the entry-level placements the ABA calls “JD-Advantage”; (iii) predictions regarding future trends in entry-level employment for new law graduates; and (iv) the implications for the legal academy of where the job market is and where it’s going.

So where are we these days?  I’ll fill you in after the jump.

How the Legal Academy Has Changed Since the Great Recession (Hint-It's Probably a Lot More Than You Think)

I’ve had the privilege over the last year of working with two exceptionally skilled and thoughtful empirical investigators—Jerry Organ of the University of St. Thomas School of Law; and Emma Rasiel, Associate Chair of the Duke University Department of Economics.  The question that brought us together, which began with a conversation Jerry Organ and I struck up when we found ourselves stranded in New York by the blizzard at the end of the 2014 AALS conference, is whether we could develop reasonably rigorous but accessible ways of describing quantitatively how the American legal academy had changed in recent years in reaction to the circumstances changing around it.

As our work unfolded, the task became twofold:  First to depict how the academy had changed; and then to see if we could find any patterns in the changes (whether or not consciously strategic) among law schools with shared characteristics.  The result is our new paper Competitive Coping Strategies in the American Legal Academy: An Empirical Study.  It will be published by the good folks at the Nevada Law Journal early next year, and a draft is now posted on SSRN here.  We think that many interested spectators will find some of the paper’s observations surprising. 

We start with a description of how—and importantly, how much—the legal academy has changed.  Most readers of this space and other legal education enthusiasts are aware of the contraction in the new-graduate job market that began after 2007, and the substantial decline in both the number and the conventional qualifications of applicants to law school that began after 2010.  But you may not appreciate how widespread and serious its effects have been.  For the vast majority of American law schools, those effects have been somewhere between significant and devastating.

From 2010-11 through 2016-17, the number of unique applicants to accredited law schools fell 36%, and the number of applications fell a little less than 44%.  The conventional metrics by which most admissions decisions are made—Law School Admissions Test (“LSAT”) scores and undergraduate grade-point average (“UGPA”)—declined even more, as more highly credentialed applicants disproportionately stayed away.  For example, while the number of applicants overall fell 36%, the number of applicants with LSAT scores greater than 160 (roughly the 80^th percentile of all test-takers) fell 46%, while those with scores under 150 (roughly the 44^th percentile) fell only 27%.  All this has been widely discussed.

But the effects on the academy have been profound, and far more widespread than many realize.  By 2016-17, the average accredited American law school had an entering class that was nearly one-third smaller and had a median LSAT score seven percentiles lower than in 2010-11.  And while Base Tuition (a school’s published “sticker price”) had risen 15% on average over that period, the average tuition discount per student had doubled, causing the average net tuition paid per student to fall over 6% in constant dollars. 

Unsurprisingly, these effects were not distributed evenly across the academy.  Instead, sectors of the academy distinguished by their relative overall reputation for quality were affected quite differently.  Yes, a small handful of super-elite law schools were essentially unaffected, but that handful is smaller than you may think.  A number of top-20 law schools had to substantially increase their tuition discounting in order to maintain their entering-class LSAT scores.  More generally, the entering classes of the top-third strongest schools reputationally shrank on average only 13% and dropped 3 LSAT percentiles, while the reputationally weaker two-thirds on average shrank by over a third and nearly a half respectively, while dropping 14-15 LSAT percentiles.  Effects on tuition revenue were also pronounced.  Even top-third schools reputationally lost an average of $5.9 million per school in annual tuition revenue comparing 2016-17 with 2010-11, while reputationally weaker schools lost on average $11 million to $12 million annually each (in constant 2018 dollars).  We estimate that aggregate annual tuition revenue for all accredited American law schools fell over $1.5 billion in current dollars between its inflation-adjusted peak in 2011-12 and 2016-17. 

We offer a chart with details on the changes by reputational group and further discussion of what's happened in the last ten years after the jump.

The 2017 Bottom 10 Law Schools in the Country

As I have noted in a recent column, 10 schools have been found out of compliance with ABA Standard 501 on Admissions,  which requires that school admit only students who appear capable of earning a J.D. and passing the bar. Although several of these schools have been found out of compliance just in the last few months, there was ample evidence during this year’s admissions season of what the ABA was willing to tolerate and what the ABA considered out of bounds.  So the big question this year was whether the ABA’s increasingly aggressive campaign to stop predatory admission practices would result in more schools raising their admission standards to acceptable levels.

Well, the new ABA Standard 509 Reports are out, and the results are not encouraging. Several schools have clearly not learned from the experience of their peer schools and have admitted classes that are very likely to incur ABA scrutiny.  It seems that some schools are so desperate to stay open that they are willfully violating Standard 501 even though they must know that sanctions are likely – call it the Charlotte School of Law strategy.   

In addition to the ABA 509 single school reports, the ABA helpfully compiles data on all the schools into sortable Excel spreadsheets, making school by school comparisons relatively simple.  Drawing from the “First Year Class” spreadsheet, I have compiled a list of the bottom 10 least selective law schools in the U.S. (excluding Puerto Rico) by LSAT score (in some cases I have also considered UGPAs as a tiebreaker) for the 2017-18 school year.

The Bottom 10

1. Thomas Cooley Western Michigan U - 146/142/139. Cooley has already been found out of compliance with Standard 501 by the ABA. They recently lost in their efforts to get a restraining order against the ABA to keep this secret from prospective students and did little to improve their chances of having that decision reversed with this year’s embarrassing incoming class, with similar credentials to last year’s (147/141/138).   But the school is still making a fortune with an entering class of 458, third largest in the country (after Georgetown and Harvard). Cooley accepted 85.6% of applicants, far and away the highest acceptance rate in the country.  (Vermont, with 159 entering 1Ls, was second, with a 78.8% acceptance rate.)

2. Texas Southern – 146/143/141. Already sanctioned by the ABA for standards violations and under remedial measures, Texas Southern nevertheless enrolled a large class of 256 students, a significant increase from last year’s 227.  Unfortunately, these students are virtually all at high risk of failure.  The school grew its class by lowering its already dismal admission standards across the board.  Last year they were at 147/145/142.  What are they thinking? The school should be placed on probation.

3. (Tie) Appalachian – 149/143/141 Found out of compliance by the ABA last January and formally notified in May, they hid this fact from prospective students and managed to nearly double their class from 38 to 73 entering students, while very modestly increasing their numbers from 147/143/140. The bottom half of the class are all at extremely high risk of failure.  Their bottom 25% UGPA is an especially woeful 2.51, the lowest of any law school in the country.  Appalachian also boosted enrollment in its upper divisions by taking 16 transfers from Charlotte School of Law, which is not likely to boost their abysmal bar passage rate. Expect Appalachian to be sanctioned.

3. (Tie) Southern – 146/144/141. Number one on my list of schools that deserve to be sanctioned but haven’t yet. Southern has enrolled 200 students, virtually all of whom at are very high or extremely high risk of failure. They increased their class size to 200 from 171, so they could have raised standards, but basically held steady at atrocious. (Last year: 146/143/141) Where are you, ABA?

5. (Tie) Charleston – 148/145/142 Charleston enrolled a large class of 251 students, up from 215, by admitting 71% of applicants.  They could have been more selective and raised their standards considerably.  Last year: 149/145/141. So, for the last three years, at least 75% of Charleston’s class has been made up of high risk students.  The ABA must put a stop to this exploitation.

5.  (Tie)  Thomas Jefferson –  147/144/142.  In January, TJ was found out of compliance by the ABA on Standard 501.   TJ dishonestly kept this secret from its prospective students, enabling the school to grow its enrollment.  But this school is digging itself is an ever deeper hole by admitting a pathetically weak class in a state where students with LSATs below 146 have extremely poor prospects of passing the bar, and everybody below 150 struggles.   Last year TJ had 232 students with a profile of 147/143/141, and the second highest non-transfer attrition rate in the country at 37.2%.  Despite knowing they were facing sanctions due in part to its lax admission policies, TJ chose to increase its class size and essentially maintain its abysmal admissions standards.  The one point increase at the 50^th and 25^th percentiles was offset by lower UGPAs across the board.  Not surprisingly, TJ was recently placed on probation by the ABA.  Expect droves of transfers from TJ this winter and next summer, and continued bar passage woes for the foreseeable future, assuming TJ can stay in business.   

7. N.C. Central – 149/145/142. Now that Charlotte has closed, NC Central takes over the mantle of least selective law school in North Carolina.  To their credit, they did raise their LSAT standards a bit by shrinking the class from 183 to 166.  Last year they were at 149/144/141.  But this one point increase at the 50^th and 25^th was offset by lower UGPAs across the board.  More troublingly, NC Central had the highest level of 1L non-transfer attrition of any law school in the country last year, at 37.7% nearly twice the rate that the ABA has stated will result in presumptive non-compliance with Standard 501.  The ABA needs to take a hard look at NC Central.

8. (tie) Atlanta’s John Marshall Law School – 149/146/144.  This school was recently notified that they are out of compliance with Standard 501 by the ABA, and now I am starting to understand why.  They’ve slipped from 2016 when they were at 149/148/145, and they had a 21.9% non-transfer attrition rate, 12^th worst in the country, and above the ABA’s presumptive exploitation rate of 20%.   Last year they had 195 entering students.  This year 216.  This was not the year for the school to grow the class by lowering their standards.

8. (tie) Florida A&M University – 149/146/144. FAMU is up a smidge from 148/145/144 last year, but not enough to reverse their sliding bar passage rate. With 222 students, up from 151, they could have maintained their class size or even grew modestly and raised standards considerably, instead of growing by 50%, remaining in the bottom 10 nationally. FAMU is practically begging for ABA scrutiny.

10. Concordia – 151/147/144. Although there are four other schools with a bottom 25% at 144 and 50% at 147,  and two of these have lower LSATs at the 75% by one point, the UGPAs at Concordia are by far the lowest of these five schools (3.24/2.90/2.59) so they narrowly edge the competition for the coveted tenth slot in the bottom 10.  Although Concordia had great bar results last summer, they have placed their future in jeopardy by lowering their standards for the fourth year in a row.  Their entering class of 2014 was at 157/152/149.  2015: 154/150/146.  2016: 151/149/145.  This year: 151/147/144.  And other than Valparaiso, which recently announced it was suspending admissions,  Concordia had the smallest entering class of any law school in the country, at just 48 students.  Not good signs.

Dishonorable Mention:

There are 6 schools with 25% LSATs at 144 – which I categorize as Extremely High Risk.  Two (FAMU and Concordia) made the bottom 10. These schools have presumably calculated that they are safe from ABA scrutiny based on prior ABA adverse actions.  In light of the recent finding of non-compliance against Atlanta’s John Marshall, these schools may find that they have miscalculated.  Even if they get a pass from the ABA, these schools should be ashamed that they have stooped so low as to be admitting over a quarter of students from the bottom 23% of LSAT takers.

 The other four at 144:

• South Dakota – this state flagship university has truly fallen on hard times. They are down to 57 1Ls (5th smallest in the country) with an LSAT profile of 150/147/144 (down from 152/148/144 in 2016) and plummeting bar pass rates. The school needs to move from tiny Vermillion to Sioux Falls to survive, but this proposal has been nixed.
• Southern Illinois – Another state school with very weak students, identical to South Dakota at 150/147/144, but with much worse UGPAs. Down from 150/148/145 last year.
• Roger Williams - 151/147/144 down from 153/148/145.
• Oklahoma City – 152/147/144. Last year 151/147/144.

The Next Tier:

There are 20 law schools with a bottom 25% LSAT at 145 (26^th percentile).  There is probably safety in numbers here, as the ABA is unlikely to take action against so many schools with similar numbers.   This group includes one school that was previously forced to take remedial measures for violating admission standards (Ave Maria), one school that is currently on probation (Arizona Summit), and another school that has recently received notice of non-compliance with Standard 501  (Florida Coastal).  Florida Coastal has argued that they should not be considered in non-compliance because they have raised their bottom 25% to 145.  But the 509 report reveals that they did so by taking a lot of students with dreadful undergraduate records.  The UPGA profile at Florida Coastal for 2017 is 3.10/2.83/2.59 – worse even than Western Michigan Thomas Cooley! (The average undergraduate GPA in the U.S. is about 3.15.)  Only three law schools in the country have comparably awful UPGA profiles:  Appalachian, Thomas Jefferson and InfiLaw sister school Arizona Summit (3.15/2.81/2.53).  For comparison purposes, Ave Maria’s UGPA numbers are 3.52/3.16/2.83, about average in this cohort of 20 schools.  Ave Maria seems to have weathered the storm under the leadership of Dean and President Kevin Cieply and is on the rebound, with 97 1Ls, up from 88 last year, and a two-point improvement at their LSAT 25%.  Meanwhile, it is not clear if Arizona Summit can survive, with just 49 matriculants, down from 143 last year (and 450 in 2011!).  In 6 years, Arizona Summit went from tied with NYU for the 12^th largest entering class in the country to the fourth smallest entering class.   With the fifth highest non-transfer attrition rate in the country last year at 27.2%, they are likely to continue to get smaller.  That is one way to go from for-profit to non-profit.   

Some Other Observations

There is a new format for the ABA 509 Reports, and overall there is more useful data, particularly regarding attrition. One area of concern with the new report format is that schools with part-time programs are no longer required to report the admissions credentials of their part time classes separately.  As I noted in a recent column, many schools were admitting substantially weaker students into their part-time divisions, leading to concerns about exploitation of this cohort. Now, there is no way to know if law schools are doing so.  The ABA should require that this data be reported in future years. 

Bar passage data, like employment data, will now be reported separately. Bar passage data should be out in March, employment data in April.

Law Schools Ranked By Job Placement Rates

Updated as noted below

The ABA has provided a ton of data on law school job placement and I've tried to summarize it here - comparing the placement rate ranking with the U.S. News ranking,  and providing actual placement numbers.  The numbers set out here represent the percentage of graduates in the class of 2015 who found full-time long term positions (JD required or where a JD was an advantage) within nine months after graduation.  This is the number many ranking organizations use, though I recognize that there are other  ways to slice the data.  As I note below, I have excluded law school funded positions.

This ranking uses the freshest data currently available and - compared to what you can find for pretty much any other discipline outside of law - a pretty damn good data set.    We only wish business schools and other similar programs would give this sort of guidance to applicants.

I think a big takeaway is that the US News ranking doesn't actually predict how well graduates of a given school will do on the job market.  One conclusion from this data is that any school recruiting solid students (as opposed to students who probably don't belong in law school)  can deliver those students good job outcomes if it trains them well and provides good support from bar prep through career planning.  (It helps to be in a region with jobs or a region that is not a net importer of law grads.)

It's also worth pointing out that this chart suffers from an infirmity it shares with US News: it tends to over-state the gaps between law schools.  Another way to read this chart is to think in clumps - schools in the +/- 85% placement range; those in the +/- 75% placement range; etc.  But I do think there may be real differences between schools with 80% rates and those with 65% rates.

This ranking doesn't get at salaries - though, having looked at the data, I have a strong sense that there are really only about 15 schools where more than 33% of the graduating class scores a large firm job.   US News is actually pretty good at this element of ranking: if you want a big firm job, the US News top 15-20 is a reasonably good guide.  But US News ceases to be very predictive of large firm placement after that point.   This list also has anomalies: Yale is #20, but that probably has more to do with the choices that students made rather than the options they encountered.  And Georgetown?  My theory is that their career strategies office is good for students seeking big firms, but has work to do with the other 63% of the class.

The schools with the biggest gaps between this ranking and US News?  Figure that out for yourself - there are some whoppers!

Update:  I've decided that it would be more accurate and useful to round placement results to the nearest whole number and describe virtual ties as ties.  As I discuss above, my goal is to provide information without overstating gaps between schools.  Schools that are tied are listed in order of their precise rank.  With this change, Yale is now #18.  In this analysis, I excluded law school funded positions - consistent with US News - in large part because it is impossible to determine how many of these graduates would have been able to secure long-term full-time employment in the absence of law school assistance.   

Update 2: A kind reader has alerted me to at least one error - the numbers provided to the ABA by South Texas on their form are inconsistent with the numbers set out in the ABA spread sheet.  I am not going to change anything yet, but I hope that over the next day or two, I can identify any other conflicts between the forms and the spreadsheet.  If there is an error or two, I can fix that easily.  If problems are more pervasive, I'll pull and rework it entirely.  For now, please note that based on the employment form submitted to the ABA, South Texas seems to have a 62% placement rate, rather than a 54% rate,  which puts it at 152 rather than 172.

Update 3, Nov. 30, 12:30 pm: I have adjusted the chart to reflect what I expect are accurate numbers.  As I discuss in this post, there are discrepancies between data reported by the ABA in individual school summaries and in the overall compilation spread sheet.  This list reflects the numbers reported by schools in their individual school summaries.  These changes shifted the rankings a bit.  I have inserted asterisks after schools whose ranking changed after switching to the data provided on the individual school summary.  In a later post, I will detail the changes.   I believe the numbers are now as accurate as I can make them.  

So without further ado: 

Law School Transparency (LST) Responds to Law School Admission Council (LSAC) Press Release

Law School Transparency’s recent report, the 2015 State of Legal Education, led to numerous articles in the national media about the trend of law schools admitting large numbers of poorly qualified applicants.  This prompted the Law School Admission Council (LSAC) -- the organization responsible for the LSAT -- to issue a press release  denouncing LST’s report and purporting to rebut certain factual assertions allegedly made by LST.  The press release mischaracterizes the conclusions of the LST report. 

The press release begins:  “A report recently released by Law School Transparency (LST) has gained headlines by claiming that some ABA-approved law schools have been intentionally admitting “high risk” students who, based on their LSAT scores, do not have a reasonable chance of passing the bar.”

LST stands by the assertion that dozens of ABA-approved law schools know that they have admitted large numbers of students who, based on their low LSAT scores, coupled with commensurately low undergraduate GPAs, are at high risk of academic failure or failing the bar.  Although LST’s risk band categories are delineated by LSAT scores, LST was very careful to explain that the risks of a low LSAT score could be offset by strong academic performance in college. LST analyzed the available data and determined that law schools were not offsetting lower LSAT scores with higher GPA requirements.  Although many of these same law schools have been allocating more resources to internal and external academic success and bar prep programs, these efforts have not yet offset the overall decrease in student capability, resulting in a strong trend of decreasing bar passage rates at the high-risk category schools.

Law schools can analyze internal data about attrition and bar passage rates by LSAT score and UGPA to make reasonably accurate predictions of the likely success of applicants with similar credentials.  Law School Transparency has urged (and continues to urge) law schools to make internal data publicly accessible.  LST acknowledges that some law schools may be better at educating students with marginal predictors and helping to prepare them for the bar. LST stands by its assertion that law schools have a duty to the legal profession to share empirically validated findings.

LSAC specifically states that LST has made three false claims.  The first alleged false claim is that “LSAT scores can be used to assign bar passage risk.”  LSAC objects to LST’s labeling of students with certain LSAT scores as high risk, very high risk, and extremely high risk.  Drawing from LST’s report, LSAC notes that students labeled “high risk” at one school had a first-time pass rate of 57 percent, while comparable students from another school had a pass rate of 23 percent. Based on this disparity, LSAC concludes: “Clearly, many factors significantly affect bar passage rates above and beyond LSAT scores.”  LST agrees wholeheartedly with this statement and has never claimed otherwise.  The LSAC statement concludes “The assertion that LSAT scores alone measure comparability is patently wrong.”  Here Mr. Bernstine is arguing against an assertion that the LST report never made.  Here is what the LST report actually says about assessing risk by LSAT score: 

The [LSAT risk bank] framework represents only a starting point for assessing the risk of bar failure.  A student with a low LSAT score but very high undergraduate GPA, for example, has less risk of failing the bar than a student with the same LSAT score and a very low UGPA. Some law schools have also been more successful than others in helping students with low LSAT scores succeed on the bar exam. Where the student takes the bar exam matters as well.

While LSAC may object to “labeling” law schools and law students, LSAC doesn’t dispute the underlying premise that there is a strong correlation between LSAT score and success in law school and on the bar exam. Mr. Bernstine does not because he cannot. LST believes that “high risk” is a fair characterization of both a 43% and 77% risk of failing the bar the first time, as reflected in the data from these two schools.

Important New Report From Law School Transparency on Law School Admissions

An important new report entitled: 2015 State of Legal Education: An in-depth look into law school admissions choices” has just been released by Law School Transparency (LST).  (Full disclosure – I had a minor role in editing some sections of the report.)  The report has already received attention in the press (see this editorial in today’s New York Times) and is likely to receive more in the coming days.  It is also likely to be a topic of conversation among current and prospective law students, many of whom rely heavily on LST for reliable information about law schools.  I recommend that all law professors and law school administrators read it in its entirety.  In the meantime, I offer below a summary of what I consider to be some of the most important findings and recommendations in the report.  

As the title implies, the report focuses on the questionable admissions policies practiced by many law schools during the enrollment crisis from 2011 to the present (although there are several other important subjects covered as well).  Using a risk profile that I developed based on LSAT scores (first reported here on TFL)  the report identifies 74 schools which admitted classes consisting of at least 25% at risk students in the fall of 2014 (statistics are not yet available for the 2015 entering classes).  Fully half of these schools, 37, admitted classes consisting of at least 50% at risk students (149 LSAT and below).  Most disturbingly, 18 schools admitted classes where over half of the students were categorized as being at very high or extreme risk of failing out of law school or failing the bar exam. (See the list of schools here.)

The report acknowledges that LSAT scores are imperfect predictors, and that the risk presented by admitting a student with a low LSAT score might be offset by requiring that the student have a correspondingly strong undergraduate GPA.  Through sophisticated statistical modeling, the report proves that this is not what is happening.  (See the section headed “Serious risk law schools did not take enough students with better undergraduate GPAs” in the Analysis section of the report.) 

To confirm the reliability of my risk-model, LST managed to acquire data regarding bar passage from two law schools. A school which chose to remain anonymous provided 7 years’ worth of data, and the University of Denver provided 14 years’ worth of data.  The data confirms that my risk categories are valid.  At the anonymous school, only 16% of students in the extreme risk category (144 LSAT or lower) passed the bar on their first attempt.  At the University of Denver, students in the extreme risk category passed on their first attempt at the slightly higher rate of 27%.   In the very high risk category (LSAT of 145 or 146), 19% passed the first time at the anonymous school, and 39% passed the first time at Denver.  In the high risk category (147-149 LSAT), 23% passed the first time at the anonymous school and 57% passed at Denver.  

The U.S. Department of Education's Non-ranking College Rankings

The NYT reports today that the Obama administration gave up its efforts to rank universities and colleges, after intense opposition.  Instead, it provides information on its new College Scorecard website.  That's true - sort of.  

Instead,the College Scorecard homepage highlights interesting blog posts from Homeroom, the official blog of the DOE.  Here we discover non-ranking posts like "23 four-year schools with low costs that lead to high incomes."  But the DOE is no different than US News.  It has to make decisions about how to assess things like "low costs."  You might assume tuition?  Or perhaps tuition minus discount rate?  Both would be fair.  

Here, though, is the methodology: This list includes schools in the top 10 percent of predominantly four-year-degree-granting schools for 1) median positive earnings 10 years after beginning at the school and 2) low net price for students receiving federal grants or loans with a family income of $0-$48,000. 

This is useful information.  It ends up preferencing elite schools, in the main, that have aggressive scholarship funding paired with excellent outcomes.  And, of course, these schools can now market their Top 23 ... or should I say "DOE 23" status.  Because the DOE is careful not to call these schools top anything, but rather "23 schools."  I am curious whether the list reflects the whole top 10% within their chosen cohort, or whether the editors - for whatever reason - listed only a selection of this 10%.

In any case, it appears that nonwithstanding the Times' reporting, the DOE is in the rankings game.  And every school is now praying that the editors choose methodologies friendly to their particular circumstances.  Perhaps Robert Morse might be available to advise them.

Ranking The Top 100 Law Schools On Employment Outcomes

You can slice and dice these ABA employment stats 1000 different ways, but Matt Leichter has ranked schools by percentage of the class employed nine months out in full-time, long-term, JD required jobs - and excluded law school funded positions.  What is utterly unsurprising, but notable, is that schools cluster in zones.  Only 13 schools place 80% of the class in these positions.  A bit fewer than half of all schools place 60% of their class in these jobs.  Also true: you can see real differences in outcomes depending on region.  Cities that import large numbers of graduates (see, e.g., San Francisco) have law schools with more dismal results.  Hastings, a fine law school, ranks at #143 because its natural market is bursting with recent law graduates from everywhere else.  The method yields anachronisms - Yale ends up at 33, for example,  And it says nothing about salaries: Yale grads' median incomes are likely higher than several schools that rank above it.  

But taking Bernie's point, there is something here for people who want to know - for example - how likely is it that they're going to be a practicing attorney a year after graduating.  

I've sheared off the comparative data, but for convenience, this is the Top 100 for 2014.  

Do Employers Actually Look at U.S. News Rankings?

In honor of the impending US News Rankings release (and in the alternative to discussing them, which I don't plan to do), I wanted to pose a question. I think I know the answer, but I could be wrong: Do employers actually look at the US News rankings in making hiring decisions? I ask because I've seen this written on blogs and in comments - that the rankings are how employers see you. This post at Above the Law suggesting that employers not do so (and thus assuming that they otherwise would) is an example. This doesn't mean students don't look at the rankings or even that they shouldn't. It merely questions this factual predicate about hiring.

My gut says most employers don't for a several reasons reasons.

Is U.S. News Still Relevant?

Update: Spivey Consulting has a leaked copy of the new U.S. News rankings -- the top 20 are here and 21-50 are here.

Update:  Dean David Yellen is talking about this over at the law deans blog in a post called "Is the Influence of U.S. News Declining?"  He points out that the response rate to the reputation surveys seems to be declining -- and that, too, may suggest that respondents are losing interest, too.

I've asked before are law blogs still relevant and now I want to ask a question about the impending U.S. News law school rankings for 2016.  For some reason I don't sense the same excitement (and in some cases fear) of the release of the U.S. News law school rankings that I remember from a few years back.  I guess I have two questions, really.  First, is my perception accurate that law schools, students, faculty, employers, and maybe most importantly prospective students have moved on from the U.S. News rankings?  That is, are the U.S. News rankings still relevant?  And second, assuming that prospective students have moved on, why is this the case?  Is it because of Kyle McEntee et al over at law school transparency have provided alternative and more useful means of comparing law schools?  Are students no longer looking to overall ranking, but to how well particular schools will meet their needs?  It seems like in a world of increasingly granular data that students ought to look beyond a single rank and maybe they're doing that now.  Is U.S. News measuring things that students don't care about?  I'd be interested in your thoughts about this and about the future of U.S. News' rankings.  

Of course, once the rankings are available on Tuesday I'll set to work to see how they compare with the alternative ranking that I've put together that looks to three variables -- the median LSAT scores of a school's entering students, the school's employment outcomes for recent graduates, and citations to the school's flagship law review.  I continue to be impressed with how highly the U.S. News rankings correlate with other rankings, such as employment outcome (hence the scatterplots that I use to illustrate this post).

Harrison and Mashburn Ask the Meaning of Citations And Suggest Reform

Jeff Harrison and Amy Mashburn of the University of Florida have a paper up on ssrn, "Citations, Justifications, and the Troubled State of Legal Scholarship: An Empirical Study."  It asks the uses that authors and courts make of legal scholarship.  The ask whether scholarship is worth the cost, which they estimate at $240 million per year.  Cribbing now from their abstract:

Recent pedagogical, economic and technological changes require law schools to reevaluate their resource allocations. Although typically viewed in terms of curricular changes, it is important also to focus on the very significant investment in legal scholarship and its impact. Typically this has been determined by some version of citation counting with little regard for what it means to be cited. This Article discusses why this is a deeply flawed measure of impact. Much of that discussion is based on an empirical study the authors conducted. The investigation found that citation by other authors is highly influenced by the rank of the review in which a work is published and the school from which the author graduated. Courts, on the other hand, are less sensitive to these markers of institutional authority. Perhaps more importantly, when the purpose of the citation is examined, a very small handful of those citing a work do so for anything related to the ideas, reasoning, methodology, or conclusions found in the cited work. This is slightly less true for judicial citation compared to citations by other authors. Given the level of current investment in legal scholarship and findings that reliance on it is far lower than citation counts would suggest, the authors offer a number of recommendation designed to increase accountability of legal scholars and the utility of what they produce.

I'm guessing there's going to be a lot of talk about this paper.

How Law Professors Use Twitter

Given some sharp comments after my last post about law professors using Twitter, I decided to see if there's anything the data has to say about it. The comments I've seen are generally of the type that professors merely use twitter for self-promotion that does not promote any scholarly or legal interest. Related comments from earlier census posts were essentially of the type, "Why do you care about tweets when your law school house is burning down?" Without accepting the burning house analogy, I want to provide the evidence I found and discuss how it bears on these criticisms. In a future post, I'll discuss my view of Twitter, as well as a way to obtain more granulated results than I'm presenting here.

(Note: some of the data here is incomplete. Updated data is presented in a later post).

More Rankings Based on LSAT, Employment, and Citations

A couple of weeks back I wrote about a brief study that ranks schools based on their rank on median LSAT score of the class entering in fall 2013, employment outcome for the class of 2013, and citations to their main law review.  I have now done two things with this paper -- first, expanded the analysis to all 194 law schools that U.S. News included in its March 2014 rankings (what it calls the 2015 law school rankings).  Previously I dealt only with the 147 schools that U.S. News provided ranks for; now I include the additional 47 that U.S. News called "unranked."  Second, in response to suggestions by readers, I have used two different measures of employment.  The initial study used the percentage of the class of 2013 employed nine months out at full-time, permanent JD required jobs and this study provides a ranking using that measure.  I now provide a separate table that uses a modified employment score (full-time, permanent JD required jobs minus school-funded positions and solo practitioners).  

The exclusion of school-funded and solo practitioners causes some schools to fall rather dramatically and a few schools rise a little.  One of the tables in the paper reports the schools that have the highest percentage of those positions (Emory tops the list at 21.9%; William and Mary is close behind at 20.7%; the University of Virginia is third at 15.9%).  The final table in the paper reports the schools' three variable rank using the "traditional" measure of employment and the three variable rank using the modified employment measure and the differences between those two ranks.

As before, there is a high correlation between the U.S. News' rank of the top 147 schools and the three variable rank presented here.  That in some ways validates U.S. News and maybe they validate the rankings here.  But importantly, there are significant differences between the U.S. News rank and those presented here, which suggests that prospective students should look very carefully at each school to see how it performs on factors that they care about.  Some schools' ranks in U.S. News seem to be supported by strong reputation scores that may or may not reflect current realities and some schools are performing significantly better (or in some cases worse) in areas like graduates' employment rates than their U.S. News ranks would suggest.

Anyway, the expanded version of the paper is now up on ssrn.  For those looking for the bottom line (i.e., faculty, administrators, students, prospective students, and alumni curious about how your school fares), tables 4 and 11 are the ones you'll want to turn to.  Table 4 provides the ranking of all 194 schools based on the rank of median LSAT for the class entering in fall 2013, percentage of the class of 2013 employed in full-time, permanent JD required jobs, and citations to the schools' main law review from 2006 to 2013.  Table 11 ranks all 194 schools based on rank on those 3 variables, but using the modified employment variable that excludes school-funded positions and solo practitioners.

Update: Lawschooli.com has some commentary on this.  Also, Laura Stantoski over at most strongly supported discusses the rankings methodology extensively.  Ms. Stantoski picked up the story from a post I had at law.com about the new rankings.

The Indiana Law Journal Supplement published the study in November 2015.  http://ilj.law.indiana.edu/?p=3711

Ranking Law Schools Based on LSAT, Employment Outcome, and Citations

Given all the talk about the U.S. News law school rankings I've been thinking for a while that I wanted to look at how a ranking of law schools based on a few of the factors that prospective students might care about (and that I certainly care about) would look.  I have put together a quick look at schools based on an average of ranks of the median LSAT score of students entering in fall 2013, the rank of schools based on the percentage of their 2013 graduates who had full-time, permanent JD-required jobs, and the rank of their main law review based on the number of citations it received from 2006 to 2012.  There's a lot one could argue about with regard to the selection (and weighting) of those three variables -- and if I do anything else with this paper I may very well refine the weight assigned to each variable (or even change the variables).  But I do think this is a nice start in the direction of ranking schools based on the factors that matter in quality of an institution. 

One of the things that I found of particular interest is how highly correlated a lot of the variables are.  For instance, correlations between each of the three variables and the overall U.S. News rank are also high, though the U.S. News rank and LSAT median rank are correlated most highly of the three (.93).  The correlation between U.S. News rank and full time, permanent JD required jobs rank is .76 and the correlation between U.S. News rank and law review citations rank is .87. 

Cribbing now from my abstract on ssrn:

This paper returns to the perennially favorite topic of ranking law schools.  Where U.S. News & World Report includes a wide variety of factors – some of which are criticized as irrelevant to what prospective students care about (or should care about) – this paper looks to three variables.  They are median LSAT score of entering students, which seeks to capture the quality of the student body; the percentage of the graduating students who are employed at 9 months following graduation at full-time, permanent JD required jobs; and the number of citations to each school’s main law review.  This paper rank orders each of those variables, averages those ranks to obtain a new ranking, and then compares those new rankings to the U.S. News & World Report rankings.

If you're interested in this, you can find the full paper here, along with tables that list the schools that are the biggest winners in this new ranking and those that decline the most.  Down the road I'd like to extend this ranking to all ABA-accredited law schools.  The image is a series of scatter plots of ranks on U.S. News, LSAT medians for students entering fall 2013, percentage of class employed at JD-required long-term, full time jobs,  and citations to law reviews, 2006-13.

Update: Over at taxprof Paul Caron has some discussion of this paper and there are a lot of comments on what should be counted in a ranking system.  One of the things that I hoped this initial post would do is help me refine the factors to use and I appreciate the comments there and here -- and I'm particularly interested in the range of responses to the use of citations to schools' law reviews.

Michael Smith is also talking about the new ranking method at his blog -- especially the virtue of avoiding the peer assessment scores.

Diving Into More Law School Employment Data From The WSJ

After my recent critique of the Wall Street Journal Law Blog's coverage of law school employment data, I received a nice note from Joe Palazzolo - one of the post's co-authors.  He shared with me a bit more data they'd found, further parsing the US New Top 50.  Pulling apart that data helps show even better how problematic  it is to conflate US News Schools into Top 50, or even Top 25.  Here is the data Joe shared with me, based on their assessment.

Percentage Long-Term Full-time JD required in Top 10  89%
Percentage LTFT JD required in Top 25                                  83%
Percentage LTFT JD required in Top 50                                 75%

But let's dig a bit deeper.  The median percentage LTFT JD required in US News 26-50 isn't anywhere near 75%.  Using Matt Leichter's chart, which excludes law school funded jobs, the median school in this group placed 64% of its students.  And more evidence of how US News rankings provide bad guidance in this area: the 25th percentile placement rate for this cohort (USN 26-50 schools) is 58.5%. 

But don't assume that things are super-rosy for the higher ranked schools.  I looked at schools ranked 12-24 and discovered that their median LTFT JD (non-funded) placement rate was 67.8%.   That made me wonder how many Top 50 schools even matched that 83% placement rate implied by the WSJ data for the Top 25.  The answer: 7.  How many hit 89%  identified for the Top 10?  

Ruh-roh.  None.

I recognize that there's apples to oranges action here.  First, I excluded law school funded jobs.  That has a big effect because, looking at the top 4 placing schools even in the Leichter world (excluding law school funded jobs), you discover that each of them places over 5% of their total graduating class in school-funded positions (NYU >8%, Columbia and Chicago >6%, and Penn >5%).  Columbia had the best placement data of any law school within these parameters, and it placed 88.3% of its students excluding law school funded positions.  Not a single school places 89% of its grads in these jobs without a little boost from the school itself. 

Second, I used medians for each school rather than placing all the students from all of the schools in one basket and tallying the percentage placement rate.  But my choice makes visible the way in which large, high-placing schools - like Harvard, Columbia and NYU - skew the WSJ data.  If a prospective student is assessing law schools, she might not realize that the strong placement rates for the Top 50, or even Top 25, are really driven by a few large, super-elite schools.  

As I have said before - along with everyone from Brian Leiter to Kyle McEntee - US News is pretty unhelpful as a tool for  picking law schools.  That unhelpfulness is compounded by the way in which the WSJ sliced and diced the data.  It is one thing for the WSJ to report US News rankings.  I have to concede that for much of the paper's readership,  the rankings are newsworthy.  My objection here is to the ways in which it then manipulated the rankings to create new information even more misleading than the rankings themselves.

Why Is Is So Hard To Report On Law Schools?

That's what I wondered when I read the last Wall Street Journal Law Blog post on law schools.  In Law School Job Data Shows Wide Gulf Between Elite and the Rest, Jacob Gershman and Joe Palazzolo make two choices that somewhat undermine the value of the post.  First, their primary focus was on total employment rate - the one that includes every job,  from Skadden to Starbucks.  Second,  and arguably more problematically, they selected the U.S. News Top 50 as a relevant category to help define "elite" schools.  Thus we get a paragraph like this:

About 5% of class of 2013 graduates from a top 50 school were still looking for work in February, about nine months after spring 2013 graduation. Meanwhile, 14% of graduates of schools below the top 50 were searching for a job.

There is a wide gulf between the "elite" law schools and the rest.  But most of the US News Top 50 are not in that elite group.  The post notes that roughly 75% of graduates from US News Top 50 schools snagged long-term, full-time, JD required jobs.  But if you exclude law school funded jobs, only 15 schools placed 75% of their grads in those jobs. 

In fact, 17 of the US News Top 50 - or over a third of that cohort - were not among the top 50 schools for placement in that category.  So not only did only 15 of the US News Top 50 place at the level the article describes as "average" for the Top 50, but a third of them actually placed fewer than 63.3% of their grads in long-term, full-time, JD-required, market-provided positions.

None of this is news for our readers, and I'm frustrated that it would be for the WSJ Law Blog.  Maybe its just the nature of news reporting today.  Reminds me of the New York Times' recent report about the five law schools that have shut down in the last two years. 

Update 5/6/14: Joe Palazzolo correctly points out that his article did not expressly discuss total employment rates, but rather focused on the "unemployed" number.  He now adds that when he referred to "unemployed", he meant "unemployed-seeking."  First off, those are two very different categories - several schools seem to have suprising numbers of graduates who have no interest in a job.  It would be worth the WSJ clarifying that point.  In any case, I see the "unemployed seeking" number as a primarily a  proxy for total employment rate - though to be fully accurate, a law school may have some number of students who are categorized as  "employment status unknown", "unemployed-not seeking", "unemployed-start date deferred", or "pursuing graduate degree".  All that said, he is right in that I was imprecise when saying that their "primary focus was on total employment rate."

Justice Alito On US News Law School Rankings & The LSAT

"They're just super-bogus."

Or something of that nature.

H/T WSJ.

Netting Out Law-School Funded Jobs

In Rankings and Employment Data, I noted that law-school funded jobs are included in the reported jobs in the Bar Passage Required, Full-time, Long-term category. In Law-School Funded Jobs, I discussed law-school funded jobs in general, but did not focus on the Bar Passage Required, Full-time, Long-term category. In New Law School Rankings: Employment Data Cleaned Of School Funded Jobs, Dan Filler netted out law-school funded jobs from that category, I thought I'd give you a little more information on the scope of law-school funding, and its affect on reported employment rates.

For the vast majority of law schools (86.3%), the school funding Bar Passage Required, Full-time, Long-term jobs is not a material issue. Over two-thirds (67.5%) of law schools did not fund any jobs in that category. For another 18.8% of law schools, such jobs added less than one percentage point to each school's total reported employment rate.

For the 27 schools that added at least one percentage point to their reported Bar Passage Required, Full-time, Long-term employment rate, the following table gives the percentage points added to the employment rate by law-school funded jobs.

Rankings and Employment Data

The state of the market for entry-level legal employment is important to current law students, and to potential applicants for admission to law school. As Al Brophy, and others, have discussed, the number of applicants for the Fall 2014 entering class continues its free fall. Before long, the ABA (and NALP) will be releasing the employment report for the Class of 2013. That report may well influence decisions about applying to law school at all, and the law schools to which applicants might apply, or enroll.

Given that, I was not entirely surprised by an inquiry from one of the editors of a Wikipedia entry on law-schools, or on recent graduate employment. Last year, when the Class of 2012 ABA employment data was released, I focused on one crucial slice of the overall data, the percentage of graduates employed in positions (i) requiring admission to the Bar that were both (ii) full-time and (ii) long-term. In my initial post, Bar-Admission Required, Full-Time, Long-Term: First Look, I discussed the general distribution of Class-of-2012 law-graduate employment rates. I also listed the 20 schools reporting employment rates in such positions of at least 75%.

Given our national obsession with rankings, and requests from commenters, I folllowed up with Full Rankings: Bar Admission Required, Full-Time, Long Term, which

• posted the Class of 2012 ABA Employment rates in that category for all 197 law schools (i) located in the continental United States and (ii) reporting employment statistics, and
• ranked the law schools from highest to lowest reported rates.

Thus, the inevitable question: is it better to focus on reported employment rates (percentages), or on rankings?

Success and the Roles of Quality and Chance

On NPR today, “Morning Edition” ran a fascinating story on the role of chance in success and failure. Is the Mona Lisa much beloved because it truly is exceptional or because of factors other than artistic quality? According to nicely-designed research by Matthew Sagalnick and colleagues, chance plays a much bigger role than most people would suspect. Their results indicate that “after you meet a basic standard of quality, what becomes a huge hit and what doesn't is essentially a matter of chance.”

This should not be surprising. Movie producers, television executives, and book publishers are notoriously ineffective at sorting the likely to succeed from the unlikely to succeed. Thus, multiple publishers rejected the first Harry Potter book, and the Beatles “struggled to get a record deal.” People often favor a particular piece of art, music, or literature simply because other people favor it.

There is no reason to think that studies would come to a different conclusion for academic research. One wonders whether law review editors, book publishers, tenure committees, and other evaluators will give these findings the weight they deserve, especially since current practices tend to reinforce rather than counterbalance elements of chance.