Cunningham on Correlation of Marketing Materials and US News Scores

Larry Cunningham has a very interesting essay up on ssrn that looks at marketing materials that law schools send to other schools and relates them U.S. News peer assessment scores (and also changes in one year in the scores).  The paper is entitled, "The Effect of Law School Marketing Materials on U.S. News and World Report Rankings."  Cribbing now from Cunningham's abstract:

In the last few years, law schools have inundated each other with glossy brochures, postcards, magazines, and other marketing materials in an attempt to influence their “peer assessment scores” in the annual U.S. News and World Report rankings. This article describes a study that attempted to determine whether law schools’ print marketing efforts to one another have an impact on their U.S. News rankings data. From June to December 2011, the author’s school collected and coded all of the materials it had received from schools, including materials that it itself had sent to others. In total, 427 unique pieces of marketing were received from 125 of the 191 schools that were the subjects of this study. They varied considerably in size, format, content, and audience. A number of statistical tests were conducted to compare a school’s marketing efforts with its overall rank, overall score, peer assessment score, and tier, along with any change in those variables from the 2011 rankings to the 2012 ones. The results showed that there was some correlation between a school’s marketing efforts and its U.S. News data. Schools that sent marketing materials had, on average, higher tier placement and peer assessment scores; however, there was not a significant change in year-to-year rankings variables. The number of pieces a school sent during the study period was, for the most part, not significant. On the other hand, the number of pages in its materials was correlated with a number of U.S. News variables. Schools that sent longer, magazine-type publications geared towards a specific audience had higher U.S. News scores and also showed a slight improvement in their overall score between the two years of rankings data in this study. However, it is possible that a co-variate, such as institutional financial resources, may be causing the results. Additional study is needed to determine whether marketing materials have a longer-term effect on U.S. News ranking variables that cannot be captured in a one year study.

I know that a number of people have weighed in on this already -- including the Wall Street Journal Law Blog, but I have a couple of thoughts about this.  First, the U.S. News peer assessment scores are so notoriously static that I'm not surprised there is essentially no relationship (especially over a one-year period) between changes and law school promotional materials.  Second, I know this is asking for a different study, but I'd be very interested in seeing a similar study that looks at the U.S. News lawyer/judge scores.  I'm wondering if lawyers and judges are more suspectible to marketing than are academics?  Third, there's a suprisingly high (.38) correlation between number of pages sent and U.S. News peer assessment score.  As Cunningham notes, this may be just an artifact of the wealth of the better-ranked schools -- the places with more money spend more money, including on promotion.  But what this reminds us is that the U.S. News peer assessment scores are correlated with all sorts of other measures that one might associate with quality (like LSAT scores, citations to their law review, student-faculty ratio, employment).    Fourth, I have noticed a dramtic decline in mailings of late -- I think this reflects that we're in a world of increasing austerity.  The era of conspicutive consumption has ended or is ending.

On a distantly related topic, earlier in the summer I put a paper up on ssrn that looks at African American student enrollment and U.S. News ranking.

Comments

[Larry Cunningham]

Thanks for the writeup. The challenge with doing a lawyer/judge survey is that we couldn't be confident that the individuals participating in the study are receiving all of the materials that are being sent. My sense is that schools do a scattershot approach to these mailings, often focusing on alums or regional practitioners/judges.

You make a good point regarding the many factors that go into peer assessment. One way to test to see which of the variables is, in fact, driving the result is through a regression analysis. Maybe next year.

I sure hope we are entering a period of declining law porn. My worry is that law schools may start moving to e-porn. At least there is always the unsubscribe option if a school is complying with CAN-SPAM ...

[Alfred Brophy]

Hi Larry,

Thanks for commenting and for the study. Really interesting stuff in there.

I completely understand on the lawyer/judge survey. It's hard to know how much law schools do and it's hard for them to target mailings, since we don't have a good idea of who's surveyed by U.S. News. For a paper on an entirely separate issue I've looked at changes in peer assessment and lawyer/judge assessment and I've been shocked at how static the peer assessments are and how fluid (by comparison) the lawyer/judge assessments are.

One thing that just occurs to me is that it would be interesting to see if there's any relationship between the mailings sent to law schools and the lawyer/judge scores over the course of a year. One might hypothesize that the size/number of mailings to law faculty is a proxy for mailings to lawyers and judges.

Then again, perhaps someone could prevail upon a friendly federal judge to toss all the mailings into a box for a year!