Tales of an Empirical Nothing (Part 2): Measuring the Impact of Judicial Nominations on the Ideology of Courts of Appeals

Continuing the tales of an empirical nothing (me, of course) [HT to Paul Secunda's "Tales of a Law Professor Lateral Nothing"] ...

So, now I had my dataset consisting of 388 opinions from the Fourth Circuit and 418 opinions from the Eighth Circuit, coded by outcomes.  The total outcomes looked different.  But, to know if the differences were meaningful, I had to learn about statistics. 

[EDIT: Note that what I was trying to figure out by analyzing the data was whether the overall ideology of the 4th Circuit had shifted.  Meaning was the court, as a whole, deciding cases differently after the arrival of Pres. Obama's successful nominees.  I was not looking at the votes of the individual judges in this initial analysis.  I have that data coded, but it was not my first focus.  A third paper perhaps. 

[EDIT CONTINUED: Also, I am well aware that correlation does not equal causation - as Derek Tokaz of LST notes in the comments.  I designed the study so that (to the best of my ability) everything other than the presence of Obama nominees was equal (or equally variable?).  The 4th and 8th Circuits were subject to the same changes in substantive law via Congress and the Supreme Court.  The facts of individual cases of course varied but I cannot imagine that the variations were vastly different in these courts throughout the decade.  The panel makeups also (obviously) shifted throughout the time periods on a random basis.  So, while I do not profess to be an expert in empirical study design, after discussing with folks who are, I think I controlled as best I could.  And, one of the reasons for publishing the results, and even blogging about them here, is so that folks can study the methodology and results -- and reach different conclusions if I screwed something up, which is possible (although I don't think I did).  So, at the end of the day is proving a positive correlation the same as proving causation?  No.  But, all other things being equal, there seems to be a very good chance that the dependent variable I tested [the addition of Pres. Obama's nominees] caused the shift.  Based on the statistical tests, I can say this with greater than 95% certainty (I think).  Of course, this blog post is a very summary description of the study and the results.  For the full discussion see the two papers I have out on the study:  (1) ObamaCourts? The Impact Of Judicial Nominations On Court Ideology, 30 [Univ. of Virginia] J. L. & Pol. 191 (2014) (the deep dive); and (2) The Clash of Old and New Fourth Circuit Ideologies: Boyer-Liberto v. Fontainebleau Corp. and the Moderation of the Fourth Circuit, 66 S.C. L. Rev. 928 (2015). (a more concise discussion of the study along with a prediction about a then-pending, now decided en banc case before the 4th Circuit).  END EDIT]

To learn enough about statistics to makes sense of my data, I went to the Conducting Empirical Legal Scholarship workshop at Washington Univ. (taught by Profs. Lee Epstein and Andrew Martin).  There I learned the basics of hypothesis testing, p-values, and suchlike.  I also learned to use Stata, which would (critically) perform the calculations. 

Now, I have read a good bit of criticism of empirical/quantitative legal scholarship and of law professors learning to do statistical analysis at CELS.   Whether those criticisms are valid depends, I think, on the nature of the scholarship and analysis.  With my data set, the most instructive test for my hypothesis was Pearson's Chi-Square (according to folks with formal training in statistics and quantitative analysis).  It does not take a degree in Statistics (or a Ph.D. in something) to perform this analysis, which not even statisticians would generally do manually.  CELS does an excellent job of explaining what results from the various hypothesis tests are statistically significant. 

Armed with my CELS knowledge and a temporary Stata license, I needed to run some numbers.  I did and what I found was very interesting.

Looking at the Fourth Circuit data set as a whole (and in isolation), there did not appear to be a significant ideological shift in the Fourth Circuit during the 2004 to 2012 timeframe. Overall, the rate of “liberal” and “conservative” outcomes remained relatively consistent ranging from an average of 72.03% “conservative” in 2004, 2006, and 2008 (the “pre-Obama years”) to an average of 68.50% “conservative” in 2010 and 2012 (the “Obama years”), a difference of 4.47%. This minor variation is not, in and of itself, statistically significant.

At the end of the “pre-Obama” years (2004, 2006, 2008), the Fourth and Eighth Circuits were separated by only 1.43%, with the Eighth Circuit being the slightly more “conservative” court. 

However, in the "Obama Years" (2010, 2012), the Eighth Circuit was the more conservative court by 12.45%.  Using Pearson's Chi-Square, the differences in the 4th and 8th Circuits in the Obama Years is statistically significant (p-value = 0.017) and shows a positive correlation between the presence of President Obama’s judicial nominees on the Fourth Circuit and the difference in case outcomes in these courts.

Based on the data collected in this study, beginning nearly immediately after President Obama’s successful nominees to the Fourth Circuit began taking their seats, the Fourth Circuit’s collective ideology began to shift toward the “liberal” end of the political spectrum, resulting in a collectively more moderate ideology and correspondingly more moderate outcomes in its labor and employment cases.

The methodology of my study and all of the statistical analyses are detailed in the first article based on this data: ObamaCourts? The Impact Of Judicial Nominations On Court Ideology, 30 [Univ. of Virginia] J. L. & POL. 191 (2014). 

So doesn't this all just sound like a pointless mathematical exercise?  Would anyone other than a law professor care about this?  I will address both of these in my final post on this topic.

TO BE CONTINUED ...

Comments

[Derek Tokaz]

"a positive correlation between the presence of President Obama’s judicial nominees on the Fourth Circuit and the difference in case outcomes in these courts"

Did you look at whether or not the decisions had an Obama-nominee as a deciding vote?

Basically, any 2-1 opinion where an Obama nominee is one of the two in the majority. If we're seeing more liberal decisions even 3-0 opinions, when it's 2-1 with the Obama nominee in the minority, or any cases where there are no Obama nominees involved, then that sort of blows the whole thing in the water and you've got a big "correlation doesn't mean causation" problem.

[Matthew Reid Krell]

Brian,

While your work is not bad (I, too, published a law review article where the test statistic was nothing more than a chi-square), I think that some more attention to quantitative methods would strengthen your work even more.

First, making inferences about the law from the results in litigated cases is fraught (see Klerman and Lee 2014, and the thirty-year debate they outline). It's not impossible (I do it myself in a piece under review on jury verdicts), but you have to be aware of the possibility of selection bias.

Dealing with the selection bias issue means you probably want to use propensity matching. Now, you started down this road, in a crude and gross way, by comparing the Fourth and Eighth Circuits ("crude and gross" referring to the level of the analysis, and not the power of the research design). So your instincts are leading you in the right direction. The problem is that the way you're analyzing the data doesn't truly leave alternative explanations controlled for. Propensity matching is a method that's already been used in evaluating the effects of changes in appeals court composition (Boyd, Epstein, and Martin 2010). If you're interested, I'd be happy to discuss with you using propensity matching in a future piece on this.

[Brian Clarke]

Reid, I appreciate the offer and would love to discuss. I still have a lot to learn (obviously), as we all do. In the meantime, i will be reading up on propensity matching.

[Travis Silva]

The failure to consider panel composition may not be fatal, though it may be a product of the ideological swing you're attempting to measure and thus confound your analysis. A swing in the entire court's ideology will have an impact on a panel's behavior, even if the panel consists entirely of pre-Obama judges. Panel members consider whether an issue will generate and/or survive an en banc call as they decide the outcome of each issue presented (and even as they decide which questions to answer in a given case) and whether to write separately. This calculus will drive individual votes and panel outcomes.

For that matter, court composition affects the en banc process irrespective of ideology. If two judges with an interest in a particular substantive or procedural area, e.g., labor/employment, replace two judges lacking that interest, panel activity in that area becomes more likely to regress toward the median position of the entire court even if the change in judges didn't alter the ideological tilt of the court.