Unknowns: Selection Bias?

In Job Statistics: What about Unknowns?, I began exploring the distribution across law schools of the rates at which law-school graduates do not respond to the employment surveys sent out by law schools for NALP and ABA reporting purposes.  I noted that most schools with "unknown" rates of 5% or higher have LSAT 75th percentiles of entering classes in the bottom two quintiles (bottom 40%).

A common problem with surveys is selection bias. For example, most law-school graduates that report their employment status do not report their salary.  As you might expect, graduates with high salaries may be more likely to report salary information than those with lower salaries.  But what about persons who don't respond to employment surveys?  Are persons who are unemployed more likely to throw the survey in the trash?  As I discussed earlier, one of the argument for counting unknowns as unemployed is the assumption that the unemployed are less likely to respond to employment surveys. 

But is that true?

Having already noted the divide in the rate of unknowns at a 157 LSAT 7th percentile , let's begin by looking at the relationship between a school's entering LSAT profile and its reported employment rate.  As the chart below shows, law-school employment rates tend to fall as LSAT falls.  That said, notice also that the range of variation increases as LSAT falls (which is what one would expect in the range shown (unknown rates between 0.55 and 1.00).

 

As the next chart shows, unknown rates do rise somewhat as reported employment rates fall, but the relationship is weak.   Changes in reported employment rate "account for" (are associated with) only about 8.1% of changes in unknown rates.

The chart below compares schools with LSAT 75th percentiles of 157 or lower to schools with higher LSAT 75th percentiles.

For  each group of law schools, the associati0n between changes in reported employment rates and unknown rates is even weaker than for all schools:  0.032, for schools with  LSAT 75th percentiles of 157 or lower; and 0.026, for schools with LSAT 75th percentiles above 157.

The comparison chart does show several stark differences between schools in the lower and in the higher LSAT ranges.  First, only one lower-range school had a reported employment rate above 0.95.  Second, only 3.4% of higher-range schools have employment rates 0.80, as compared to 25.7% of lower-range schools. Third, many lower-range schools have unknown rates of 0.05 or above, while few higher-range schools have unknown rates that high.

But what strikes me about both panels (lower and higher range schools) is the large range of variation of unknown rates across almost all employment rates.  Even at employment rates above 0.90, a number of schools in both ranges had higher unknown rates--around 0.05, for higher-range schools, and above 0.10, for lower-range schools.

It is probably true that lowering unknown rates would also lower employment rates somewhat.  But it is not clear to me that all unknowns ought to be counted as unemployed.  The difference in the distributions in lower and higher range schools with similar employment rates--ranges of about 0.10 or higher versus 0.05 or so--suggests that factors other than employment rates are at work.

Comments

[Brian Tamanaha]

Gary,

The argument for treating "unknown" as unemployed is based upon the general assumption that unhappy people are more likely to not respond at all, which is consistent with your overall picture.

This does not exclude other possible explanations for some "unknowns." The school may have limited resources or the career services office may be less than competent in tracking down graduates. Treating "unknowns" in this fashion creates an incentive for law schools to better a job of this (even if that requires expending more resources). That may be tough on a school, but it's a good thing if we believe that prospective students deserve full information about the likely economic return on their degree.

I should add a caution about the numbers you are working with. After putting up a post on "misleading salary numbers," a lower ranked law school informed me that the number they reported was incorrect (stated above 80% when actual number was below 30%). It is impossible to know whether, or how many, other law schools have done this with your numbers. But you should keep it in mind. The conclusions you draw are only as good as the data you use.

Brian

[Gary Rosin]

Alas, we are all no better than our data. Mistakes in reporting, even in good faith, affect the data reported. And then there are errors from re-keying the data.

But then, when I was working with the employment information in the 2012 Official Guide, I was adding categories in my head, and often came up short of the number of graduates. I used a calculator, and got the same results. I looked at my school, and our data did not add up. I talked to one of our people who complete the ABA Questionnaire. We pulled what we had reported, and found the problem. The Official Guide entry for Unemployed says it includes both those seeking employment, and those no longer seeking employment. Our Official Guide entry included only those seeking employment. We looked at several other schools, and compared the numbers from the take-offs to those in the Official Guide. Sure enough, only the numbers for Unemployed, Seeking were included.

[Anonymous]

You raise a good point about inconsistent reporting from different schools. One factor to keep in mind is that Career Services offices can be pretty small and vary in levels of expertise. The reporting requirements are confusing and it would be easy for an inexperienced or new staff person to report the wrong numbers. A contributing factor is that there is no external audit to catch mistakes. And oftentimes even no one internally "looking over the shoulder". Law schools are often relying on one person to get it right - the director or dean of career services.

Do I think most career services folks get it right? Yes. Frankly there are resources available to new career services folks that smart newcomers rely on. But I am sure some percentage of well intentioned folks are reporting false numbers. And probably an equal percentage of not so well intentioned folks who employ a head in the sand approach to checking if their numbers are as good as they look.