Sociology's Denominator Blues

I have this essay today on Social Science Space:

Ethnography’s Denominator Blues

Published on 08/11/2021 By Steven Lubet

A friendly sociologist recently drew my attention to a scathing review of my book, Interrogating Ethnography. I have realized for the past few years that sociologists tend to be thin-skinned about suggestions from outsiders, while often demonstrating surprisingly poor reading skills. It had not previously occurred to me, however, that someone with a doctorate in social science might lack a fundamental understanding of basic mathematics, such as the role of the denominator. Alas, that seems to have been the case with UCLA’s professor Stefan Timmermans, whose review, titled “Hypocriticism,” appeared in the journal Contemporary Sociology and was cited approvingly in this year’s Annual Review of Sociology.  

In what he evidently intended as a devastating riposte, Timmermans wrote:

I tallied up all the fact-checking in this book and grouped them into three categories:

* Lubet doubts ethnographer/finds ethnographer is wrong: 23

* Lubet finds ambiguity/unable to factcheck: 10

* Lubet finds ethnographer is correct/shows ethnographer fact-checking own claims: 22

Considering that this is selectively drawn from 50 ethnographies, each making ‘‘many thousands’’ (p. 136) of claims, Lubet’s indictment of ‘‘ethnographic malpractice’’ (p. 1) falls flat.

Now, it is true that a problem rate of 23 out of thousands would be inconsequential, but it is always possible to trivialize an observation by assuming an exaggerated denominator. What Timmermans did not choose to recognize – although clearly stated multiple times in the book – was that my sample consisted of only those claims that could possibly be fact-checked. Of that number, at least half turned out to be dubious – 23/45 – which is a troubling error rate, to say the least.

Epidemiologists calculate the COVID-19 positivity rate by comparing the number of positive results (numerator) to the number of tests actually administered (denominator). Only a pandemic denier or conspiracy theorist would seek to distort the outcome by expanding the denominator to include the entire population of a locality or state. That sleight of hand might make a disquieting situation appear relatively inconsequential, but it would be misleading and disingenuous.

Ethnography, as I wrote in Interrogating Ethnography, is the only academic discipline – save, perhaps, oral history – that relies on anonymous sources, as reported by a single investigator whose underlying data are never released. Timmermans makes the silly argument that critics of ethnography should be dismissed unless they have done “the real shoe leather work” of reinvestigation. In most cases, of course, that is conveniently impossible, given that ethnographers typically go to great lengths to anonymize their subjects and mask their research sites.

I would be more inclined to attribute Timmermans’s math error to simple negligence, rather than intentional spin, if his other cavils were not so tendentious. He wrote, for example:

Time and time again, he ‘‘fact-checks’’ an ethnographer’s observation with a couple of calls to a prosecutor, police detective, or hospital administrator about a sticky observation. Lo and behold, they deny that the ethnographer ‘‘correctly’’ represented their institution in a possibly negative light. QED: Lubet’s verdict is that the ethnographers were wrong, misled, or naıvely sympathetic with their interlocutors.

Sure, I spoke with the establishment figures Timmermans lists, but I also made far more than “a couple of calls” to public defenders, journalists, housing advocates, economists, educators, and social workers – all unmentioned by Timmermans. Nor does he acknowledge that I also interviewed or corresponded with many of the ethnographers whose work I explored, while cross-checking their observations against census figures, newspaper accounts, court records, school records, hospital records, media records, medical data and CDC reports, historical data, and other available sources. Timmermans’s scare quotes aside, these are exactly the same means used by prominent ethnographers – including Matthew Desmond and Mitchell Duneier – to verify their own work.

Timmermans would have a better point if I had set out to discredit every statement ever made in ethnography, but I intended nothing of the sort. Rather, my objective was to investigate whether ethnography’s characteristic reliance on unverified accounts may sometimes produce misinformation. I reported everything I found, including instances – about half the time – where my doubts were not confirmed. In any other academic discipline, my findings would have provoked less umbrage and more reinvestigation. In ethnography, not so much.

My book was published in late 2017. It would have required very little shoe leather to prove me wrong, if I really was wrong. It would take only a few days for a graduate student to find out whether there are actually police cordons at the doors of Philadelphia hospitals, checking the IDs of young Black men for outstanding warrants (there are not). It would take a little longer to locate a group of Mississippi adolescents, in the 21st century, who are completely unaware of the existence of elevators (because there are none). Social science, like all science, can only be self-correcting if replication is taken seriously. Perhaps Timmermans can explain why that seldom happens in ethnography.

My fact-checking may have been imperfect – although no one has ever identified factual errors – but it can nonetheless help identify potential shortcomings in ethnography or raise valid questions about methodology and conclusions, which is all that I have ever proposed. It is more than a little ironic for Timmermans – a non-physician specializing in medical sociology – to reject suggestions from an outsider who doesn’t “play in our playground.” He needs to spend less time in the sandbox and more time brushing up on math.

Comments

[anon]

I was immediately struck by this: "I have realized for the past few years that sociologists tend to be thin-skinned about suggestions from outsiders, while often demonstrating surprisingly poor reading skills."

Reading the comments in the FL for a few years, and some of the posts, reminds one that projection is alive and well in the "liberal elite."

I found this statement intriguing: "Lubet finds ambiguity/unable to factcheck: 10."

Now, Lubet states that "What Timmermans did not choose to recognize – although clearly stated multiple times in the book – was that my sample consisted of only those claims that could possibly be fact-checked."

Are these statements consistent?