Setting the Standard for Fact-Checking Ethnography

I have just finished reading Matthew Desmond’s Evicted, which I strongly recommend to anyone who is interested in:  ethnography, social policy, urban economics, journalism, or the use of evidence in social science.  In addition to making a compelling case for meaningful changes in United States housing policy, Desmond sets a new standard for verification in ethnography.  Where others have been content to take their informants at their word, Desmond has engaged in extensive fact checking, including court and police records, statistics, medical records, and third party interviews, all of which are documented in his notes.  In addition, he retained a professional fact checker – whom he thanks by name – to compare the manuscript to his field notes.  This level of attention to the record will be familiar to lawyers, but I have not seen it before in any ethnography.

Perhaps inevitably, Evicted has drawn comparisons to Alice Goffman’s On the Run.  As some readers may recall, Goffman has steadfastly resisted source checking, telling the New York Times Magazine that

The point of the book is for people who are written off and delegitimated to describe their own lives and to speak for themselves about the reality they face, and this is a reality that goes absolutely against the narratives of officials or middle-­class people. So finding ‘legitimate’ people to validate the claims — it feels wrong to me on just about every level.’

Desmond’s book is almost exactly the opposite, explaining precisely how and where he got his facts, and “combing public records” (according to the Chronicle of Higher Education) rather than rely on unverified accounts.

Most observers and reviewers have applauded Desmond’s admirable commitment to accuracy.  Frankly, he makes many other ethnographies look imprecise or careless (at best) by comparison.  Surprisingly, the, shall we say, looser approach still has its defenders, as is evidenced in this remarkable interview of Desmond by New York Magazine’s Jesse Singal:

Your book is going to draw comparisons to Alice Goffman’s On the Run: Fugitive Life in an American City, since both books involve a white ethnographer “embedding” with poor black people, though in your book about half of your subjects were white. I’m curious if, when you saw that book’s reception and some of the controversies that popped up around it, that had any effect on you and your work on Evicted.
I think one of the things we’ve learned from the reception of that book is how deeply people care about our methods, you know, and our claims, and how we know what we know. The truth is absolutely paramount, it’s so paramount, and we have to be dogged about it and transparent and accountable to those claims. And I think that’s something that’s come out from the conversation around that book.

I was doing fieldwork long before that book was published, so it didn’t affect the work in and of itself. I think that one thing is that I had always thought about hiring a fact-checker for this book, and I did in the end. I think I probably would have done that regardless, but I think that the reception of that book influenced that decision, too.

You and Goffman have pretty different views of the fact-checking process. I interviewed her, and she said that if one of her subjects said, “I went to court facing this charge,” that would go in the book, and she thought her subjects’ understanding of their situation mattered a great deal, regardless — to a certain extent, at least — of whether it matched up with the legal reality. So some of the differences between how your book and hers approached fact-checking are a bit philosophical. But it is a resource thing, too, right? It took you a lot of money, and money in the form of time, to fact-check everything in Evicted.
It takes a lot of time. It takes an enormous amount of time.

So it’s not crazy to say that, materially, fact-checking was quote-unquote “easier” for you, as someone further along in his career and who had access to more resources, right?
I think there are ways that graduate students can fact-check their work. I think there are ways that we can do this that don’t require massive amounts of resources. So let’s say you and I were in grad school together and I was doing an ethnography — I could give you my fieldnotes and you could do the same for me, and we could fact-check [each other’s] claims, and we could write that in our publication so that we hold each other accountable for that. That could be rather costless.It does take time — it does take time. But again, like, I think we have to be obsessive about the truth and go to whatever lengths we can to get it.

So you don’t think there are ethical problems with sharing fieldnotes with another scholar just so they can help you fact-check, right? Because part of the debate here has been about the boundaries of ethical concerns and where they lie.
So in my experience, what I did is the fact-checker signed a nondisclosure agreement, because she was going to be interacting with some sensitive material and people’s real names and that kind of stuff, and then I handed everything over to her. I don’t think that’s an ethical issue. For the New Yorker excerpt, the New Yorker fact-checker interviewed Arleen and Sherrena [a landlord and entrepreneur] and other people that were in that excerpt. I called everyone and just said, “Hey, would you guys be okay with talking to a fact-checker? And they said sure. The fact-checker I hired for my book ended up conducting over 30 independent interviews with folks who were in the book and other sources to corroborate stuff. So, I dunno — I feel like those are pretty established practices in journalism.

Despite Singal’s repeated importuning, Desmond politely declines to defend Goffman’s methods.

Comments

[Grumpy]

Kurtz: Did they say why, Willard, why they want to terminate my command?
Willard: I was sent on a classified mission, sir.
Kurtz: It's no longer classified, is it? Did they tell you?
Willard: They told me that you had gone totally insane, and that your methods were unsound.
Kurtz: Are my methods unsound?
Willard: I don't see any method at all, sir.

[twbb]

"This level of attention to the record will be familiar to lawyers, but I have not seen it before in any ethnography."

How familiar are you exactly with ethnography, Steve? I've pointed this out before but you are critiquing the entirety of a 100+ year methodology based on what seems to be a very small sample size. Even if you assume Goffman was a bad actor, and her advisors/editors/defenders are too, that is still an almost imperceptible percentage of ethnographers and sociologists.

[Jeff]

If one looks at the long history of ethnographies going back to Durkheim's "Suicide" or Malinowski's work on the South Sea Islanders we have a variety of data collection methods ranging from statistical to oral interviews etc. Some classic ethnographic studies have been found empirically lacking by later research. The standard ethnographic of Africa done under the aegis of the British government would ask interview subjects whether their societies were matrilineal or patrilineal and checked off the box only to be found wrong through more rigorous examination (such as actually documenting inheritance patterns). Ethnographic studies that are viewed as brilliant at the time and later found to be 'wrong' are often held up as significant in the history of a discipline despite their accuracy. People who study non-western societies are quite used to watching important ethnographies lose their standing as sources for analyzing a particular society. In today's world with the research techniques available to us, the widespread accessibility to documents, archives, etc etc it behooves the ethnographer to make an effort either to check for accuracy of stories told by informants or document them as perceptions rather than facts. Had Goffman done that and made that gap a part of her story then this would be a different story. The issue is not the percentage of ethnographies that stand up to Mitch Duneir's standard (being cross examined by a lawyer) as "twbb" asks but what is the verification process for this research practice moving forward. Mead's "Coming of Age in Samoa" turns out to be more of a period piece than a valuable analysis of Samoa; no need to repeat that mistake.

[twbb]

Jeff, that's exactly what I'm saying; to assume that standard practice in ethnography is to take informants at their word just seems to betray a really deep-seated lack of knowledge about the methodology and its history. The suggestion that modern ethnographic practice is to just credulously go into communities and take what informants tell you as fact, or that this ethnographer was doing something just completely unheard of, just bears little resemblance to the field as I know it.

[Steve L.]

As a rule, I do not respond to anonymous comments, but twbb has raised a fair point (although I think it is based on a misunderstanding):

I have not said that "modern ethnographic practice is to just credulously go into communities and take what informants tell you as fact." That critique is specific to Prof. Goffman's work, as she herself has described it an interview with Jesse Singal. I do think that the ethnography establishment has been far too accepting of Goffman's loose approach, but I have read many other ethnographies that are more exacting.

Desmond's work, as I noted in the OP, takes it to another level. His sourcing is indeed more rigorous than I have seen in any other ethnography. His notes could easily comprise another book.

In other words, Desmond has set a new standard for fact checking, and I hope it becomes a model for the works that follow.

[twbb]

Fair enough, Steven, I don't mean to pile on you, but I do think that the statements you made pretty explicitly place your evaluation well beyond Goffman, most notably:
1. "Desmond sets a new standard for verification in ethnography" and
2. "This level of attention to the record will be familiar to lawyers, but I have not seen it before in any ethnography."

The steps Desmond takes, while laudable, just don't seem novel to me. Anthropologists have been using primary and secondary sources to validate and supplement ethnographic work for decades. See, for example, the work of E.E. Evans-Pritchard, who was doing so back in the 1940's, and who was one of the field's most influential practitioners. I would suspect that Desmond himself would not claim that he is introducing some hitherto unseen level of rigor to the field, though maybe you can reach out to him and ask him.