Symposium on Interrogating Ethnography

The journal Contemporary Sociology, published by the American Sociological Association, recently featured a symposium on Interrogating Ethnography, with reviews by three prominent sociology professors. Their opinions covered a broad spectrum, ranging from a reviewer who did not much care for the book to one who really, really hated it. On the plus side, they all allowed that my critique of ethnography requires acknowledgement and discussion in the field. On the negative side, the reviews confirm my observation that many sociologists do not appreciate criticism, or even questions, from outsiders.

Unfortunately, Contemporary Sociology is behind a paywall, although most academic libraries should be able to get access to the articles. In any case, here are links where you can at least see the abstracts and opening paragraphs:

Tim Hallett, "Bits and Pieces of Ethnographic Data on Trial"

Timothy Black, "Verisimilitude and the Art of Ethnography"

Stefan Timmermans, "Hypocriticism"

I am linking the essays here because I think that academics benefit from recognizing critical, and even hostile, commentary. Having blogged a lot about ethnography, including links to favorable reviews, it seems only right to acknowledge the negative reviews as well.

Needless to say, I do not agree with most of the criticism -- although there are some arguments I would concede, and in one instance the reviewer is absolutely right about a failing of mine -- but I don't see the need to engage in any general rebuttal. I do, however, want to address a couple of points that strike me as more broadly relevant.

Tim Hallett, of Indiana University, expresses his concern that I conducted my fact checking "apparently without IRB approval," adding, in a footnote, "There is a separate essay to be written on this topic and the ethical implications and responsibilities of scholars who do research that aims to interrogate the confidential data collected by other scholars with IRB approval."

IRBs are Institutional Review Boards, whose approval is necessary before conducting human subjects research in the physical, social, and behavioral sciences. However, IRB approval is only required for "systematic investigations" designed to result in "generalizable knowledge." Seeking specific investigatory information about certain events or occurrences -- as I did, for example, by contacting some Philadelphia hospital employees and Mississippi school teachers -- is quite obviously not "human subjects" research. My informants were witnesses, not subjects.

The relevant HHS regulation is 45 CFR §46.102 (l)(1), which specifically excludes:

Scholarly and journalistic activities (e.g., oral history, journalism, biography, literary criticism, legal research, and historical scholarship), including the collection and use of information, that focus directly on the specific individuals about whom the information is collected.

Even so, I would welcome an essay on the ethics of interrogating others' ethnographic data. I had such an exchange with Beth Mertz, a law professor and anthropologist at the University of Wisconsin, here and here, which in part spurred me to write Interrogating Ethnography. I am sure there is more to be said.

Stefan Timmermans does not like my methods.  "If you are going to play in our playground, you have to do the real shoe leather work. Get into the hospital and observe the ways that police and staff do favors for each other," he says, in reference to my fact-checking the claim of police cordons outside Philadelphia's emergency rooms. There are at least four things wrong with Timmermans' assertion.

First, he seems to  to think that only ethnographers can critique ethnographers. In order to play in their playground, you have to do exactly what they do. Thus, he rejects the whole idea that other methods can be used to validate or disconfirm the claims of ethnographers, which apparently have to be taken on faith by everyone outside the field. The implication here is that an ethnographer may be privy to some sort of occult knowledge, unavailable to non-initiates, that is not subject to verification through ordinary sources.  

Second, he ignores the fact that ethnographers themselves seldom, if ever, reinvestigate each other's claims, with shoe leather or otherwise, and that the unnecessarily extensive use of site-masking makes much reinvestigation difficult or impossible. I first raised the issue of Philadelphia hospitals in 2015, and I have since noted the many ways the story of police check-points has been negated -- not only by me, but also by investigative reporters. I have called upon ethnographers -- of whom there are many in and near Philadelphia -- to revisit the likely hospitals, but no one has ever done so. In a more rigorous discipline, someone surely would have checked it out, which informs part of my critique.

Third, note how Timmermans recasts the issue. I questioned improbable claims about (1) a constant police gauntlet outside hospital emergency rooms, checking the IDs of every young African American man walking through the door, (2) police regularly reviewing patient and visitor lists to check against outstanding warrants, and (3) the arrest of three new fathers on a maternity floor on a single night. Timmermans restates it as "police and staff do[ing] favors for each other," which is not at all the same thing. Timmermans is himself a medical sociologist, so it is significant that he does not appear willing to verify any police behavior beyond trading favors with staff.

Finally, the demand for "shoe leather work" is a form of ableism. I am not kidding about this, as regular readers will understand.

[NOTE: The manuscript for Interrogating Ethnography was reviewed by five prominent sociologists, none of whom raised IRB issues. I did, however, consult the Northwestern IRB regarding one aspect of the project, and was told that approval was not necessary.]

Comments

[Anon]

Steve

I admire your work, and I think your book is great. But there is a substantive difference between calling officials to verify information and actually visiting a place to observe information yourself. It is a bit naive to expect public officials or hospital staff to verify a practice they would want to hide. I think your view of evidence gathering is common among lawyers who are often accustomed to gathering information this way. I’m not saying it is invalid but that it does not give the whole picture. In this case you may be right but observational data are different and sometimes more valuable than “fact-checking” of the type you describe.

[anon]

Steve

I too think that your work is valid and relevant, even if in another's sandbox. However, there are also valid concerns about dilettantism in legal academia.

Often, persons with only a JD, which, as I hope we all realize, ain't much of a degree in anything in particular, believe they can question the competence of persons with far more expertise, e.g., physicians. After all, lawyers can sue doctors for malpractice, shred them on cross, etc. THis leads to a belief for some that, with a JD, one is capable of the mastered every nuance of every profession.

Of course, this isn't so. In legal academia, this false belief is especially troubling. If a law professor studies history, then that law professor is a "historian" - but this isn't the same thing as holding a professorship in the History Department. If a law professor studies economics, then that professor is a "economist" - but this isn't the same thing as holding a seat in the Department of Economics. Too often, law professors, with just their JD and a whole lot of free time, fancy themselves to be something they are not.

You applied a very rigorous test to a young scholar's work. You made it seem that she had lied and committed crimes. At least, that's how I read your blog posts, and forgive me if I've overstated this. Those were my impressions.

It seems you've taken off from there to basically question an entire field of scholarly endeavor. You've mined some important insights. However, I believe you should remember that you aren't an expert in this field (again, forgive me if you've obtained any degree, or had any formal training whatsoever in the field you are attacking).

Just as lawyers can uncover malpractice and perhaps improve medicine, your work is valid. But, again, no one, I think, could say that just because a lawyer won a malpractice suit, that lawyer is an expert in medicine.

[anon]

One more point. Steve, you are approaching this topic - explicitly or implicitly - as a matter of the standard of care.

As I think you know, the standard of care for a professional is usually established by expert testimony: an expert in the same field! To be sure, in egregious cases, no expert will be required. But, you are starting to tread into closer cases.

Are you an expert in the field of ethnography? If not, please don't pretend to be one and act as prosecutor, judge and jury. Interrogate? Ok, fine. Ask questions.

The problem here is when you start answering in a way that condemns a whole field of scholarly endeavor while lacking the necessary education, training, professional and life experience.

[anon]

first anon here. I don't agree with either of the other comments. the second anon 's criticism is just "you're not a X so therefore you're not qualified to opine on X field." I think this is a shallow criticism. I also think it is particularly odd to claim that the JD "ain't much of a degree in anything in particular." The idea that law professors tend to overstep their expertise probably has some merit. But the idea that the degree doesn't provide any training for critical thinking--or that the law professor/lawyer cannot offer valuable insights or learn another field--is preposterous. The deeper criticism of Steve's work is he thinks that the lawyers' tools for fact-checking are the best tools, or even valid tools. But his general point and some of his specific criticisms are spot on. And ethnographers are really doing themselves no favors when they circle the wagons and cry foul. Despite the flaws in his approach, Steve's work sheds light on some questionable practices. I don't think he said anywhere that someone committed a crime, or anything close to it. People should really take a hard look at what they are doing. Many ethnographers paint deeply sympathetic portraits, and have biases that should be outed and expunged whereever possible.

[anon]

anon

Do a quick search on this blog. Read Lubet's post on June 15, 2015. Correct the record, anon. Or, at least admit that perhaps you really weren't familiar with the backstory of the debate here before accusing others of misstating the facts.

As for your attempted mischaracterization of what I've said above, I can only say that, once again, you are seemingly not reading. Much of what you say, actually, is merely restating my points. Thank you.

[MyPhDisbetterthanyours]

The irony here is delicious. Second Anon responds to the challenge that (a subset) of a given discipline's methodology is (at least somewhat) contestable with the claim that the challenger's own professional training is inadequate to allow him or her to proffer any serious challenges to other disciplines.

"You can't possibly mount a significant and comprehensive methodological criticism of Y because you have X-style education (and, presumably/implicitly, your education and critical reasoning training is limited to, and exhausted by, your X-style education). However, if you have had training in Y, then I apologize (and, presumably, my criticisms are moot???)."

One doesn't need a philosophy PhD to know that THAT is an ad hominem fallacy. And yes, appealing to a knowledge threshold (i.e., one obtainable EXCLUSIVELY via PhD training in Y) as a basis for bypassing the ad hominem charge fails here.

Second Anon: does your charge in turn give rise to a pro tanto reason for thinking that sociologists aren't very clever, or that your own particular training (in whatever it is you studied) was subpar?

[anon]

MyPhD

You are mistaken in labeling your fictional quote as an "ad hominem fallacy." Perhaps you will want to learn how to apply that charge properly, or make up a different quote. In general, your reading comprehension, if you are addressing what I said above, is poor indeed! Your comment has nothing whatsoever to do with anything that this commenter stated!

Finally, know this. Unlike some, I don't personally find obnoxious attacks, like yours, are justified without any basis. It does not appear that Steve has, in fact, any professional training in ethnography. to the contrary, his point seems to be that, like a plaintiff's attorney, he can identify malpractice without it: a point I conceded above. However, I haven't studied Steve's background. If he has any such training, one would think he would have said so by now, but, one never knows for sure without research.

Before an expert takes the stand to inform the jury, the burden is on the proponent to qualify the expert.

SUffice it to say that Steve's more general criticisms of ethnography appear to this reader to be devoid of any such training and experience, and tend to prove, in my view, that he lacks the requisite expertise and likely could not qualify as a expert in the field of ethnography (just as most plaintiff's attorneys are not medical experts). Again, the burden on Steve to show otherwise, IMHO. This is an impression.

Underscoring this impression, Steve's relentless attack of Goffman took on, again, in my view, an unseemly, hyperbolic tone. This again, seemed to be evidence of a lack of expertise and professionalism.

[Jeff]

If I may weigh in on this debate which has been going on for some time and never fails to fascinate. Research methods are a form of applied epistemology in which data is collected and conclusions are drawn. Some of use historical data, statistical data, data easily subjected to statistical analysis, data that is comparative, and ethnographic data which may or may not include some or all of the above. Ethnography involves collecting data via conversation with informants. (When I was a graduate student in the 70s we were required to put all our field notes into the university library for anyone who wished to check out conclusions, an innovation which struck me at the time as common sensical). As I used multiple kinds of sources in my graduate career I had to learn the "rules of the road" for each one and the main rule for ethnography was to figure out how to deal with the Rahomon problem or otherwise put, how to keep the goalposts reasonably steady for making truth claims. In other words, we were not writing novels, we were making truth claims. And we notated a big and interesting middle ground between fiction and truth claims, some said X, others said Y. The task was to evaluate claims against other forms of data. Then we made an estimation. Ethnographers desperately seek to retain the legitimacy of their method which, having read "Investigating Ethnography" cover to cover, is not dismissed by Lubet willy nilly. While he avoids the philosophical debates (such as over positivism v. verstehen), he makes a more grounded case for not saying something as a fact if it isn't, at least investigate it as an example of false consciousness. If people think that probation disallows you to have a drivers license and it isn't true, find out why people do think its true and add that story into the narrative. The value of Lubet's book lies far beyond Goffman whose book is both weak and probably more right than wrong in the final analysis. He is asking social scientists to apply a critical eye on ethnography as they would on any other methodological option available. To do otherwise, is to invoke both Lubet and Mertz, "circling the wagons".

[Steve L.]

Jeff: The notion that one must be an ethnographer to critique ethnographers is untenable. So is the suggestion that ethnography must be evaluated solely by its own standards. There are many other fact-seeking professions -- including law, history, and journalism -- all of which have developed means of gathering and evaluating information. Cross-fertilization is essential in any academic or scholarly pursuit. Lawyers have benefitted from listening to sociologists since the days of Louis Brandeis; ethnographers, likewise, have at least something to learn from lawyers, even if they do not want to admit it.

[MyPhDisbetterthanyours]

Hi again Anon,

It certainly is an ad hominem.

Charging that professional training in Y is a sine qua non for engaging in criticisms of Y's methodology is a token of the type; you speak to the man, not to the content of his claims. Another irony is that your criticism of Steve is obnoxious for this reason!

Now, if his substantive claims were indeed off the mark, then a partial or complete explanation may be his lack of adequate understanding of the methods, which could in turn be attributed to a lack of sufficient training. But that isn't actually what was levied in the/your posts above; you didn't even attempt to establish that the content of his claims were erroneous.

This may help you: https://leiterlawschool.typepad.com/leiter/2013/11/what-an-ad-hominem-argument-is-and-isnt.html

This is fun, though! What additional dubious claims will you now assert to further put your own lack of critical reasoning skills on display? (I'll simply put aside your claim that the - putative - use of hyperbole evidences a lack of expertise as too stupid to be taken seriously and too easy a target for refutation).

[Patrick S. O'Donnell]

Just an aside (although I think the criticisms of Steve’s work in this thread falls wide of the mark and are patently absurd), but one should keep in mind that an ad hominem argument is only presumptively fallacious from the vantage point provided by informal logic and the corresponding notion of informal fallacies (‘informal’ in this case meaning not invariably fallacious; this is the case with most if not all informal fallacies, a fact often forgotten or neglected). There are two main types: abusive and circumstantial, and an analysis of it specific employment in the discourse in question is required to determine if the particular argument involves a fallacious or non-fallacious use of the ad hominem argument. There is a nice introduction to this in chapter 6 of Douglas Walton’s Informal Logic: A Handbook for Critical Argumentation (Cambridge University Press, 1989).

[anon]

If the delicious nature of this statement were not enough for readers -- "your claim that the - putative - use of hyperbole evidences a lack of expertise [is] too stupid to be taken seriously and too easy a target for refutation" -- remember, this statement is coming from the "MyPHDisbetterthanyours": a critic of ad hominem arguments! You are stupid, therefore your argument is wrong! Great understanding of the ad hominem fallacy, oh Better than Everyone Else.

Claiming that someone lacks the knowledge and expertise to make a claim is not an ad hominem argument, oh great one. In the law, when a question about the ability of a person to opine is relevant, the inquiry is legitimate, and not based on a fallacy. Never saw My Cousin Vinny, did you?

A plaintiff's attorney can, as stated above, shred a doctor on the stand, even if the plaintiff's attorney could not, in turn, qualify as a medical expert. Steve purports to ground his criticisms of ethnography in legal standards.

Frankly, in this regard you sound sort of ignorant. Not stupid: ignorant. See how that works? Saying you are ignorant isn't ad hominem, oh greater than everyone else, because you are! It is simply an observation about your demonstrated lack of knowledge and seeming unwillingness to read the comments causing you to so vociferously attack the commentator (not the arguments, of course, because you are an ad hominem debater!)

[MyPhDisbetterthanyours]

Anon, you simply aren't good at this.

Your claim above is about some inadequacy in Steve's views can be due to a lack of CREDENTIALS. You didn't know what training he has, remember? "(again, forgive me if you've obtained any degree, or had any formal training whatsoever in the field you are attacking)." More importantly, it doesn't speak to the MERITS of Steve's propositions - whether or not he be trained (to some standard) in ethnographical methodology, or employs exogenous methods in furtherance of, or developing, his claims. (Furthermore, a presumption that only endogenous methodological criticisms are valid begs the question). You certainly haven't established his claims' inadequacy, let alone shown how his lack of experience or (inside) knowledge has driven any such errors.

Per Patrick's post, my "Now, if..." paragraph might have given you an out, i.e., a lack of knowledge and expertise served as part of a causal explanation of diagnosed errors, if any - IF you'd addressed the substance and merits and established any such errors, which you do not. (Seriously, take a gander at the references Patrick and I provided here).

Will you now deign to even implicitly defend your "Hyperbole Evidences Inexperience" proposition? This is too good to be true!!! (Once you read the aforementioned sources, moreover, ask yourself whether my remarks in that regard are in turn an instance of the fallacy, or an insult levied at your CLAIM and as potential evidence of your reasoning skills).

Note the further irony that you now assert that I don't attack the arguments, but only the commentator! It's like shooting fish in a barrel with you, Anon. You are genuinely a barrel of laughs. Thank you!

[MyPhDisbetterthanyours]

Just one last remark, since I'm fairly certain you don't actually follow, Anon. Imagine this: what if Steve had actually had years or decades of INFORMAL training? What then? Would you still apologize?

[anon]

IF he does, he'd say so. Read my comment above.

what is funny is your hyperbolic attacks (yes, the hyperbole does demonstrate a lack of merit: like cursing to get a point across, your rants don't address any relevant issue). Sift thru the screeds above, and all we have is ad hominem (and the infantile "Your so stupid I don't even need to say why.").

You obviously have not been following the path that Steve has: from AG to here. You are just demonstrating your ignorance of the backstory, the law, of Steve's main critique and the nature of this debate.

Yet, you are hurling insults like a third grader.

What is most striking is your poor reading comprehension.

HolierThanThou, I'm finding your comments funny too, but, you are just repeating yourself and the nastier you get the less entertaining are your attacks.

[MyPhDisbetterthanyours]

The point is, instead, that, alone, it would be irrelevant to the merits (or demerits) of his claims.

No, Anon. Instead, you simply do not - and perhaps cannot - understand why your own argument is off the mark. (And projecting about insults and ad hominems won't help your cause).

Would you please also do the academic world a favor and decline from ever peer-reviewing anyone else's work? One shudders at the atrocities you would commit.

[anon]

HolierThanThou, who do you think readers would more fear: a hyperbolic, juvenile, insult throwing megalomaniac who says he's better than others, etc., or someone who praises Steve's work in general, identified the advancements he might foster, but cautions him against posing as a subject matter expert in ethnography.

Steve's post was about, among other things, the reactions of so many in that community, who perceive him to be going too far in posing as a light in the darkness of ethnography. I tend to think they have a point, which Steve really hasn't addressed.

Steve characterizes their claims (as HolierThanThou characterized mine) by stating: "First, he [a critic] seems to think that only ethnographers can critique ethnographers."

That canard is misdirection. I'm not saying that, and I don't think the chorus of critics are saying that. That retort deeply misses the point and is just like HolierThanThou's repeated resort to the straw man argument.

The overall point here is that ethnography is not a trial to determine legal liability or criminal culpability. If Steve wanted to dine out for years on the claim that some ethnographer inadvertently stumbled into a confession of a heinous crime (a totally bogus charge, in my view) then he arguably possessed the knowledge to so opine, even if he was wrong. If he had a bigger point about truth telling (about the hospital, for example), then his observations may have value that can't be cavalierly dismissed.

But, if Steve misses, for example, the fact that an "ethnographer" may have become too involved in her field work, and who may not have distinguished, as Jeff noted above, between what was "factual" and what was "believed", if he fails to see that perhaps by that reason she became a part of the story rather than telling it, that may be because Steve isn't an expert in ethnography. That seems to be the objection. (I'm not prepared to resolve it and Steve is free to show the basis for his claim to be a subject matter expert.)

Again, to the extent that Steve has made positive contributions and pointed out some flaw in the standard of care in the community of ethnography, good for Steve. Lawyers are good at opining on the standard of care.

But, his expertise in the law only takes him so far. IF Steve went to prove his standard of care in court (the standard that HE wants to live by) I don't think any court would allow him to testify. Period. If I'm wrong, Steve, I'd like to hear your view of this.

Ethnography, again, is not an adversarial process designed to ferret out the "truth" and, moreover, as any experienced trial lawyer will tell you, with all the rules of evidence, etc., the "law" often fails as well.

There are deeper issues involved in ethnography that don't lend themselves to law professor dilettantism. If Steve wants so live by his legal expertise, then he is stuck with it. In the law, we voir dire putative experts before we hear their opinions. I'm waiting for Steve to make the proffer.

Of course, he need not do this. But, just saying "Everybody says I need to be an expert and I don't need to be, so there, because I'm just really, really smart and I talked to some ethnographers who agree with me" isn't very convincing.

I'm sure that holier than thou will regale us now with some brilliant "Triumph" like commentary. If only we could see the puppet and cigar.

[MyPhDisbetterthanyours]

"Often, persons with only a JD, which, as I hope we all realize, ain't much of a degree in anything in particular, believe they can question the competence of persons with far more expertise, e.g., physicians... It seems you've taken off from there to basically question an entire field of scholarly endeavor. You've mined some important insights. However, I believe you should remember that you aren't an expert in this field (again, forgive me if you've obtained any degree, or had any formal training whatsoever in the field you are attacking)... Just as lawyers can uncover malpractice and perhaps improve medicine, your work is valid. But, again, no one, I think, could say that just because a lawyer won a malpractice suit, that lawyer is an expert in medicine."

Now you make yourself out to be a liar, not just a fool. You aimed to delimit the scope and merits of his critique based on credentials. You also presume (or betray the presumption) that the epistemic claims are delimited to law and legal methodology.

Who do you you think my handle is in reference to here? You. You, Mr./Ms. "you're not a real ethnographer so don't pretend that you can offer any SERIOUS or systemic methodological critiques."

You can try to change the subject about the ad hominem, but you won't fool anyone.

I weep for the academy that you form part of it.

Good night, Anon.

[anon]

Wow. Add name calling to your incoherent rants.

Throw in some jargon, and there you are: a blowhard who doesn't have any arguments.

"You're stupid, you're a liar, you're a fool ..."

And, all this coming from someone who isn't even aware of what the debate is about, and who constantly complains about ad hominem attacks.

What a piece of work! Steve, you should be thinking about the "support" you are attracting and wondering whether you are being well-defended!

I sense it is time to shut this thread down, lest MyBetterThanYou goes off in an even more unseemly way.

Remember folks, this person purports to speak for the academy.

Are you all proud of this?