Daniel Kahneman issued an open letter to researchers doing social priming research, which has become the subject of skepticism after some studies were found to be fabricated and others were not able to be independently replicated. His letter offers advice to scholars about how to address the situation: Find out the truth through extensive replication and announce it.
The text of the letter is below:
From: Daniel Kahneman
Sent: Wednesday, September 26, 2012 9:32 AM
Subject: A proposal to deal with questions about priming effects
Dear colleagues,
I write this letter to a collection of people who were described to me (mostly by John Bargh) as students of social priming. There were names on the list that I could not match to an email. Please pass it on to anyone else you think might be relevant.
As all of you know, of course, questions have been raised about the robustness of priming results. The storm of doubts is fed by several sources, including the recent exposure of fraudulent researchers, general concerns with replicability that affect many disciplines, multiple reported failures to replicate salient results in the priming literature, and the growing belief in the existence of a pervasive file drawer problem that undermines two methodological pillars of your field: the preference for conceptual over literal replication and the use of meta-analysis. Objective observers will point out that the problem could well be more severe in your field than in other branches of experimental psychology, because every priming study involves the invention of a new experimental situation.
For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. These people have now attached a question mark to the field, and it is your responsibility to remove it.
I am not a member of your community, and all I have personally at stake is that I recently wrote a book that emphasizes priming research as a new approach to the study of associative memory – the core of what dual-system theorists call System 1. Count me as a general believer. I also believe in a point that John Bargh made in his response to Cleeremans, that priming effects are subtle and that their design requires high-level skills. I am skeptical about replications by investigators new to priming research, who may not be attuned to the subtlety of the conditions under which priming effects are observed, or to the ease with which these effects can be undermined.
My reason for writing this letter is that I see a train wreck looming. I expect the first victims to be young people on the job market. Being associated with a controversial and suspicious field will put them at a severe disadvantage in the competition for positions. Because of the high visibility of the issue, you may already expect the coming crop of graduates to encounter problems. Another reason for writing is that I am old enough to remember two fields that went into a prolonged eclipse after similar outsider attacks on the replicability of findings: subliminal perception and dissonance reduction.
I believe that you should collectively do something about this mess. To deal effectively with the doubts you should acknowledge their existence and confront them straight on, because a posture of defiant denial is self-defeating. Specifically, I believe that you should have an association, with a board that might include prominent social psychologists from other field. The first mission of the board would be to organize an effort to examine the replicability of priming results, following a protocol that avoids the questions that have been raised and guarantees credibility among colleagues outside the field.
The following is just an example of such a protocol:
* Assemble a group of five labs, where the leading investigators have an established reputation (tenure should perhaps be a requirement). Substantial labs with several students are the most desirable participants.
* Each lab selects a recent demonstration of a priming effect, which they consider robust and most likely to replicate.
* The board makes a public commitment to these five specific effects
* Set up a daisy chain of labs A-B-C-D-E-A, where each lab will replicate the study selected by its neighbor: B replicates A, C replicates B etc.
* Have the replicating lab send someone to see how subjects are run (hence the emphasis on recency – the experiments should be in the active repertoire of the original lab, so that additional subjects can be run with confidence that the same procedure is followed).
* Have the replicated lab send someone to vet the procedure of the replicating lab as it starts its work
* Run enough subjects to guarantee power (probably more than in the original study)
* Use technology (e.g. video) to ensure that every detail of the method is documented and can be copied by others.
* Pre-commit to publish the results, letting the chips fall where they may, and make all data available for analysis by others.
This is something you could do quickly, and relatively cheaply. The main costs are 10 trips, and funds to cover these costs would be easy to get (I have checked). You would have to be careful in selecting laboratories and results to maximize credibility, and every step of the procedure should be open and documented. The unusually high openness to scrutiny may be annoying and even offensive, but it is a small price to pay for the big prize of restored credibility.
Success (say, replication of four of the five positive priming results) would immediately rehabilitate the field. Importantly, success would also provide an effective challenge to the adequacy of outsiders’ replications. A publicly announced and open effort would be credible among colleagues at large, because it would show that you are sufficiently confident in your results to take a risk.
More ambiguous results would be painful, of course, but they would still protect the reputations of scholars who sincerely believe in their work – even if they are sometimes wrong.
The protocol I outlined is just an example of something you might do. The main point of my letter is that you should do something, and that you must do it collectively. No single individual will be able to overcome the doubts, but if you act as a group and avoid defensiveness you will be credible.
In response to Ed Yong’s article in Nature in which Kahneman’s letter is published, Norbert Schwarz, has written a response, saying “There is no empirical evidence that work in this area is more or less replicable than work in other areas.”
REFERENCE
No comments:
Post a Comment