Home » Science » Repeatability in studies falls over time: Can you give this phenomenon a name?

Repeatability in studies falls over time: Can you give this phenomenon a name?

In “The Truth Wears Off: Is there something wrong with the scientific method?” (New Yorker, December 13, 2010), Jonah Lehrer reported,

Many results that are rigorously proved and accepted start shrinking in later studies.[ ... ]

… now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.

[ ... ]

For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved?

Suggestions for names, with rationale, gladly accepted. Also, any idea why it is happening?

  • Delicious
  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • RSS Feed

6 Responses to Repeatability in studies falls over time: Can you give this phenomenon a name?

  1. Repeatability falls over time so than does the falling of repeatability.

  2. Denise:

    I certainly don’t have a name (I don’t know, how about “decorrelation” as in “decoherence” in Quantum Mechanics?), but what I’m guessing is happening is that as information builds up, and as they use computer programs to derive correlation coefficients and such, that the ‘scatter’ of empirical data becomes less and less amenable to such constructions of correlations.

    IOW, with a limited amount of data, a ‘pattern’ emerges via statistical methods. However, do enough experiments, and the pattern begins to be lost. Why? Because basically you’re dealing with ‘noise’, but just a little bit of ‘noise’ might ‘sound like something’, to use a metaphor.

  3. I should have added this:

    “Eventually, randomness emerges.”

  4. Well, I think part of it is because the first studies are the best studies they can publish, but the repeated studies are the ones with average results. That’s my guess.

  5. 5
    EndoplasmicMessenger

    I believe the article calls it the “Decline Effect.”

    I guess its the opposite of the Placebo Effect: in this case, the researchers are no longer surprised by the finding, and therefore no longer expect it or see it.

  6. I believe the article calls it the “Decline Effect.”

    I guess its the opposite of the Placebo Effect: in this case, the researchers are no longer surprised by the finding, and therefore no longer expect it or see it.

    I believe the “opposite” of the Placebo Effect is the Nocebo Effect.
    Placebo effect is the effect that believing in a treatment improves the symptoms while the Nocebo effect is the effect of disbelieving in a treatment worsening the symptoms.

    It would maybe be fair to assume that your mind somehow influences your body.

    I am guessing that the “Decline effect” might be another variation of the same “effect”; the effect of our expectations actually affecting what we observe…

Leave a Reply