• @Eheran
    link
    428 months ago

    What do you mean, all different? Most are exactly the same. The first 4 are a bit low and the last 3 a bit high, but last 2 and first also extremely wide, so irrelevant anyway. Everything else agrees, most within >99 % confidence with only slight differences on the absolute values.

    • @bhmnscmm
      link
      168 months ago

      9 of the teams reaching a different conclusion is a pretty large group. Nearly a third of the teams, using what I assume are legitimate methods, disagree with the findings of the other 20 teams.

      Sure, not all teams disagree, but a lot do. So the issue is whether or not the current research paradigm correctly answers “subjective” questions such as these.

      • @Eheran
        link
        78 months ago

        If we only look that those with p <0.05 (green) and with 95 % confidence interval, then there are 17 teams left. And they all(!) agree with more than 95% conference.

        • @[email protected]
          link
          fedilink
          38 months ago

          And you missed the pint in the very article about how p value isn’t really as useful as it’s been touted.

          • @Eheran
            link
            48 months ago

            That’s not the point, which is that the results are indeed mostly very similar, unlike what OP claims.

            I never said that only looking at p values is a good idea or anything else like that.

        • @bhmnscmm
          link
          28 months ago

          So ignore all non-significant results? What’s to say those methods result in findings closer to the truth than the methods with no significant results.

          The issue is that so many seemingly legitimate methods produce different findings with the same data.

    • Ragdoll XOP
      link
      58 months ago

      I wish science was a simple as taking the mean and confidence intervals.

  • @[email protected]
    link
    fedilink
    158 months ago

    The chart sure makes it look like there was an overall consensus that refs are about 1.5x as likely, though.

  • @[email protected]
    link
    fedilink
    English
    118 months ago

    Obligatory link to Statistics Done Wrong: The Woefully Complete Guide, a book on how statistics can and has been abused in subtle and insidious ways, sometimes recklessly. Specifically, the chapters on the consequences of underpowered statistics and comparing statistical significance between studies.

    I’m no expert on statistics, but I know enough that repeated experiments should not yield wildly different results unless: 1) the phenomenon under observation is extremely subtle so results are getting lost in noise, 2) the experiments were performed incorrectly, or 3) the results aren’t wildly divergent after all.

    • @[email protected]
      link
      fedilink
      48 months ago
      1. the whole point of statistics is to extract subtle signals from noise, if you’re getting wildly different results, the problem is you’re under-powered.

      Thanks for taking the time to post these links, just letting you know you’re efforts have benefited at least one person who’s gonna enjoy reading this.

  • @[email protected]
    link
    fedilink
    78 months ago

    I really found this out while writing my essay. If I wanted to I could interpret it slightly differently, resulting in totally different results.

  • @[email protected]
    link
    fedilink
    3
    edit-2
    8 months ago

    Scientists who fiddle around like this — just about all of them do, Simonsohn told me — aren’t usually committing fraud, nor are they intending to. They’re just falling prey to natural human biases that lead them to tip the scales and set up studies to produce false-positive results.

    Since publishing novel results can garner a scientist rewards such as tenure and jobs, there’s ample incentive to p-hack.

    I mean really, making claims they aren’t committing fraud yet in the very next paragraph demonstrates their motivation… To commit fraud

    Nevermind the numerous cases of published papers being bunk. And that something like 80% of published science isn’t reproduceable…which is part of what publishing is to enable.

  • Zagorath
    link
    fedilink
    28 months ago

    Why have 4 of the studies seemingly not used error bars at all‽ Like I get that different analyses will arrive at different results, but they should always have error bars, right?