Very interesting. I wonder if there is any website or resource that would allow a layperson to see how a specific study or even concept is considered by scientists in the relevant field. I think there are plenty of articles about studies that get a lot of traction on social media or even in traditional media that are perhaps quite preliminary or are not well designed and very quickly a good chunk of the population is using them to extrapolate way too much and considering it borderline gospel. It would be great to have some kind of a resource where you could look up a specific study and have scientists commenting or rating any concerns around reliability or replicability or the overall impact of the study.
As a neurologist, who has recently specialised in Long Covid. I’d say 95% of studies in the field are untrustworthy.
As with any health condition where the biological factors are unknown, a group of psychiatrists and psychologists, the same that targeted autoimmune diseases before they were proven to be “real”, have flooded the field claiming patients are suffering from hysteria or psychological issues.
None of their studies manage to prove anything whatsoever. They have published thousands of studies which go something along the line of: “People with long covid have more mental health issues, therefore, long covid is not real, but caused by hysteria”.
This proves no correlation whatsoever though. Every single chronic illness / disability has higher rates of mental health issues, yeah turns out being poor and marginalised + loosing a lot of what you had in life + chronic pain is not good for your mental health. In fact studies have found that before the covid infection these people have the same rate of mental illness as the normal population.
This is just one example, but legitimately > 90% of psychiatry / psychology studies I read in my field are extremely poorly done to the point of being worthless or even harmful.
There’s also a problem in that genuine biological studies get reported on really poorly by the mainstream media. I co-authored a paper showing that in 3 tests of cognitive function, spread over three days, people with long covid did not preform worse than the general population. However, if you made them do these three tests in a row, people with long covid would preform markedly worse than healthy controls, so they have a problem with cognitive endurance.
A handful of news outlets reported on this, and oftenso the news articles did not include the second part of the study, just saying “people with long covid do not have cognitive dysfunction” or “brain fog debunked”. There’s really a push from the right to delegitimise long covid and it is hurting scientific integrity hard.
Science reporting is a major related problem for sure. Science “journalism” is often barely more than reporting on a sentence in the abstract and then putting opinions about what that means. They report on single studies that show tiny effect sizes, and then use that to say things like “people who eat potatoes are better at pickleball” and often ignore statistical significance and effect size. Then they’ll do the same thing with a study that says the opposite, and we wonder why people don’t understand or trust science.
This isn’t what you were asking about, only tangential related at best, but I really like the work that the guys over at Data Colada do. Example link: http://datacolada.org/104
More journals are including the peer reviews at publication. There is also a site called pubpeer which is pretty much exactly what you’re thinking of, but usually only more high impact articles, or controversial articles get comments
Some have advocated that Bayesian methods should replace p-values. This has not happened on a wide scale, partly because it is complicated and partly because many users distrust the specification of prior distributions in the absence of hard data.
So instead we trust the frequentist methods cooked up a hundred years ago by a handful of statisticians trying to manufacture empirical support for eugenics.
