Of course I do! Not all of course, and taking (subjectively measured) impact into account. "We tried to replicate the study published in the same journal 3 years ago using a larger sample size and failed to achieve similar results..." OR "after successfully replicating the study we can confirm the therapeutic mechanism proposed by X actually works" - these are extremely important results that are takin into account in meta studies and e.g. form the base of policies worldwide.
More than anything. That might legitimately be enough to save science on its own.
(I am not seriously proposing this, but it's interesting to think about distinguishing between the very small amount of truly innovative discovery versus the very long tail of more routine methods development and filling out gaps in knowledge)
But they don't, and that's the problem!
In my own experience I was unable to publish a few works because I was unable to outperform a "competitor" (technically we're all on the same side, right?). So I dig more and more into their work and really try to replicate their work. I can't! Emailing the authors I get no further and only more questions. I submit the papers anyways, adding a section about replication efforts. You guessed it, rejected. With explicit comments from reviewers about lack of impact due to "competitor's" results.
Is an experience I've found a lot of colleagues share. And I don't understand it. Every failed replication should teach us something new. Something about the bounds of where a method works.
It's odd. In our strive for novelty we sure do turn down a lot of novel results. In our strive to reduce redundancy we sure do create a lot of redundancy.
That sort of Orwellian doublethink is exactly the problem. They need to move it forward without improving it, contribute without adding anything, challenge accepted dogma without rocking the boat, and...blech!
> challenge accepted dogma without rocking the boat
You must create paradigm shifts without challenging the current paradigm!
[0] https://www.scientificamerican.com/article/katalin-karikos-n...
[1] https://www.globalperformanceinsights.com/post/how-a-rejecte...
I can tell you that it doesn't match my own experience. I also think it doesn't match your example. Those cases of verified image fraud are typically part of replication efforts. The reason the fraud is able to persist is due to the lack of replication, not the abundance of it.
I'm pretty sure most image fraud went completely unrealized even in the case of replication failure. It looks like (pre AI) it was mostly a few folks who did it as a hobby, unrelated to their regular jobs/replication work.
> 'm pretty sure most image fraud went completely unrealized even in the case of replication failure
> It looks like (pre AI) it was mostly a few folks who did it as a hobby
>>> Advanced groups usually replicate their competitor's results in their own hands shortly after publication
Or are you suggesting that the groups you didn't work with (and are thus speculating) are the ones who replicate works and the ones you did work with "just trust their competitor's competence")? Because if this is what you're saying then I do not think this "mostly" matches your experience. That your experience more closely matches my own.
[0] I should take that back. I started in physics (undergrad) and went to CS for grad. Replication could often be de facto in physics, as it was a necessary step towards progress. You often couldn't improve an idea without understanding/replicating it (both theoretical and experimental). But my experience in CS, including at national labs, was that people didn't even run the code. Even when code was provided as part of reviewing artifacts I found that my fellow reviewers often didn't even look at it, let alone run it... This was common at tier 1 conferences mind you... I only knew one other person that consistently ran code.
Replication of an experiment and finding image fraud are kind of done as two different things. If somebody publishes a paper with image fraud, it's still entirely possible to replicate their results(!) and if somebody publishes a paper without any image fraud, it's still entirely possible that others could fail to replicate. Also, most image errors in papers are, imho, due to sloppy handling/individual errors, rather than intentional fraud (it's one of the reasons I worked so hard on automating my papers- if I did make an error, there should be audit log demonstrating the problem, and the error should be rectified easily/quickly in the same way we fix bugs in production at big tech).
This came up a bunch when I was at LBL because of work done by Mina Bissell there on extracellular matrix. She is actively rewriting the paradigm but many people can't reproduce her results- complex molecular biology is notororiously fickle. Usually the answer is, "if you're a good researcher and can't reproduce my work, you come to my lab and reproduce it there" because the variables that affect this are usually things in the lab- the temperature, the reagents, the handling.
See https://www.nature.com/articles/503333a (written by Dr. Bissell).
All because journals prefer novelty over confirmation. It's like a castle of cards, looks cool but not stable or long-term at all.
Actually, yes, I do. The marginal cost for publishing a study online at this point is essentially nil.
The marginal cost for doing a study remains the same, which is quite a bit. Society doesn't have unlimited scientific talent or hours. Every year someone spends replicating is a year lost to creating something new and valuable.
I'm sure you can more narrowly tune your email alerts FFS.
> Replicating work is far more difficult than a lot of original work.
Only if the original work was BS. And what, just because it's harder, we shouldn't do it?
Hell yeah. We’re all trying to get that Nature paper. Imagine if you could accomplish that by setting the record straight.
I believe people will enthusiastically say yes but that they do not routinely read that journal.
"It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so."
Knowing that something I thought was true was actually false would have saved me years in several situations.
What's even your point here? Hopefully we are at least in agreement that Nature is seen as prestigious and worth looking through precisely because of the sort of content that they publish. Diluting that would dilute their very nature. (Bad pun very much intended sorry I just couldn't resist.)
This is partly why much of today's science is bs, pure and simple.
I don’t regularly read scientific studies but I’ve read a few of them.
How is it possible that a serious study is harder to replicate than it is to do originally. Are papers no longer including their process? Are we at the point where they are just saying “trust me bro” for how they achieved their results?
> Do you want issues of Nature and cell to be replication studies?
Not issues of Nature but I’ve long thought that universities or the government should fund a department of “I don’t believe you” entirely focused on reproducing scientific results and seeing if they are real
They aren't. GP was on point until that last sentence. Just pretend that wasn't there. It's pretty much always much easier to do something when all the key details have been figured out for you in advance.
There is some difficulty if something doesn't work to distinguish user error from ambiguity of original publication from outright fraud. That can be daunting. But the vast majority of the time it isn't fraud and simply emailing the original author will get you on track. Most authors are overjoyed to learn about someone using their work. If you want to be cynical about it, how else would you get your citation count up?