Also, Brandolini's law. And Adam Smith's law of supply and demand. When the ability to produce overwhelms the ability to review or refute, it cheapens the product.
Only in stupid university leaderships is that truly what gets you hired or promoted. It's simply not true. Junior researchers in fact are believing it stronger than the facts actually support. Yes, you have to have a solid amount of publications, but doing a ridiculous amount of low-impact salami-sliced stuff or getting your name on a ton of papers where you did no real work is not going to win you a job. People who evaluate applications also live in this world and know that these metrics are being gamed. It's a cat and mouse game but the cats are also paying attention. You can only play this against really dumb government bureaucracies that mechanically give points for publications and have hard numerical criteria etc. Good institutions don't do that.
There was this guy, well connected in the science world, that managed to publish a poor study quite high (PNAS level). It was not fraud, just bad science. There were dozens of papers and letters refuting his claims, highlighting mistakes, and so... Guess what? Attending to metrics (citations, don't matter if they are citing you to say you were wrong and should retract the paper!), the original paper was even more stellar on the eyes of grants and the journal itself.
It was rage bait before Facebook even existed.
If the fraudsters “fail to replicate” legitimate experiments, ask them for details/proof, and replicate the experiment yourself while providing more details/proof. Either they’re running a different experiment, their details have inconsistencies, or they have unreasonable omissions.
We can't look for failed replication experiments if none exist.
the effort to publish a fraudulent study is less (sometimes much less) than the effort to replicate a study.
>>95% of the time, the fraudsters get off scot-free. Look at Dan Ariely: Caught red-handed faking data in Excel using the stupidest approach imaginable, and outed as a sex pest in the Epstein files. Duke is still giving him their full backing.
It’s easy to find fraud, but what’s the point if our institutions have rotten all the way through and don’t care, even when there’s a smoking gun?
Machine Learning papers, for example, used to have a terrible reputation for being inconsistent and impossible to replicate.
That didn't make them (all) fraudulent, because that requires intent to deceive.
So the answer is that we still want to see a lot of the papers we currently see because knowing the technique helps a lot. So it’s fine to lose replicability here for us. I’d rather have that paper than replicability through dataset openness.
This doesn't mean the model only works on that specific dataset - it means ML training is inherently stochastic. The question isn't 'can you get identical results' but 'can you get comparable performance on similar data distributions.
By definition, they involve variance that cannot be explained or eliminated through simple repetition. Demanding a 'deterministic' explanation for stochastic noise is a category error; it's like asking a meteorologist to explain why a specific raindrop fell an inch to the left during a storm replication.