Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions.
This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change.
Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.
"Large controlled experiments are costly and can hurt people who opt-in to informed consent. Instead, we should do significantly, significantly larger experiments, with undefined success/failure conditions, and no informed consent."
Insane opinion
Even if that is true, is it an intrinsic problem with trials or just bad regulation? If it is the latter then you need to change the regulations? Is the problem global - is every regulator everywhere demanding byzantine trails?
Yes, that was because of things like
https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment
https://en.wikipedia.org/wiki/Stateville_Penitentiary_Malari...
https://en.wikipedia.org/wiki/Operation_Sea-Spray
https://en.wikipedia.org/wiki/Nazi_human_experimentation
I understand that certain people are salivating at the thought of a return to those times; I'm not one of them.
Wow.
A flippant dismissal in an HN comment does not actually negate reality.
This is silly.
FDA has essentially one requirement: prove that your drug is safe and effective.
The reason trial designs get more and more byzantine is because the drugs themselves work less-and-less well. They're far more nuanced and precise. The experiments have to be extremely well-controlled, and then this has to balance against cost/timeline of the trial, and that's why sponsors choose to use byzantine trial designs.
This is especially great because it puts anyone who actually wants to actually make a working drug at a significant disadvantage. It'll take them longer to get to market, cost them a billion dollars more, then their medicine gets to sit next to a thousand variations of "Vitamin C for Leukemia" that all cost a lot less.
There would be virtually no incentive for anyone to make an actual drug.
> Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.
"Easily" is doing a ton of work. Postmarketing surveillance can sometimes give low-confidence signal as to what's effective and what's not.
Dumb question: Wouldn't the market discover what works and what does not automatically, like any other product?
Step 1. Ensure extremely widespread use to ensure that effects are noticed relatively quickly (i.e. within a few decades)
Step 2. Hope someone notices correlations between that drug use and the positive/negative downstream effects.
Step 3. Hope someone who has access to the data required to actually conduct an observational study actually wants to run the observational study. By the way, you better hope all the info you need is available in insurance claims data (it's not) because that's really the only data you'll have at high scale. For most of these fraudulent drugs, you won't even have that! Because why would you go through the hassle of gaining reimbursement when you can just sell bullshit for cash? So in practice the required real-world data on these drugs and their effects literally will not exist in most cases. Where the data does exist, it'll be confounded by the unrecorded use of all the other fake drugs too.
Step 4. Hope the company selling a non-working drug doesn't have the clever idea of publishing countervailing "studies" backed by much larger budgets and with more existential motivation. They won't be assessed by FDA so these studies can just say whatever they want, of course.
Step 5. Lobby for outlawing of a drug that, in fact, didn't break any law whatsoever. Obviously it can't be illegal to sell a non-working in a regime that has no way to assess whether a drug works. So you really just gotta hope to post enough times that a drug is bad that people stop taking it.
But cigarettes' negative effects are significantly stronger than most drugs' positive (or negative) effects. So every single step is going to be dramatically harder, require more people taking the drug, more data, more time, more sophisticated analysis, more direct scrutiny from "the market."
Give it a few decades and, maybe a couple hundred or couple thousand deaths, and with a bit of luck "the market" will have eliminated one of the thousands of fake drugs proliferating across the market.
Thankfully for the company behind it, they can always just start up another totally legal fraud by just picking a different chemical (inert or not) and claiming a different medical benefit.
Oh yeah, meanwhile, doctors are just flooded with absolute bullshit claims 24/7 with no repercussions for anyone just lying to them. It's good stuff.
Literally one of the dumbest ideas I've ever heard.
>Literally one of the dumbest ideas I've ever heard.
It is capitalism 101. Not sure if that is the dumbest idea, because the whole world seem to run on it.
I am clearly talking about both positive and negative effects of drugs. Both would be generally extremely hard to detect, and literally impossible to detect without both 1) massive consumption of that drug and 2) a civilizational-scale rewriting of the data infrastructure that actually exists.
Perhaps you should explore Capitalism 201. You'd learn that markets only have all their most lovely features in an environment of complete information, which this is not. The more the information asymmetry (or complete ignorance) there is, the less-well markets work.
The proposal being put forth here is: what if we just had virtually no actual information about these products?
Well, the markets don't work. Capitalism 201.
Then you are answering to a point that was not raised.
> most lovely features in an environment of complete information, which this is not. The more the information asymmetry (or complete ignorance) there is, the less-well markets work...
I am well aware. But for what I as well as the parent is saying, this only require information regarding if the thing works or not for the people who use it as long as that information is not censored.
That's answering the question.
> this only require information regarding if the thing works or not for the people who use it as long as that information is not censored.
Can "the market" currently detect whether there is complex life in the Messier 54 star cluster?
No.
Is it because that information is censored?
No.
It's because that information doesn't exist.
Or more specifically, because that information is indistinguishable from noise. Even in the scenario of cigarettes' extremely strong effect (stronger than nearly every drug) deployed at massive scale (higher scale than nearly every drug), it required decades to establish causality.
Especially in the regime of drugs being released to market solely on safety data, the information about whether they work would be generally indistinguishable from noise. For way, way longer than cigarettes given the smaller population, smaller effect sizes, and the (presumed) proliferation of noisy concomitant "medications."
Do you think companies spend billions of dollars running RCTs for fun? The FDA doesn't require RCTs, they require only strong evidence of efficacy and safety. They do ultra-expensive RCTs because without them, it's virtually impossible for anyone to know – including the patients themselves – whether the drugs work.
Not if the signal is strong. Whether a thing works or not, is a pretty strong signal. Your equating it to bad effects of smoking is flawed and is not comparable.
You other arugments are also pretty bizzare, but not going into that for sake of focus.
> if the signal is strong, it will be visible
> your specific example of it taking decades of massive deployment and close scrutiny to detect an effect significantly stronger than most drugs is ummm... flawed and not comparable for no reason I can articulate
okay buddy
That's okay, you're someone who thinks the FDA is in cahoots with insurers and drug companies to approve ineffective or unsafe drugs at scale lmao. And here you are arguing that the market would actually naturally detect exactly that.
What the heck are you smokin? Did paracetamol have a weaker affect on temperature than smoking causing cancer?
Thank you for demonstrating my point.