> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.

You just think that because you don’t know how the drug discovery process works.

There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).

The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.

Human trial are very expensive, so I guess it's difficult for the AI to find a way to hack the rulebook.

I'm worried for experimental particle physics. They get the raw data and then someone do the number crushing of one very specific case. But the same raw data can be used for multiple ideas.

So the number of random ideas to try with the data is bounded by the ammount of researchers. Now a big part of that can be made with AI and it will increase the number of false positives. So I guess they will have to raise "discovery" to 6 sigmas.

"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""

No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.

There are probably more mis-aligned humans than mis-algined AI's today.

I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.

> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?

I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?