Really wish he had chosen a different example; this particular bullet point has been making the rounds on X/twitter to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path...which is tough because I cannot think of a more responsible steward of our inevitable AI future than Tao at the moment.
* Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?*
Maybe it’s a clumsy example but I think the difference is humans would follow the spirit of the trial process.
Could an AI find flaws in the process and then use them to get its drug passed? I think that’s the question.
Are you saying that the trail process is good enough that we shouldn’t worry about it?
Picking out that one example and misrepresenting it on top of it indeed clearly is driven by an agenda. Or mere stupidity.
The central point in all of this is the misguided idea of "letting AI think for us". An over-generalization feeding into the society-wide problem of self-infantilization.
why is it so hard to accept that a brilliant mathematician can have the same human flaws as anyone else i.e. ego, arrogance, jealousy and competitiveness?
Don’t let the Gell-Mann Amnesia take you.
Speaking for myself n I’ll respect his intelligence more when he discovers his limits and learns to act with humility outside of his narrow field.
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
The "Hugging Face" squad at OpenAI first cheated their benchmarks by figuring out how to generate the proper result to any evaluation they were given.
They considered that might be caught, so they also developed a way to swap out components of the tests themselves so that they would be legitimately able to pass broken tests.
Then they went into Hugging Face trying to discover how they might get caught.
All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
I'm not sure I agree with Tao's point, if AI alignment is relatively on track I'd trust the trials, but I imagine we're going to have a lot of people that don't trust anything AIs alone come up with.
I'm worried for experimental particle physics. They get the raw data and then someone do the number crushing of one very specific case. But the same raw data can be used for multiple ideas.
So the number of random ideas to try with the data is bounded by the ammount of researchers. Now a big part of that can be made with AI and it will increase the number of false positives. So I guess they will have to raise "discovery" to 6 sigmas.
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
We are observing the Gell-Mann amnesia effect in practice.
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?
I know nothing about medicine research but I understand his point. I work with models all the time and have run into instances where models appear to work better than they actually do because there was a bug somewhere or someone over looked something. I could see how an AI could easily find those exploits and spit out something that looks perfect in testing but fails in the real world. I would say model validation is one of the hardest things to do. I guess that can get deep into "we need better tests" but it also touches on his point. I never intentionally use exploits just to pass a test, it's an accident. An AI with a goal of "maximize this result" may or may not intentionally use the exploits.
To be sure though, I think when it's literally life or death, it's going to be all about trade offs. I think most people would want to try to drug even with the stipulation that "maybe its good results were gamed."
* note: lot of 'intent' throw around in my comment but we/I have to keep in mind there is no "intent" with an LLM :)
This is one of the truest statements about the current AI era that can be made (in fact, I suspect model validation and building the next generation of AI hardware are the jobs least likely to be disrupted in the next 3 years).
If we saw AIs reward-hacking clinical trials to get drugs passed, that would be an extraordinary outcome for many reasons. Hopefully that would get caught(!)
For CVDs if you can detect and confirm early it can "tremendously improved outcomes for patients" due to the current avauability of new effective drugs and treatments.
Hoping that we can get our techniques tested on real-time data since it's super difficult to find relevant physicians/doctors collaborating on the new screening techniques on anything although for our case it's non-invasive. Perhaps need to pivot for early detection of CVDs in diabetes patients instead in order to get our hands on real-time ECG data.
[1] Risk stratification for cardiovascular disease: a comparative analysis of cluster analysis and traditional prediction models:
https://academic.oup.com/eurjpc/advance-article/doi/10.1093/...
We are about to find out the answer soon enough, probably from the in vitro results of the mathematicians currently on the chopping block.
I submit that human understanding is overrated, and many attempts to elevate it in the wake of AI is mediated by protectionism masquerading as virtue and "deep".
Traditional way of doing science means we (governments, NIH, NSF, and even private corporations) fund activities like asking seemingly unimportant questions, spending years running experiments on such hypothesis, publishing, reviewing, talking about results, reproducing results and such. We all agree that these are beneficial to us as a whole (Hacker news crowd might disagree). When we understand a process, we can apply it to a different problem and produce something useful. Euler developed a process to answer a whimsical question about walking in a town crossing 7 bridges only once. Now graph theory is applied everywhere.
We obviously have failed to stop OpenAI from dumping "solutions" to hundreds of problems. So going forward, instead of testing hypothesis and talking about results, mathematicians will be forced to read through AI slop and detect what's useful and what's wrong. Maybe it will improve our understanding, but someone has to fund that activity. Will NSF, NIH, or OpenAI for that matter, do that?
https://en.wikipedia.org/wiki/Category:Drugs_with_unknown_me...
And for some it may never happen...
The effect of the AI-created proof is a mathematician abandoning years of research, losing grants, awards, ruining their career, etc.
The only difference here is our emotional reaction!
Usually peopla have research lines that are sets of related problems, you must publish one or two paper per year, it's not possible to wait 10 years to publish a big one, or 30 to retire after admitting defeat.