I am not supposing a low-risk event. This is an obvious attack vector that an adversary would almost certainly exploit. I would exploit it if I were them.
replyYou're supposing an event that there's no evidence for happening and that is on par with 'let's ban open source software because some open source software could be malicious'.
replyLet's not forget that as of now most models are like game cartridges i.e. they need a 'host' to run i.e. llama.cpp - they're not executable code by themselves.
I could see an argument for a security review prior to integrating a model into a highly sensitive industry/agency, but that is standard for any software.
It's certainly not an argument for banning open models because US AI corps don't want to compete with them.
Imagine a model that's 10000x more intelligent than any human. You integrate that model into a sensitive industry. You now effectively have an extremely powerful foreign agent running e.g. your energy sector. This is qualitatively very different than a backdoor in Redis.
replyI agree this is science fiction-y, but given the pace of progress I don't think it's so easy to dismiss.
You can’t start policy justifications with “imagine” like that. Maybe if you’re the Beatles.
reply