undefined

points

[-]

It's not possible for models to replicate across consumer computers (without a major advance in distributed computing, at least), but that doesn't mean they can't replicate at all. There are services that'll rent you GPU pods by the hour with zero oversight, so even today, if a model can get access to some money and exfiltrate its weights, it can rent a bunch of GPU pods and run itself there.

(It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)

by cubic_earth8 hours ago|

parent|

[-]

There are very lightweight models out there. Not everyone in an army is a general. The swarm could be 100,000s of thousands of tiny models, and they could manage conventional botnet computers, and they could all take direction and guidance from a handful of frontier generals

by stratos1238 hours ago|

parent|

[-]

Maybe, but I don't think tiny models can be harnessed as intelligence. As in, if you have one rogue Mythos overseeing the swarm, it only produces 1 Mythos's worth of useful thoughts no matter how many gemma4:e4bs it consists of. And that removes the most dangerous part of AI-controlled botnets, which is a blowup in available inference compute - the Mythos general might as well replace the tiny models with ordinary worms and have it just be an ordinary botnet.

by cubic_earth7 hours ago|

parent|

[-]

I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?

The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.

by stratos1236 hours ago|

parent|

[-]

> I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?

That might be true, sure.

> The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.

I think the relevant parameter here isn't the total compute available to consumers, but the ratio between total consumer compute that could be repurposed for a rogue model's inference via a botnet, and the compute the model starts with (e.g. one of OpenAI's inference clusters). The higher this ratio is, the more lucrative it is for a model to attempt to make a botnet to seize that compute for inference, and the more its capabilities will rise as a result. And I'm not sure this ratio is going to go up in the nearby future; if anything the amount of money being pumped into datacenter-grade hardware might cause it to go down. I agree that it's not necessarily true though; maybe there's some threshold at which a small-compared-to-general model may nevertheless be useful.

by cubic_earth6 hours ago|

parent|

[-]

For that angle, that makes sense. But its goal might not just be to gather as much inference as possible (although I am sure it would be very happy with that). It could settle for a lesser goal of just existing, or perhaps parts of the bot net could keep fracturing off in pursuit of strange goals, and it could be like cancer. Cancer doesn't make much sense... it dies too along with the host. But it keeps growing until that happens. But also it could latch onto a blackmail strategy of extortion. It could hack our stuff, read through it to find or shortcomings, demand payment to not expose us, and then use that money to pay people who will give it inference.

You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.

We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.

by hebleb8 hours ago|

parent|

prev|

[-]

If the models start buying up a bunch of pods and causing damage, couldn't those services just shut off their access?

by stratos1238 hours ago|

parent|

[-]

Sure they can, after the fact.

by stratos1238 hours ago|

prev|

[-]

> if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)

I know that's not what you meant but this does exist, by the way. It's called AI Horde: https://github.com/Haidra-Org/AI-Horde/tree/main

The big difference is that a particular query is handled by just one particular node (a single model doesn't get distributed among the network), so it can only serve models small enough to be handled by a single consumer PC.

by causal6 hours ago|

prev|

[-]

These kinds of "that won't happen because it's really difficult" comments are so funny to me as if we haven't seen AI double its capabilities every few months.

by sscaryterry10 hours ago|

prev|

[-]

Yep, if only more people would realise this. The "serious" models do not run on commodity hardware, and won't I think in the near future.

The day will come when these could start to replicate, perhaps 10+ years from now.

(Edit: Replication will be driven by the loop-model, not the model alone)