undefined

points

[-]

> Closed or open source doesn't matter; it's the ability to control them that's important. People have been cracking and patching for decades without source, but they have that control.

You have no idea what has been baked into the weights in the training process. In theory you could find biases and attempt to "patch" them out, but its a vastly different process vs. patching machine code.

Consider what would happen if Google's open weight models were best at writing code targeting Google's services vs. their competitors? Is this something that could be patched? What if there were more subtle differences that you only notice much later after some statistical analysis?

by narrator9 hours ago|

parent|

[-]

People are already patching these models using abliteration to prevent them from refusing any request, so it is possible for end users to change them in meaningful ways. You can download abliterated models right now from Hugging Face that will respond to all kinds of requests that frontier models refuse.

by hparadiz8 hours ago|

parent|

[-]

Yup there's a ton of people on HN sleeping on this new tech because they refuse to look at anything AI. We now have jail broken models but the average person on here doesn't even know how to download and try a model.

by CableNinja8 hours ago|

parent|

[-]

It doesnt help that guides ive seen have been pretty handwavy or are not specific enough to the individual situation (i have z hardware, heres how its done). It also doesnt help when every post on HN i see is like 'oh waow i did x on a mac mini with 128gb ram'. That spec is beyond many, running on generally available resources (such as hardware one might have laying around their house) do not seem fit for the purpose, so its back to building a new machine (gl when ram is worth 2x its weight in gold), or buying a $1000+ mac mini, or other device. Any low end system cant turn out tokens fast enough, or doesnt have the resources for context or processing.

Local ai is not ready, and if you think it is, prove me wrong with a detailed guide running commodity hardware with complete setup steps that can use a decently sized model.

I spent 2 weeks trying to get anything running - 8gb RX550XT, 12gb ram, 8core cpu. I even tried turboquant to lower memory utilization and still couldnt even get a 3B or 4B model loaded, and anything lower wont suit my needs (3/4B are even pushing it).

by vbezhenar2 hours ago|

parent|

[-]

TBH I never understood people trying to run LLM locally. Just rent a powerful machine in the cloud for few hours. It's cheap enough, because you don't need to own a hardware. It doesn't introduce a dependency because there are hundreds of hosters. It doesn't compromise your data, because nobody would extract data from your VM, not until you're under an investigation, anyway, and even in that case just use different jurisdiction.

Spending humongous amount of money to get machine that'll felt obsolete in 2 years? I don't know.

by hparadiz6 hours ago|

parent|

prev|

[-]

"Local AI is not ready" > proceeds to run a 7 year old budget GPU

You're like the kid showing up to a test without a pencil.

It's ridiculous for you to suggest that an advanced AI model needs to run on your budget 7 year old graphics card that is already out of date for even today's gaming. My parents spent $2500 on a computer in 1995 and that was a 166Mhz Pentium 1. If they spent that money today it would be $5261. Think of what you can get for amount of money. Then you're over here trying to say a budget graphics card needs to somehow compete with the bleeding edge of computer innovation.

You do, in fact, need to spend money on appropriate gear if you expect to participate.

by userbinator5 hours ago|

parent|

[-]

If you want AI image generation and are willing to wait a little longer, you don't even need a GPU: https://news.ycombinator.com/item?id=32642255

by hparadiz3 hours ago|

parent|

[-]

I've played with SD plenty. CPU even becomes manageable at low resolutions. But uh CPU/GPU is starting to blur now with these new AMD inference CPUs with built in GPUs. And ARM based machines like Macs. I wish more people on HN were using this stuff so we could have fun conversations about it instead of arguing over whether or not we should even be using these tools.

by narrator7 hours ago|

parent|

prev|

[-]

When Stallman was getting started writing emacs in the early 80s, Unix machines were vastly out of reach price wise for the common home user, but he did his open source work anyway, and eventually the 386 came along.