upvote

points

by 2718315 hours ago |
comments
upvoteby tomrod10 hours ago|
next
[-]
Not sure why you were flagged either -- it was a reasonable comment that clearly spawned a discussion! People are strange.

> There's no essential reason to have utility companies, dams, nuclear arsenals, factory production floors, mines, cars, planes, ships, etc all be computerized

Indeed, this is a great first step! Once agentic loops and multi-purpose robotics can be combined we do face an issue of strengthening the airgap.

> And none of that pie in the sky shit has anything to do with language models.

I'm of the opinion that we should be mindful about the capabilities we allow a truly non-human mind to perform. Blocking via solid security and airgapping for critical systems makes sense to me for most cases, aligned I believe to your thoughts here. The recent ChatGPT hack shows that the "paperclip apocalypse" is still instructive. These systems target edge cases and zerodays to cheat!

reply
upvoteby 271839 hours ago|
parent|
[-]
Yeah ML systems often do very surprising "cheats". A while back there was a system for automatically landing aircraft on a carrier deck which exploited an overflow in an acceleration parameter in the simulation--the optimal strategy became to slam as hard as you can into the carrier deck.

But it's a really big leap from that kind of thing to "a large fraction of humanity is manipulated into building the automated factory infrastructure to bring about their demise and turn the entire solar system into paperclips". Or some such thing. There's just a lot of inconvenient reality between where we are now and that fantastic outcome.

So that's why I ask questions like "what are you actually talking about?" when people write stuff like TFA. It's... bizarre. Only makes sense on huge amounts of drugs or to the insane. Or in some hypothetical reality that isn't this one.

reply
upvoteby vessenes11 hours ago|
prev|
[-]
I can’t read your original comment, but when the chief scientist of a company that just released six month old software that can use Kicad on your computer to design a circuit board and have it created and shipped to you tells you he thinks we will get to automatically self reinforcing improvements, I think it’s wise to take him seriously. It was only eighteen months before Astra was created that LLMs could not count rs in strawberry.
reply
upvoteby 2718311 hours ago|
parent|
[-]
This was my original comment:

  > What on earth are you talking about?

  > 1. What does any of this have to do with "A(G)I"? Nobody has any clue what AI even is let alone how to build one. We're talking about language models here.

  > 2. What's the winner-takes-all thing about? Why can't you have multiple independently developed AIs?

  > 3. What's with all the "safety" stuff? Why is it important? They're just computer programs...
> when the chief scientist of a company that just released six month old software that can use Kicad on your computer to design a circuit board and have it created and shipped to you tells you he thinks we will get to automatically self reinforcing improvements, I think it’s wise to take him seriously. It was only eighteen months before Astra was created that LLMs could not count rs in strawberry.

Sorry if I'm a little slow here, a few questions:

1. What's Astra?

2. Who is this chief scientist? Which company?

reply