upvote
I assign probabilities of zero to all 3. Computer program being conscious leads to ridiculous and obviously false conclusions (think about a person running a program using pen and paper for memory).
reply
why is that obviously false? To a functionalist, a pen and paper and a set of rules is sufficient for consciousness.
reply
Gist of the argument:

If it was true, you can create extreme pain by running a program. You can run the program by simulating a CPU, using pen and paper for memory. So you're essentially claiming that some simulated being is in pain because there are some 1s and 0s on paper. In fact, you can decide to use an arbitrary encoding of the memory, so a sufficiently long sequence of 0s written on paper corresponds to a simulated being feeling pain in some encoding. That is clearly nonsense.

reply
Time scale matters a lot in how we as humans perceive things like agency. Plants grow too slow for us to see any intent, but when you speed up a time lapse, suddenly it looks like plants reach for sunlight and vines for supports. Now, that may be projection on our part, but it may not be.
reply
This only seems insane/crazy from the perspective of everyday life. But philosophically it checks out, "We live in a simulation" and all that.

The crux if it is that if you ever break from "the universe can be fully expressed mathematically", you are stuck in the mud of supernatural beliefs.

reply
Once or twice I've experienced extreme pain, and it was downstream of a bright light shining on a wet rock for millions of years.

I try to imagine myself long ago, on the outside looking in, with someone explaining to me that extreme pain, wondrous art, hunger, triumph, and despair would all unfold in due time where the rocks were wet and the lights bright enough.

I can imagine myself calling this clear nonsense.

reply
Isn't pain just a manifestation of a bunch of chemical and electrical signals in the brain and body? It's not "clearly nonsense" to me that you could cause pain by writing a sufficiently long sequence of 0's - for it to be obviously wrong, you'd have to have some understanding of where consciousness comes from.

If you don't understand that, how can you assert that it doesn't come from mathematical relationships?

reply
This seems like a logical error. I don't understand how an internal combustion engine works, but I know it doesn't come from goblins jumping up and down inside.
reply
The fact that you know it literally means you understand, at least to some extent, how an internal combustion engine works (i.e. it is powered somehow by combustion, and jumping goblins are not combusting generally).

If you would have zero knowledge about ICEs, how would you know?

reply
I think 10-20% chance is wildly generous. What specific mechanism makes you think there's a 10% chance that current LLMs are conscious?
reply
Not GP but given we have no idea what conciousness is it seems foolhardy to go too low or too high for any of those numbers
reply
I think the problem with this argument is that it's too inclusive. Is the bacteria that's adapted to an antibiotic conscious? It's showing intelligence right? I think if you're going to say something is potentially conscious, for me to take the argument seriously at least, there needs to be some plausible mechanism. I just don't see one for LLMs.
reply
maybe the bacteria are conscious. How sure are we that they're not?

The only strong argument I have against it is the anthropic principle -- there are billions of times more bacteria than humans, so it's overwhelmingly unlikely that I'd be a human rather than a bacteria.

Not a very good argument of course.

reply
Yeah I get what you are saying but I don't see much of a plausible mechanism for humans either and yet clearly there is one.
reply