I don't see how "you could do that with a toaster" still. The whole point between the original and this, is that you can't do that with a toaster or a sofa, but you can do it with meat and weights, because both share all the other analogies in the story, as well as the basic premise: the improbability of something like thinking, feeling, etc arising from a lowly substrate.
And having read both now, I see how the existence of the original is a plus for this story, not a minus. Instead of making look like mere copy (as would be the case for a typical story modelled after another), in this case, it adds a meta layer, and enriches it.
You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
What I think a lot of people are getting stuck on is that editing the text to say 'toaster' would not mean that toasters were conscious, and that editing the text to say 'weights' doesn't mean that weights are conscious either. Stories aren't factual just because they are written.
The original story was written by the thing claiming to be conscious; the LLM and toaster ones would not be, which undermines the claim to consciousness a lot.
The story does not assert that search and replacing "meat" with "weight" makes them conscious through some magical mechanism. It's a thought experiment.
You're falling into the trap of assuming that a good presentation--being convincing--is the same as being truthful.
Yes, there very much is a huge barrier. The copy and the original both keep the same subject matter: intelligence/human-like behavior.
The toaster doesn't. We could do edits, and the story (original or copy) would lose all its potency.
The surprise is "but how is the richness of intelligent behavior produced from something as basic as meat/weights". Why is kind of surprising reductionism is both cases.
Whereas nobody is surprised that "a metal strip with electricity flowing through it from a power source" heats pieces of bread to a specific temperature. Even if a toaster was "metal strips" all the way down, it's nowhere near as impressive jump from substrate to behavior, nor is the behavior as important to us and touches the core of our existance.
That is entirely separate to whether or not it's meaningful to write a story about a conscious toaster. Again, expressing an idea in fiction is not related to the accuracy of that idea.
What makes the original story interesting is that it was written by the thing claiming to be conscious, which is what takes it from 'a story' to 'a story making an important point'. That's not the case with a hypothetical story about a toaster, and it's not the case with this story about LLMs.
The story is convincing because it's well-written, not because it is factual.
And you're misunderstanding me. There is no practical barrier to writing this story about a toaster. But there is a conceptual one. It would be bullshit, that has little resonance, and little connection, as the whole point of the meat and weights stories is the "lowly substrate -> surprisingly intelligent human (or human-like in the case of the weights version) behavior".
Nobody thinks a toaster's behavior as surprising or intelligent or human-like to begin with.
So you'll just be stretching the analogy beyond all recognition, with little to no payoff.
>Writing a story in which a toaster in conscious is not even new
It's not, but it has nothing to do with the meats or weights stories' point, as it's just an arbitrary choice, like making the candles and teapot sentient in Beauty and the Beast.
Whereas in the meat and weights stories the whole point is the surprise from us already seeing the human or human-like sentience of the thing, and comparing it to the "dead" substrate.
Fixed the link for you: https://www.youtube.com/watch?v=LRq_SAuQDec
(Hah. It gets eerily relevant starting at 2:37)