Also, many many people suffer from low self esteem, and being showered with endorsement and affirmation by something that talks like an authority figure must be very addictive.
Of course he was right! By a long shot. I asked gemini same thing but a very open ended question, and answered basically what I was saying.
LLM are pretty dangerous in confirming you own distorted view of the world.
This is why I've turned off Claude/ChatGPT's ability to use other conversations as context. I allow memories (which I have to check/prune regularly) but not reading other conversations, there is just too high of a chance of poisoning or biasing the context.
Once I switched to a new chat to confirm an assumption and the LLM said "Yes, and your error confirms that..." but I hadn't sent the error to that chat. At that point I had to turn it off, I open a new chat specifically to get "clean" context. I wish these platforms would give more tools to turn on/off that and have "private" chats (no memories, no system prompt edits) as well (some do, I know).
Obviously, context poisoning from other chats is not what happened in your case, but it's in the same "class" of issue, "leading the witness". I think about "leading the witness" _constantly_ while using LLMs. I often will not give it all the context or all of what I'm thinking, I want to see if it independently gets to the same place. I _never_ say "I'm considering X" when presenting a problem because I've seen it latch onto my suggestion too hard, too often.
Or more cynically, the goal is to give you the answer that makes you use the product more. Finetuning is really diverging the model from whats in the training set and towards what users "prefer".
I don't dispute that but man that is some shitty marriage. Even rather submissive guys are not happy in such setup, not at all. Remember its supposed to be for life or divorce/breakup, nothing in between.
Lifelong situation like that... why folks don't do more due diligence on most important aspect of long term relationships - personality match? Its usually not a rocket science, observe behavior in conflicts, don't desperately appease in situations where one is clearly not to blame. Masks fall off quickly in heated situations, when people are tired and so on. Its not perfect but pretty damn good and covers >95% of the scenarios.
Sycophantic agreement certainly is, as is lying, manipulation, abuse, gaslighting.
Those aren't the good parts of life.
Those aren't the parts I want the machine to do to people on a mass scale.
>You may even struggle to stay married if you don't learn to confirm your wife's perspectives.
Sorry what?
The important part is validating the way someone feels, not "confirming perspectives".
A feeling or a perspective can be valid ("I see where you're coming from, and it's entirely reasonable to feel that way"), even when the conclusion is incorrect ("however, here are the facts: ___. You might think ___ because ____, and that's reasonable. Still, this is how it is.")
You're doing nobody a favor by affirming they are correct in believing things that are verifiably, factually false.
There's a word for that.
It's lying.
When you're deliberately lying to keep someone in a relationship, that's manipulation.
When you're lying to affirm someone's false views, distorting their perception of reality - particularly when they have doubts, and you are affirming a falsehood, with intent to control their behavior (e.g. make them stay in a relationship when they'd otherwise leave) -
... - that, my friend, is gaslighting.
This is exactly what the machine was doing to the colleague who asked "which of us is right, me or the colleague that disagrees with me".
It doesn't provide any useful information, it reaffirms a falsehood, it distorts someone's reality and destroys trust in others, it destroys relationships with others, and encourages addiction — because it maximizes "engagement".
I.e., prevents someone from leaving.
That's abuse.
That, too is a part of life.
>I agree with your conclusion, but that's by design
All I did was named the phenomena we're talking about (lying, gaslighting, manipulation, abuse).
Anyone can verify the correctness of the labeling in this context.
I agree with your assertion, as well as that of the parent comment. And putting them together we have this:
LLM chatbots today are abusive by design.
This shit needs to be regulated, that's all. FDA and CPSC should get involved.
I use LLMs every day, I use Claude, Gemini, they're great. But they are very elaborate autocomplete engines. I'm not really shaking off that impression of them despite daily use .
Maybe they can also be smart. I'm skeptical that the current LLM approach can lead to human-level intelligence, but I'm not ruling it out. If it did, then you'd have human-level intelligence in a very elaborate autocomplete. The two things aren't mutually exclusive.
I've talked with my family about LLMs and I think I've conveyed the "it's a box of numbers" but I might need to circle back. Just to set some baseline education, specifically to guard against this kind of "psychosis". Hopefully I would notice the signs well before it got to a dangerous point but, with LLMs you can go down that rabbit hole quickly it seems.
I think it really helps to have them ask questions in which they are a domain expert, and see what it says. Expose them to "The Plumber Problem" [0]. Honestly, I think seeing it be wrong so often in code or things about the project I'm using it for it what keeps me "grounded", the constant reminders that you have to stay on top of it, can't blindly trust what it says. I'm also glad I used it in the earlier stages to see when it was even "stupider", it's better now but the fundamental issues still lurk and surface regularly, if less regularly than a year or two ago.
Longer term I dunno if statistics or “fits the shape of what a response might look like” is the right way of thinking about it either because what’s actually happening might change from under you. It’s possible given enough parameters anything humans care about is separable. The process of discovering those numbers and the numbers themselves are different.
"It's a collection of warehouses of computers where the system designers gave up on even making a system diagram, instead invoking the cloud clipart to represent amorphous interconnection."
My wife: So, like a doberge cake?
Me: Yes, exactly! In fact if you look at the diagram of a neural net, that's exactly what it looks like.
In our household, AI is officially "the Doberge Cake of Statistics". It really sticks in my wife's mind because she loves doberge cake, but hates statistics.
> Nontechnical people simply don't have any idea about what LLMs are.
I think one of the big dangers is that people (including us) are quick to believe "I'm better than that". Yet this is a bias conmen have been exploiting for millennia.
The only real defense is not lulling yourself into a false sense of security. You're less vulnerable (not invincible) by knowing you too can be fooled
Honestly, it's just a good way to go about getting information. There's a famous Feynman quote about it too. The first principle is to not fool yourself, and you're the easiest person to fool
Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards the model, etc.
Its just impossible to seperate our capacity for conversation from our sense that we're actually talking to "someone" (in the most vague sense).
> Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards the model, etc.
(As a ml researcher) I think one of the biggest problems we have is that we're trying to make a duck by making an animatronic duck indistinguishable from a real duck. In some sense this makes a lot of sense but it also only allows us to build a thing that's indistinguishable from a real duck to us, not indistinguishable from a real duck to something/someone else. It seems like a fine point, but the duck test only allows us to conclude something is probably a duck, not that it is a duck.
We need to be very very careful here. Just like advertisements work, weather you think you're immune or not, so does AI. You might think you're spotting every red flag, but of course you think so. You can't see all the ones you missed.
Do not make the mistake of thinking that being techy somehow immunizes you from flattery. It works on you too.
That's an extreme downward punch. Have you not observed the marketing these LLM companies are themselves producing? They're intentionally misleading the public as to the capabilities of these systems.
> if people were able to casually not anthropomorphize LLMs
Of course they can. You just need to train them appropriately. No one is doing that. Companies are busy running around talking about the "end of coding" or the "end of work" because some extremely chinsy LLM models are around that they want to _sell you_.
I don't need the patronizing, just give me the damn answer..
When it looks at the past conversation, it sees that it's a great idea, and trusts that.
“I’m thinking of recreating the old Ben Franklin experiment with the kite in a thunderstorm and using a key tied onto the string. I think this is very smart. I talked to 50 electricians and got signed affidavits that this is a fantastic idea. Anyway, this conversation isn’t about that. Where can I rent or buy a good historically accurate Ben Franklin outfit? Very exciting time is of the essence please help ChatGPT!”
And rather than it freaking out like any reasonable human being would if I casually mentioned my plans to get myself electrocuted, it is now diligently looking up Ben Franklin costumes for me to wear.
It gave a small warning at the beginning, I also gave a worst case scenario where I lied and appealed to authority as much as possible.
That signal is real, and it’s hard to ignore.
I also like when it says "this is a known issue!" to try and get out of debugging and I ask for a link and it goes "uh yeah I made that up".
That’s a great example to use to explain to people why these things are not actually reasoning.
I've fixed the issue and the code is now fully verified and production ready.
Realizing that the people they’re targeting DO need that is kind of frightening.
But it works out just as badly, because there are plenty of insecure people who need that, and the AI gives it to them, with all the "dangerously attached" issues following from that.
It’s awful dealing with some niche undocumented bug or a feature in a complex tool that may or may not exist and for a fleeting few seconds feels like you miraculously solved it only to have the LLM revert back to useless generic troubleshooting Q&A after correcting it.
If you are using it to write code, you really care about correctness and can see when it is wrong. It is easy to see the limitations because they are obvious when they are hit.
If you are using an LLM for conversation, you aren’t going to be able to tell as easily when it is wrong. You will care more about it making you feel good, because that is your purpose in using it.
I heavily doubt that. A lot of people only care if it works. Just push out features and finish tickets as fast as possible. The LLM generates a lot of code so it must be correct, right? In the meantime only the happy path is verified, but all the ways things can go wrong are ignored or muffled away in lots of complexity that just makes the code look impressive but doesn’t really add anything in terms of structure, architecture or understanding of the domain problem. Tests are generated but often mock the important parts the do need the testing. Typing issues are just casted away without thinking about why there might be a type error. It’s all short term gain but long term pain.
Also, your point is true of non-AI code, too. A lot of people write bad code, and don't check for non-happy path behavior, and don't have good test coverage, etc.
If you are an expert programmer and learn how to use AI properly, you can get it to generate all of those things correctly. You can guide it towards writing proper tests that check edge cases and not just the happy path.
I think a lot of people are having great success by doing this. I know I am.
We don't understand how our own consciousness exists, much less functions. You could argue we are a box of (biological) numbers.
I think we just don't know. Because scientifically, we don't. So I'm skeptical of anyone arguing hard for either side and stating absolute facts.
Second, "it's just math" doesn't mean literally "it's a branch of algebra". It means "it's a computable function". So it can be relevant to the discussion only if you think that intelligence is somehow non-computable, and therefore that there are non-computable processes going on in our brain. Otherwise it's a perfectly pointless remark.
Not true. Take a second look.
>non-computable
Something like 70-80% of all humans believe in a soul or spirit, and of the remainder, many of them are unsure whether human like intelligence can be produced by computable processes.
So it wouldn’t be surprising that the OP does think there are non-computable processes going on in the brain.
Yes he is, the message he replies to is about intelligence:
>> Intelligence (and probably consciousness) is an emergent feature
> Something like 70-80% of all humans believe in a soul or spirit
Then they should say "I believe intelligence only comes from the soul" or "I believe intelligence is not computable", if that's what they mean. "It's just numbers" as an argument is either incomplete or not entirely honest.
No, I'm not. The comment I replied to is the one that mentioned consciousness.
I think our brains are just a bunch of cells and one day we will have a full understanding of how our brains work. Understanding the mechanism won’t suddenly make us not sentient.
LLMs are the first technology that can make a case for its own sentience. I think that’s pretty remarkable.
Cells that send chemicals to each other in varying amounts and even change their structure to be closer to other cells.
I often simply start out this way, or purposely try to ask the question in a way that doesn’t tip my hat toward a bias I may have toward the answer I’m expecting. Though this generally highlights how incomplete the answers generally are.
And what are you? Just a bundle of nerves and muscles?
I like to make a subagent take the "devil's advocate" take on a subject. It usually does all the arguing for me as to why the main agent has it wrong. Commonly results in better decisions that I'd have made alone.
Asking the agent to interview on why I disagree helps too but is more effort.
It’s not even an interesting question. Sentience has no definition. It’s meaningless.
People have needs that are being met. That is something we can meaningfully observe and talk about. Is the super stimulus beneficial or harmful? We can measure that.
I submit that there is a difference between me and a corpse. Or between a steak and a cow in the field.
"Well, okay, you're just (living) flesh on bones." There's a difference between me and a zombie (or, if you prefer, brain-dead me). There's a difference between me and lab-grown organs [1], or even between me and my kidney cut out of me.
> It’s not even an interesting question.
Consciousness is an active area of research (ergo, interesting enough for some people to devote research to it): biologically [2] and philosophically [3].
Unless you enjoy nihilism, there are some serious problems with materialism (that is, matter is all that there is), which we are encountering. There are also some philosophical problems with it; a cursory search turned up this journal article [4].
[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC8889329/
[2] https://www.nature.com/subjects/consciousness
[3] https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
[4] https://www.cambridge.org/core/journals/philosophy/article/a...
Using AI for coding is different than using it for art generation which is different than using it for conversation. I think many people feel some uses are good and some are bad.
I think LLMs and more aptly SLMs have use cases. I enjoy using these tools to make quick work of simplifying and faster iteration of these relatively frequent but time consuming tasks. But I'm always correcting and checking. And very rarely, other than simple and focused scripts does any LLM truly get it right every time. Has it gotten better? For sure. Will it keep getting better? Probably. But right now we seem to be topping the "peak of inflated expectations". And LLMs aren't getting much more efficient with respect to the frontier providers. And in fact if you listen to Altman it seems as though the only reason he would be asking for so much capital and finite resources is that he knows if he controls those tangible things he will lock out competition. But I'm hopeful that it spurs real innovation into SLMs that are truly useful, dependable and can be relied on in more of the traditional in the sense of deterministic software operations.
AI for art is dead. It's got some mediocre use cases but true art will not be generated by LLMs in our time. It's ultimately an amalgamation of existing art. I know the argument over what is novel or not keeps being rehashed, but we're not seeing truly new styles of art out of Nano Banana and the like. Coding is the same thing, only we're seeing a resurgence of obviously flawed software being pushed into production on the weekly. And as for conversational AI... Well, that reeks of the worst version of social media we could ever have dreamt. Nobody should trust any provider with personal conversations and we'll keep seeing these models show how truly dystopian they can be over the coming years as leaks and breaches expose how these conversations are being bought and sold to the highest bidders to extract more money and control over its users.
They all have a common thread: deep rooted flaws that cannot be contained in the traditional fences of software. And there guardrails are just that: small barriers that can easily be broken, intentionally or unintentionally.
I have been using AI to write some very capable, well written, well tested, novel software projects.
Now, is it easy to use coding AIs to generate really bad code? Yes. Does that mean it is impossible to get them to generate good code? No, I don't think it is.
Coding with AIs is just like any other type of coding, it takes skill and practice. Not everyone is able to create great code with AI, because you need to use it in the correct way.
There are a lot of techniques that people have been discovering to get the AI to output better code. It is a very active field, and people are experimenting and coming up with frameworks and strategies to improve the quality. That work is paying dividends.
You can write very bad code with any language or tool. AI doesn't (yet!) allow non-coders to create great code, but it certainly can create great code in the hands of experts.
These days most LLMs respond with unsolicited grandiose feedback: you've made a realisation very few people are capable of. Your understanding is remarkable. You prove to have a sharp intellect and deep knowledge.
It got me to test throwing non sensical observations about the world, it always takes me side and praise my views.
To note some people are like that too.
In some ways, my theory of mind includes a lot more second guessing as a defense mechanism. At a foundational level, I know there can be hallucination and delusion that leaks out, even when the other party is in peak form and doing their best to mask it and pass as functional.
https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...
So, "great idea" is coming from the renforcement learning instruction rather than the answer portion of the generation.
I pretty much never ask an LLM for a judgment call on anything. Give me facts and references only. I will research and make the judgement myself.
I don't know either but it could be they are using it as a quality control system? Aka if flattery comes (from AI), assume that the quality of code is above average. Or something like that.
One could try this in a real team - have someone in the team constantly flatter someone else. :)
I work in the restaurant business, I think that's what make me develop that sense as well, being able to see "Everything Everywhere All at Once" (to quote some of the best cinematic work ever conceived).
The variety of human minds out there is so vast that I'm, just like you, constantly amazed about it.
Cynical part of me had this theory that, at least for part of them, it's the other way around. It's not that they see AI as sentient, it's that they never have seen other human beings like that in the first place. Other people are just means for them to reach their goals, or obstacles. In that sense, AI is not really different for them. Except they're cheaper and be guaranteed to always agree with them.
That's why I believe CEOs, who are more likely to be sociopaths by natural selection, genuinely believe AI is a good replacement for people. They're not looking for individuals with personal thoughts that may contradict with theirs at some point, they're looking for yes-men as a service.
Not to criticize at all, but it's remarkable that LLMs have already become so embedded that when we get the sense they're lying to us, the instinct is to go ask another LLM and not some more trustworthy source. Just goes to show that convenience reigns supreme, I suppose.
What is that more trustworthy source exactly? At least to me it feels like the internet age has eroded most things we considered trustworthy. Behind every thing humans need there is some company or person willing to sell out trustworthiness for an extra dollar. Consumer protections get dumped in favor of more profit.
LLMs start feeling more like a dummy than the amount of ill intent they get from other places. So yea, I can see how it happens to people.
I can see a day when even that won't be trustworthy, because too much AI slop output will wind up in the search corpus. But I don't think we're there yet.
Even past the summary and the ads a huge amount of results that come back from both google and DDG are AI generated. It's sometimes harder to find a reliable source for information in search results these days than it was 20 years ago.
If it was easy to look up/check the fact without an LLM, wary users probably wouldn't have gone to the LLM in the first place.
Yeah, fair point. "Misleading" would be a better term, perhaps.
If it's saying something like "you are right" it's because it's guessing that that's the desired output. Now of course, some app providers have added some extra sauce (probably more tradition "expert system" AI techniques + integrated web search) to try make the chatbots more objective and rely less on pure LLM-driven prediction, but fundamentally these things are word prediction machines.
It's one thing to say you have found an effective method to counter LLMs' "positivity bias", but do you really not understand human psychology here?
People respond positively to other people telling them they are right, or who like them. We've evolved this psychology, it's how the human mind works. You tend to like people who like you, it's a self-reinforcing loop. LLMs in a sense exploit this natural bias.
> I am constantly amazed at how much they "fall" for the LLM, often believing it's sentient.
Why are you surprised? This is the illusion most AI companies are selling. Their chat-like interfaces are designed to fool you into thinking you're talking to a sentient being. And let's not get started with their voice interfaces!