I think you're underestimating both their reliability for standard problems and the usefulness of that level of reliability.
replyThis is a good point. Opus does some silly shenanigans sometimes but then catches it later. It’s still an order of magnitude faster at getting to a working system than I am, for ones I don’t know.
replyIt’s really a dream for setting up a homelab
> out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
replyWhat are some examples?
There's a benchmark for this and a lot of models get negative scores because they're so unreliable: https://artificialanalysis.ai/evaluations/omniscience
replyI wanted some examples they actually experienced. Because I use these things daily and haven’t seen a hallucination in a long long time.
replyI saw a hallucination just this afternoon about a spurious ca cert error. Definitely happens less often, but I do need to correct it occasionally. Maybe once a week so it still requires vigilance.
replySearch a terminal with Claude Code for things like, “I got it wrong twice. I should look up the documentation instead of guessing.”
replyDoes it about once a day, that I notice.