That's my experience on M4 Mac Mini and M3 MacBook Air.
Is it amazing for LLM stuff? No, but that's the same story for anyone else _not_ spending boatloads, or willing to hack on weird mining cards.
Given how much more you can get buying used versus new, I'd go used unless there's some feature/characteristic only the new hardware has.
Having said that, I could see stuff slowing down over time with OS updates/etc. I've had my M1 air for a while, and some UI stuff was slower after specific updates (I think glass in particular).