At every UX job I've had we generally ran usability tests with 6 participants. This was before the online usability testing tools proliferated and we had a lab in our office (one way mirror with observation room and everything). Every UX researcher I worked with said based on the available research and personal experience it is as you said: more than 5-6 participants would hit diminishing returns pretty hard and wasn't really worth it.

Probably you learned that from them! The first user is 31%, from their testing.

https://www.nngroup.com/articles/why-you-only-need-to-test-w...

https://www.nngroup.com/articles/how-many-test-users/

Its not just QA and bug hunting.

You want to hire your target audience, which may be vastly different from yourself, and suddenly you discover that the language is throwing them off, the color scheme brings different meanings, and they just don't understand the flow which felt completely natural for you.

It's more like the first user catches all of the critical bugs and a handful of users catch all of the high impact bugs. Cause that's kinda the definition of the impact of critical and high impact bugs.