There is a down to earth guy on YT that performs a series of tests against LLMs running on non-god-tier commodity hardware. He will likely be testing this soon enough.
replyhttps://www.youtube.com/@lukesdevlab
I don't know if that is what you are looking for or not and as always your experiences may be different.
So much potential for that channel. He's got a nice range of tests and a no nonsense presentation style.
replyHowever, watching tests of heavily quantized models that weren't designed for it (non-QAT) is frustrating. There's no way to tell if the actual model fails because it's dumb or if the lobotomy made it that way.
I noticed he does pay attention to feedback on his videos and I think some people have pointed that out.
replythaaanks man, this channel seems really informative, although < 10K subs only !
replyIt's a relatively new channel - but yeah - I feel the guy puts a lot of effort into what he does and deserves more subs.
reply