People aren't saying that. They are saying that other candidates will put in more time. If I do a professional job in four hours, then an equally talented candidate who puts in eight hours will produce a much more polished effort, and they will get the job.
I guess the fix is to ask the candidate to pick a four hour window, and to ask them to complete the task in that time.
I understand that some people are concerned that they're competing with candidates who will put in 12 hours to do what they should be doing in 4. But that's not their problem. Their problem as a professional is to evaluate whether they can do the challenge in 4 hours; that's the expectation the job is setting.
It is perfectly reasonable for someone to look at the hiring process we're running and say "no, this communicates to me that this job wouldn't be a good fit for me". That's a good outcome! Most jobs aren't a good fit for most people; that's the whole challenge of hiring.