This is a real issue, where conditions that are legally considered sufficiently clear for visual flying aren't necessarily safe for it. A famous example is JFK Jr. He was flying over water on a hazy night with poor illumination without an instrument rating. This was considered clear and was perfectly legal, but he got disoriented and crashed into the ocean.

It's hammered into pilots that safe and legal are two different things, and you must evaluate whether the conditions are something that you can actually handle safely, not merely whether you're legal to fly in them.