Wearing gloves and washing your hands five million times is more annoying than just using lead free.
And even with perfect precautions, the flux expands and makes it spit tiny little balls, which then contaminates your whole work area with a somewhat fine dust.
RoHS was more about removing lead from electronics as a whole than health risk to an individual from soldering. To put it into perspective, if you lived in a city before the 1990s, you've already inhaled more lead than you ever will from soldering.
But in a non-occupational exposure setting you're not likely to have issues if you solder occasionally even with lead solder and don't take any precautions.