Anyone who is mad that you politely passed them at a safe speed is just too sensitive about these things. You're totally fine there. But "room to pass safely and silently" could still piss people off depending on your speed and distance.
Back peddling or coasting gets people’s attention. Though moving slowly uphill and needing to back peddle is a bit of a test.