First, in the rare cases you describe (when was the last time you stepped on feces?) one would indeed leave their shoes outside. But in the day-to-day your shoes have some dust on them and that's it - you won't even notice it's there when compared to the dust that already accumulates daily. If you live in a climate where this assumption doesn't hold (snowy areas come to mind) then the equation changes.
Second, the floor is usually stone and therefore designed for being walked on with shoes. I would feel awkward walking on some polished wood, but if the difference between inside and outside is "this one is less coarse but just as hard" then I wouldn't see a good reason for taking my shoes off. I would probably avoid the rug, but that's me.
As for comfort, keeping your shoes on also means one less thing to do when going in and out. Do not underestimate the path with less friction.
So shoes on is not universal in the West.
Try this , when was the last time you stepped somewhere that had previously been stepped on by someone who had walked on a spot which had been stepped on by someone who may have stepped on a spot trod on by ... go far enough back and it's all had feces on it to some small degree.
An easy way to keep this concept clear and understand the reasoning behind a 'leave the outside shoes out of the house' is something learned as a child.
Just like there is a Law of Conservation of Mass, and a Law of Conservation of Energy, and these two are the same law as defined by Einstein, there is also a Law of Conservation of Filth. The law may be stated as: "A spoonful of wine in a barrel of pigshit is a barrel of pigshit. A spoonful of pigshit in a barrel of wine is... a barrel of pigshit."
Taking off the outdoor shoes to come indoors changes the "is there trace amount of feces on my dining room tatami floor?" risk from "I cannot be reasonable sure," to "I can be reasonably sure that there is none." If guests find that too much friction to understand and accommodate, one might argue for a different set of guests who are less accepting of an allowable amount of pigshit in their wine.
I'm not going to say that every type of floor is to be walked on, with rugs and tatami mats being perfect counter-examples. But if a five-star restaurant doesn't ask me to take my shoes off (not evening Japan, AFAIK) then keeping them on is, from a sanitary point of view, probably okay.
And of course, toothpaste is a cleaner, so it's not relevant.
But I find this all amusing, because wherever you take your shoes off? That's where all this supposed crap sits from all these shoes left outside. Which means you are walking in your socks in that area, after shoe removal, and entering the house.
So unless you levitate out of that area, your feet are still covered in this supposed crap.
The shoe rule in Japan stems from damaging floors. And checking your feet, shoes or not, is just prudent. But your feet or clean looking shoes?
Equally dirty.
However, later this changed, in good part because much of the population moved from bigger rural houses to smaller apartments in the cities. In a small apartment, there was not enough space to have at the entrance a big enough wardrobe with dedicated places for the visitors to store their shoes, and to provide slippers for them. Moreover, a small apartment was easier to clean after the visitors left.
Thus the customs changed, and it became normal when visitors came and attempted to remove their shoes, according to tradition, to assure them that there is no need to do that. Nowadays I think that few remember that in the past it was really mandatory to never enter a house with your shoes on.
Anecdote, I recall reading about older family members being disgusted by the removal of outhouses and the idea of shitting indoors.
I have to imagine the person who leaves their shoes on either never goes outside, or just has filthy floors.
Unlike the commenter above you, I don’t really agree that it’s completely sanitary, though. I’m just saying that keeping you clean is actually pretty far down the list.
Another thing about whether the floor is sanitary: that changes a lot based on how you use your floor. If you eat, sleep, or sit on the floor often like in a traditional Japanese home, keeping the floor clean is a lot more important.
Keeping your feet clean and protecting them from surfaces are the primary purposes of shoes. Some people need them for support after having worn shoes too often (my parents are in this situation and have a separate set of "indoor shoes" just for this), or some sort of disability that needs those super-wide-sole shoes, but in general they are not for supporting your feet.
Hiking boots can significantly enhance stability and of course walking over sharp rocks becomes trivial.
I also forgot to mention steel toed boots, no slip restaurant shoes, and other injury-protective shoes.
Then you’ve got temperature: hot pavement, cold snow.
Nah, the main function of shoes is (to support your feet and) prevent injury. Cleanliness is second on the list.
[ed.: added braces]
Well, more directly, if you want to be going around barefoot outside, you have to go barefoot for a while in order to harden the skin on the bottom of your feet. It will happen naturally, given enough time, but walking around outdoors with soft feet will hurt like crazy. Bones and muscles are secondary.
In an American context, the primary reason to wear shoes is to prevent parasites on the ground from infecting you through your feet. Shoes are the single most important public health measure in American history, beating out handwashing.
But not on hard floor.
Removing one's shoes before entering a house isn't a thing in South Africa.
Filth can sometimes be a good thing.
Take a second, get a tissue, and go wipe the bottom of your shoe.
Then sanitation itself is a but of a moot. Bacteria can spread million different ways, it’s mostly futile fight. If you wear flip-flops all day and don’t wash your feet, don’t shower after using public bathrooms, etc - whats the point?