The biggest challenges of Georgism are that it is basically communism for land (George straight up admits this in one of his books) and creates some issues with efficiently allocating land resources, especially bad with the fact that it can wipe out land speculators which perform an important role in doing time-allocation of land. But it's probably worth the tradeoff if you can eliminate the other taxes.
Interesting, I have always thought the opposite. My undertsanding/reasoning: It's extremely difficult to find land for good purposes because speculators maintain land banks, preventing better uses of it. The speculator causes a ton of market friction, and the tendency for people to hold onto land because of limited supply are a fundamental hindrance to so much economic activity.
If there's a high carrying cost to land, a lot more of it will be on the market and available for people to use when they need it. Especially as land values rise, which is the most important time to reallocate land. Rising land values are exactly the time that the land speculator holds tightest, because they want to sell at the peak, not on the way up.