This is why cities are popular for this exact type of person. For centuries. People with niche interests move to a city, which by sheer density, have others with said interest.
Plenty of people have a reason why they can’t do it, but plenty do it and are happier for finding their community IRL.