(1) It doesn't get any more difficult to fix your accent. But most people won't, because there's virtually no benefit to doing it.
This is related to
(2) Once you learn to speak a language, you're not at any risk of people thinking that you can't speak it, even if you speak with a strong accent.