Merely calling it invasive does not answer that question, ergo it is not a useful term in this conversation.
And yes, timescales do matter. A species can become critical to its new ("invaded") ecosystem within a few hundred years such that removing it within a few years will cause catastrophe.