First, I would just naturally assume you are wrong as Fermi would have known that FTL was practically impossible.
Fermi evidently made the comment in the 1950s at Los Alamos, as I understand it became widely known due to Carl Sagan's 1963 paper "Direct contact among galactic civilizations by relativistic interstellar spaceflight"
https://www.sciencedirect.com/science/article/abs/pii/003206...
What the Fermi paradox involves is near light speeds, because, as Sagan says in the paper above "Interstellar spaceflight at relativistic velocities has several obvious advantages... all points in the Galaxy are accessible within the lifetime of a human crew, due to relativistic time dilation."
There is also the Michael Hart paper "Explanation for the Absence of Extraterrestrials on Earth" from 1975 but I have not found any satisfactory for open reading source of this online.
At any rate it seems absurd that any famous physicist of the last 90 years at least would ever have suggested that aliens should have been here because they would have had faster than light travel. You are essentially saying Fermi thought Einstein's Theory of Relativity was very much wrong.
on edit: I said 90 years to hedge bets in case there was some really old guy still hanging around in 1935, science progressing one funeral at a time and all that. But it seems really unlikely that there would have been.
"York believes that Fermi was somewhat more expansive and 1‘followed up with a series of calculations on the probability of earthlike planets, the probability of life given an earth, the probability of humans given life, the likely rise and duration of high technology, and so on. He concluded on the basis of such calculations that we ought to have been visited long ago and many times over. As 1 recall, he went on to conclude that the reason we hadn’t been visited might be that interstellar flight is impossible, or, if it is possible, always judged to be not worth the effort, or technological civilization doesn’t last long enough for it to happen. ” York confessed to being hazy about these last remarks."[0]
The recollection of the conversation by all the parties present is fragmentary and disagrees on exactly what was discussed but the gist of it appears to be about the feasibility of interstellar travel, not colonization or percolation or whatever you want to call it. The whole question of "where are they" doesn't make much sense when you think of it in terms of one off events such as permanent settlement of a planet using generation ships or other feasible means of slower than light travel simply because you would always miss each other in time. Either we wouldn't be here because they settled Earth first and interrupted our development or they're not going to show up for another million years.
I'm not contending that Fermi necessarily believed that the theory of relativity was wrong or incomplete but that the essence of the Fermi Paradox is that either interstellar travel is impossible due to distance + time required or there is some way to overcome the restrictions of distance + time which during this informal discussion over lunch seems to have supposed faster than light travel as the solution. Based on the recollected pieces of the conversation I'm pretty sure Fermi's napkin math didn't involve fudging the likely duration of a technological civilization up to millions of years, the likelihood of that happening is certainly less than Einstein's theory being incomplete.
Honestly, the argument as it is proposed today should be attributed to Hart since it conforms exactly with what he proposes so from now on I'm going to call this form the Hart Paradox.
[0] https://upload.wikimedia.org/wikipedia/commons/5/54/%22Where...