"I have a vague recollection, which may not be accurate, that we talked about flying saucers and the obvious statement that the flying saucers are not real. I also remember that Fermi explicitly raised the question, and I think he directed it at me, ‘Edward, what do you think? How probable is it that within the next ten years we shall have clear evidence of a material object moving faster than light?’ I remember that my answer was’ 10–6.’ Fermi said, ‘This is much too low, The probability is more like ten percent’(the well known figure for a Fermi miracle.)”[0] --Edward Teller

"York believes that Fermi was somewhat more expansive and 1‘followed up with a series of calculations on the probability of earthlike planets, the probability of life given an earth, the probability of humans given life, the likely rise and duration of high technology, and so on. He concluded on the basis of such calculations that we ought to have been visited long ago and many times over. As 1 recall, he went on to conclude that the reason we hadn’t been visited might be that interstellar flight is impossible, or, if it is possible, always judged to be not worth the effort, or technological civilization doesn’t last long enough for it to happen. ” York confessed to being hazy about these last remarks."[0]

The recollection of the conversation by all the parties present is fragmentary and disagrees on exactly what was discussed but the gist of it appears to be about the feasibility of interstellar travel, not colonization or percolation or whatever you want to call it. The whole question of "where are they" doesn't make much sense when you think of it in terms of one off events such as permanent settlement of a planet using generation ships or other feasible means of slower than light travel simply because you would always miss each other in time. Either we wouldn't be here because they settled Earth first and interrupted our development or they're not going to show up for another million years.

I'm not contending that Fermi necessarily believed that the theory of relativity was wrong or incomplete but that the essence of the Fermi Paradox is that either interstellar travel is impossible due to distance + time required or there is some way to overcome the restrictions of distance + time which during this informal discussion over lunch seems to have supposed faster than light travel as the solution. Based on the recollected pieces of the conversation I'm pretty sure Fermi's napkin math didn't involve fudging the likely duration of a technological civilization up to millions of years, the likelihood of that happening is certainly less than Einstein's theory being incomplete.

Honestly, the argument as it is proposed today should be attributed to Hart since it conforms exactly with what he proposes so from now on I'm going to call this form the Hart Paradox.

[0] https://upload.wikimedia.org/wikipedia/commons/5/54/%22Where...