Not just your ISP. If an attacker slipped a device onto your LAN and also you happened to be sshing to a new box for the first time then TOFU poses a problem. But that's an awfully limited attack surface. It's similar to the difference between leaking a fax while it's sent versus leaking years old emails that are just sitting there on an internet accessible server.

As for your ISP I think you should never rely on TOFU over the public internet. If you really don't want to do ssh certs it's easy enough to make the host key available securely via https.