In particular, the hereditarily finite sets are in correspondence with the natural numbers.
There’s a fair bit of mathematics that relies on the axiom of infinity. Uh. And a lot also uses the axiom of choice (or something equivalent), though ZFC doesn’t have a greater consistency strength than ZF (both can prove that if either is consistent, then both are).
I think it is fair to say that one’s choice of foundations does matter at least a little bit, though typically not all that much.
That’s not to say that it is always necessary to state what foundations one is using. One can understand a work to be done in any foundations in which it works. Though sometimes it is important to specify.
Reducing the correctness of the lean code here to set theory is then also reducing it to everything you can reduce the correctness of set theory to.