All memory safety on von Neumann architectures is specified with respect to a TCB. For Rust, the TCB includes the compiler. The TCB can have bugs. This doesn't invalidate the memory safety promise, given that it is made with respect to the TCB.
In contrast, memory-unsafe languages don't even try to have a TCB relative to which memory safety is specified. That's the main difference.