With unsafe, you’re telling the compiler “I’ve taken extra care to make sure that what I’m doing is safe and doesn’t break your rules” and the compiler can go ahead and assume that you don’t, in fact, break the rules, and therefore can verify everything else as if the rules never got broken.
In less safe languages, the entire program is “trust me, it’s safe”, while in rust only the parts flagged as unsafe are.
The point is that you should only use unsafe when 1. It’s absolutely necessary for functionality or performance and 2. You have verified and are very certain that the code is correct.
That’s a very useful property to have.
Obviously you should try to avoid writing unsafe Rust to begin with.
As for the actual unloaded question, "What's the point of unsafe in Rust?" it is to contain and make it easier to identify sources of UB.
Maybe in the future the unsafe code will go down to 1%, bringing that to two orders of magnitude.
Of course, only time will tell if that is true or not, but from experience I’d be willing to bet it is.