A `thread_local`'s actual virtual address is not merely what the pointer contains - it's an offset from some other virtual address stored in the FS or GS register (on x86-64), which is swapped when you change thread. Casting the pointer to `intptr_t` does not retain the segment base address - only the offset. The pointer is not the absolute address.
If you dereference in another thread, it's the same offset, but from a different base address.
There may be other things besides segment registers on other architectures that also make it unsafe. The C standard makes no guarantee that you can safely dereference a pointer cast from `intptr_t`.