x86_64 seems not to care much. I've had a UART mapped as normal memory, and it somehow still worked.