It's true that anyone with the key and an Internet connection can move money out of a crypto wallet at any time, but the point of the cold wallet is that they don't get the key because it's kept on a computer (or other storage medium) that is physically offline and disconnected from the Internet. Same reason government intelligence agencies use a SCIF for any important information work: you keep the information physically disconnected from the Internet.

of course you can move crypto currency to an offline wallet and by offline I mean its private key is held in some physical form/device that is not connected to the network.

Only spending from that wallet needs online access to private key - depositing to it does not.