either way your phone number, profile info, and a list of your contacts will be sent to signal and stored forever in the cloud. It isn't hard to tell who you are from your social graph
It also wouldn't store a list of your contacts, your phone number, your name, and your profile pic in the cloud forever while lying about that in their privacy policy either.