What can Signal do to annihilate your concerns without giving you access to their system?
At best they (very) passively fight against third party clients connecting to their official servers.
At best they (very very very) passively fight against self-hosting the server by not providing elaborate guides and having the public repo lag behind production.