You'd still need to memorize the IP address without DNS, otherwise how would you know to which server to connect?

The fact that a server can serve multiple vhosts and do TLS cert selection via SNI is not related to the lookup of what server to connect to.