1. It's really really hard to verify that you have not left a vulnerability in (for a good time, try figuring out all the different "standards" needed in x509), but, more importantly,
2. You already have options for a reduced attack surface; You don't need to use OpenSSL just for TLS, you can use WolfSSL (I'm very happy with it, actually). You don't need WolfSSL just for public/private keys signing+encryption, use libsodium. You don't need libsodium just for bcrypt password hashing, there's already a single function to do that.
With crypto, you have some options to reduce your attack surface. With HTTP you have few to none; all the HTTP libs take great care to implement as much of the specification as possible.
That's actually not really crypto, though - that's writing a parser (for a container that includes a lot of crypto-related data). And again... if you import a 3rd-party x.509 parser and you only need DER but not BER, you've got unnecessary bloat yet again.