This post is modded down (I think because of the "roll your own crypto vibe", which I disagree with), but this is actually spot on the money for HTTP.
The surface area for HTTP is quite large, and your little API, which never needed range-requests, basic-auth, multipart form upload, etc suddenly gets owned because of a vulnerability in one of those things you not only never used, you also never knew existed!
"Surface area" is a problem, reducing it is one way to mitigate.
Again, you run into the attack surface area here. Think about the Heartbleed vulnerability. It was a vulnerability in the DTLS implementation of OpenSSL, but it affected every single user, including the 99% that weren't using DTLS.
Experienced developers can, and should, be able to elide things like side-channel attacks and the other gotchas that scare folks off of rolling their own crypto. The right solution here is better-defined, well understood acceptance criteria and test cases, not blindly trusting something you downloaded from the internet.
1. It's really really hard to verify that you have not left a vulnerability in (for a good time, try figuring out all the different "standards" needed in x509), but, more importantly,
2. You already have options for a reduced attack surface; You don't need to use OpenSSL just for TLS, you can use WolfSSL (I'm very happy with it, actually). You don't need WolfSSL just for public/private keys signing+encryption, use libsodium. You don't need libsodium just for bcrypt password hashing, there's already a single function to do that.
With crypto, you have some options to reduce your attack surface. With HTTP you have few to none; all the HTTP libs take great care to implement as much of the specification as possible.
That's actually not really crypto, though - that's writing a parser (for a container that includes a lot of crypto-related data). And again... if you import a 3rd-party x.509 parser and you only need DER but not BER, you've got unnecessary bloat yet again.
Good luck