Who else than Cloudflare (or similar company in expertise and size) would be a better fit to implement distributed consensus?
This is obviously true, and there are surely more web devs than, say, compiler engineers.
But then the logic seems to go as far as to imply nobody does or should be doing these things. Where do they think the “off the shelf” solutions came from?
I worked at <big internet tech company>, and many critical systems written in the 90’s and early 2000s used bespoke compilers, caches, and consensus protocols that were hand rolled by the original developers.
Back then, most of these ideas were already present in academia. But the industry tech wasn’t quite there yet to run these huge services at scale without running into problems (paying a vendor, licensing, the open source solution not quite there or tunable enough to easily integrate it at the needed scale, no company wide consensus on a shared approach yet).
A lot of these handwritten implementations eventually informed the “off the shelf industry solutions” we have today.
I think that’s what gets me down about tech these days. There’s a weird almost anti innovation attitude, which is the opposite of what made me fall in love with this profession. I’m not saying to hand roll a consensus algorithm at your next start up. But there’s definitely a vibe these days that any sort of theoretical, creative, or innovative thinking is suspect. Get back to selling ads!
Yeah they kinda suck and make mistakes right now, but tech only gets better.
It’s no different than 3D printing. Instead of cutting and joining a bunch of stock parts, you just print the desired part.
In the same way printing with plastic kinda sucks, the tech will improve. Now you can print in metal or in resin without supports.
The future will be take all the ideas from various off the shelf products but tightly integrate them together for my specific use case