Most people who use drug as a pejorative term also wouldn't use it to describe caffeine.
That's primarily because it's most commonly diluted 90%. Spirits (or everclear) have much lower intake required.
GHB ~8x
Ethanol ~10x
Methamphetamine ~10x
DXM ~10x
Cocaine ~15x
MDMA ~16x
Codeine ~20x
Methadone ~20x
Mescaline ~24x
Flunitrazepam ~30x
Ketamine ~40x
Phenobarbital ~50x
Nitrous oxide ~150x
LSD ~1000x
Cannabis/THC ~1000x or greater
Citation: Gable RS. Comparison of acute lethal toxicity of commonly abused psychoactive substances. Addiction. 2004;99(6):686–696. doi:10.1111/j.1360-0443.2004.00744.x
Statins for example are probably 10x to 20x (quick napkin math based on an animal study I found). Tylenol is 10x and is OTC.
Personally I consider alcohol to be much safer than many other substances given the form it typically takes. Utterly irresponsible binge drinking aside, most people will suffer serious adverse effects long before they're in danger. Compare that with pills or injections where an OD can be entirely inadvertent even when observing basic common sense.