To me the connotation is that it's always an unreasonable theory, such as one that requires thousands of people to maintain nearly perfect secrecy for an indefinite period, with no mistakes and dedication even when they'd benefit from blabbing.

In contrast, imagine a 2-person conspiracy where both have lifelong reasons to keep it secret. Yeah, I could have a theory about that conspiracy existing, but it wouldn't be, y'know, a conspiracy theory in the usual sense.