upvote
Both of these are better addressed in the books. It was an intentional choice to have no override and no maintenance access. And book 2, Colossus and the Crab, actually spends a bunch of time with Colossus testing the rationality of various humans.
reply
Point #1 might seem unrealistic, but it's exactly how IT security of most companies operate now: "We are concerned about malware so we give full control of our systems to CrowdStrike". That is, having a single point of failure is shocking common.
reply
I've worked with companies whose infosec dept. is little more than "see tool alert, ask user what's going on", and then keep searching for the right _tool_ than injecting any human agency in that loop.

If any role is ready for an LLM to take over (or even a shell script), it's that one.

reply
> Second: Colossus' plan - and its ultimate actions - assume everyone else on earth is a nuclear-disarmed-rational-actor

The plan would still work. Colossus couldn’t be destroyed with nuclear weapons and would retaliate against any attack. It could force compliance of conventional forces as well, and force automation on them, also force populations to rearm it.

In the end, the population would appreciate the eradication of poverty, hunger, disease, and the surplus from not maintaining military capabilities. Colossus could afford democratic institutions while acting as a guard rail against humanity’s worst impulses.

reply
Colossus was not a panopticon, it was operating on limited intelligence and information about the world. Now, consider a hypothetical secret and clandestine science and engineering team (think: Black Mesa East) could exist completely hidden from Colossus and fabricate a workable fission bomb, then place and detonate it somewhere as a false-flag attack that Colossus would act against.

...or even just from recent middle-eastern history: an outrageous death-cult militant faction like ISIS.

reply
Colossus didn’t have infinite intelligence, so it might be possible to hide an effort like that. Not trivial though, as it would have satellite surveillance and likely cooperation from governments who fear what it could do as a retribution. This incentive should be sufficient to make governments police violent activist groups.
reply
> difficult for me to suspend disbelief

Were you able to suspend your disbelief when watching Idiocracy [1], either in the year of its release or in the subsequent decades? (^;

[1] https://en.wikipedia.org/wiki/Idiocracy

reply
You are rational. There are plenty of moments where doing the rational thing would end the story.

You are not the president of the United States. That is Donald Trump.

Do you see how the plot is consistent?

reply