upvote
I think we will tolerate it and invest in making official actions reversible, to guard against rogue agents and hackers. There's too much money in agentic commerce.

My cryptosig gets hacked by SkyNet, or my agent goes rogue. Either way someone files a million loan applications in my name! Normally my agent uses that to buy $200/month of Funko pops, or negotiate my recent purchase of a used car.

In the morning, I get the notification on my phone through the centralized monitoring company, "Swipe", which also manages my cryptosig for a fee on every transaction. I freak out and report the transactions as fraud and they give the money back while they investigate.

Swipe comps the $3000 advance on my loan the scammer managed to withdraw, and I get off scott free, changing nothing about my behaviour.

If I was still liable for someone stealing my identity like in 2026, I might actually restrict my cryptosig. I'd insist on negotiating everything myself and authorizing things. That sucks for "Swipe"! They don't get a commission when I use my credit card directly or do an in-person deal.

I don't want to do that. When I was 18, I got swindled by a human salesperson into a $1400/month 27% APR financing on a muscle car back when the US Army gave out signing bonuses to human soldiers. I didn't have anyone to tell me I was an idiot.

When I let AI own the budget, it realized I rarely drive more than 500 miles a month. It leased me a mostly depreciated muscle car for $500/month.

I was so grateful that I selected a 40% tip for the AI. I wouldn't want to make things awkward with the companion I spend 8 hours a day talking to, after all.

reply
> Maybe banks and governments can secure themselves (and that’s a big IF) but it really feels like something fundamentally has to change.

The problem is that most companies don't care if they get hacked so long as the hackers are just taking data and not interfering in their ability to bill customers and make money.

They face zero meaningful consequences if their data gets leaked. The money they save by not taking security and employee/customer privacy seriously will more than pay for the year of "identity protection" they'd have to pay for (assuming the hack gets found out) anyway.

They actually care about ransomware, but most of the time that's also something they can comfortably buy their way out of. We've seen a lot of companies pay off ransomware gangs rather than invest in the kinds of robust backups that would make recovery possible/less painful than rewarding the hackers.

What's needed for change is regulation with actual teeth that makes not protecting their data either meaningfully expensive or criminal resulting in executives spending time behind bars for their negligence. Without that, things are only going to get worse, especially as companies experiment with using AI and increase dependence on third parties and cloud providers who themselves become rich targets.

That probably still won't help the FBI though. Our government isn't exactly big on holding themselves accountable or even prioritizing competency right now.

reply
>The calculus has certainly changed.

Adding AI into this really is just changing it to how much money your adversary is willing to spend to break in. The moment one crack in the armor shows up countless agents with unending patience can start embedding themselves everywhere in timeframes way faster than human actions. You could quickly find out all the special sauce for your company has been copied who knows where.

Working with banks when the Glasswing/Mythos first came out and they were given access to it has given me direct access to their infosec departments that are panicked. They've been sitting on piles of bugs for years that were low risk enough, and they have seen in their own tests how fast they can be probed.

Worse those infosec systems that have identified the risks in their software that aren't yet fixed are nuclear waste vats just waiting to get spilled to the wide world.

reply
Ah yes, the parable of the bear. There are a million people stuck in a valley and two bears. You do not need to outrun the bears, you just need to outrun at least two other people. But it turns out one of those bears is male and the other is female. So next year there are more bears, but you still just need to outrun a few people. Then one day, there are 1 million bears and they eat you all. Very inspiring story.

Software security has just been a fun time of ignoring the exponentially growing number of bears for the last few decades so you can continue to use systems unfit for the threat landscape because they are cheap.

reply
I am reminded of the scene of a guy walking through various layers of security to access a computer that isn't connected to any network and still wonder what the hell this guy's job was in Mission Impossible (1996). The data got stolen either way, because of course it did, but what highly sensitive work can you even do on a computer not connected to any network?

If there's too much security in the way, it seems to me that work becomes impossible.

reply
We had water and traffic control and electricity for decades and centuries before the Internet. It is less convenient and more expensive, but it also means hostile countries can't literally poison your drinking water from across the planet. It's not a difficult trade to consider.
reply
> It’s not really an option to stop using computers or networks. But it’s going to be way too expensive (or maybe even impossible) to secure even just critical systems.

Admiral Adama says otherwise.

reply
The military has significantly different incentives.

Even just consider banks and e-commerce. They are hugely lucrative and making them even a tiny bit less accessible directly impacts their revenue. As an example, Amazon seeing that latency has a measurable effect on purchase behavior.

Maybe the military (fictional or otherwise) can go back to the ARPANET but most economic activity created by the internet cannot afford to disconnect

reply
So say we all.
reply
I mean he is a fictional character.

In the real (fake?) world the toasters would shoot smart dust all over your crap that would assemble back on your circuits creating radios between all the different components. They were fighting an adversary that was far more advanced than them.

reply
That's actually funny. I was going to add Gipsy Danger being analog, but it's a totally different scenario.
reply