> I love everything about this direction except for the insane inference costs.

If this direction holds true, ROI cost is cheaper.

Instead of employing 4 people (Customer Support, PM, Eng, Marketing), you will have 3-5 agents and the whole ticket flow might cost you ~20$

But I hope we won't go this far, because when things fail every customer will be impacted, because there will be no one who understands the system to fix it

Inference costs at least seem like the thing that is easiest to bring down, and there's plenty of demand to drive innovation. There's a lot less uncertainty here than with architectural/capability scaling. To your point, tomorrow's commodity hardware will solve this for the demands of today at some point in the future (though we'll probably have even more inference demand then).

I worry about the costs from an energy and environmental impact perspective. I love that AI tools make me more productive, but I don't like the side effects.

This is the wrong way to see it. If a technology gets cheaper, people will use more and more and more of it. If inference costs drop, you can throw way more reasoning tokens and a combination of many many agents to increase accuracy or creativity and such.

I mean theoretically if there are many competitiors the costs of the product should generally drop because competition.

Sadly enough I have not seen this happening in a long time.