Frankly I don’t think this is that relevant. When software breaks, most people don’t want to get support from another piece of software. They want human support. As the models get better, I’d wager the problems that fall through will get increasingly messy by nature of a) being hard enough to stump the model or b) the model running in circles making changes that obscure or worsen the problem. Especially if you have non-technical users who can’t accurately describe what they want or the problem with the current state.
Benchmarks and guardrails are nice and all, but as you said problems will never be 100% self correctable. Models are good enough until they aren’t. I like to take a “what’s the worst that could happen” approach to planning support. Murphy’s law and all that.
> And as a last point, most of our users are not creating business-critical features with Vendo right now.
Just give it some time… you’d be surprised what becomes business critical in the eyes of a customer. Sooner or later a report will become part of an accounting workflow etc, and it’ll break on the wrong day and suddenly payroll is late. You know how many Excel sheets are “business critical”? Same thing.
An alternative model you might consider is selling direct to the end user companies and taking a partnership approach to the apps you work with. End users come straight to you for help, you have a list of integrated apps you already support with the required plan/license from that vendor plus custom MCP support and features that don’t need an API, then sell professional services engagements if an end user client wants you to work with one of their vendors to set up a new integrated app. That way you reduce support friction, and your end users are aligned with your buyers.