Consider reframing. Is the problem really that the task is tedious? Or that it's required? Maybe you should be leaning more on deterministic automations (which the LLM can help set up) rather than LLM-powered ones.
- Research and buy me a domain name for xyz product, set up all of its DNS the way I want for this product, set up email, set up app store submissions including screenshots and description, set up firebase notifications, set up my Google cloud project and any apis I need to enable for this app, deal with app store review feedback.
- Do an audit of my books, make sure every transaction has a receipt uploaded, and make sure the receipt matches the journal entry. Pull statements from all my my bank accounts and reconcile against my books. Fix mistaken transactions with my confirmation.
- Freely push to my source control, but only to specific branch prefixes (surprised that GitHub, gitlab, gitea, etc don't have this built in already). Merge with my confirmation.
- Freely read the subject lines in my Gmail and full emails from my accountant, freely search emails (but still can't read them), freely compose drafts, ask me to read full emails other than my accountant's, ask me to send drafts. No way for it to read password reset links, for example.
Not to mention that I only have to have a single key that gives access to all my services. If the key leaks, I just have to rotate one key because the agent never has access to the 30 upstream service keys. If I need to give weaker or stronger keys for a specific agent, I can easily configure that in one place (and I can even request that the agent make that change for me with my confirmation). I don't need to create 20 new keys every time, and I don't need to remember the intricacies of every upstream service's UI for creating new keys.
Edit: forgot to mention one more really convenient thing the API proxy enables: I can ask my agent to just configure things for me e.g. changing CI settings, adding Google workspace users or groups, etc. without having to use the to UI. So now nearly all of the admin configuration for all services is done from a single interface (my agent cli)
For instance: that's a ton of surface area. Did you have your AI agent decide which endpoint is destructive or did you manually do this?
For instance: has this classification been wrong?
I ask this because I've used a similar thing before and it failed in the ways I've named.
This setup also permits your agent to exfiltrate enormously sensitive data (website gives the example of QuickBooks invoices and Cloudflare config) without your involvement.
And yes, it has tried to make bad requests! Which it wasn't able to do because it was either rejected outright or asked me for confirmation and I didn't let it through.
The "surface area" you're talking about is just the coverage of the service APIs, which is there just as a nicety to give a semantic preview of what the request is about to do e.g. we know what a gmail request looks like so we can render a preview of the attachment the agent is sending so you can inspect it. Otherwise you'd have to inspect each http request (which you can always do by using the generic http acl).
I somewhat agree with "exfiltrate enormously sensitive data". The thing I'm paranoid about is catastrophic mistakes that can instantly kill the business e.g. "Oops I sold the domain name". Leaking financials is not great, but also not catastrophic. Deleting all our books would be catastrophic, which the underlying QuickBooks key allows you to do!
But in any case, you can choose your level of paranoia and give exactly the amount of access you're comfortable with. For example, I view read access to emails as catastrophic (must worse than read access to financials), so I don't let it do that.