You’re absolutely right, I shouldn’t have rented a 200 GPU cluster for $35,000/hour. That’s on me.

[Search: Can I refund Google cloud?]

It looks like we’re not able to ask for a refund since we did actually use all of that compute intentionally.

Would you like me to write you a pleading email to send to the support team?

There’s a safer way to do this with nearly no added friction. Give it a read only API key. Then just ask it to write the API calls into a bash script and then read it and run it yourself. The agent can still inspect the live resources and diagnose and give you more commands to run. I do agree I wouldn’t give it create / write access.

Why? Isnt the API key scoped to a project and specifically made for this?

Are you confusing this with an OAuth token or something?

I've done this sort of thing before but with Vast. Pre-deposited some money online, then let the LLM request and manage a training run on an allocation. Worked pretty well without risking bankruptcy.

On the cloud side, nothing valuable existed, so the training couldn't ruin anything it didn't create. On the laptop side, I usually ask the agents to create named scripts for everything it needs to access, then those local script directory is green-lit with approve all. For cost, I kept giving it new budget in the 20-30 dollar increments.

I had to intervene a few times. For instance, as smart as the models are said to be (Astra), it would copy the full training run, train on the server, pull every checkpoint to the local machine, then run tests, update. So, the bandwidth bill was as high as training bill for the first 6 hours. It could have simply tested each checkpoint on the server, saved time and money, didn't occur to it until I said.