+1. I noticed ChatGPT now pulls out a calculator or writes a script instead of hallucinating numbers. Initially it was terrible, but recently it seems pretty good... that said, if you're doing serious work with this you should probably double check. And if you're doing repeated work, obviously get it to write a script or formula instead of doing 1-offs.

It makes sense to use an LLM to write the queries when exploring data, but if you are building a dashboard that’s going to be used repeatedly, you should save the queries so they can be validated and improved over time. The LLM shouldn’t be rewriting a new query every time you open the dashboard.

Definitely. I’ve found Cursor + Grok to be the best budget models for this type of tool use, although I’ve yet to try out the latest Anthropic ones.

You can go “here’s a spreadsheet of some data, do a thing to it” and it’ll go through, have it plan out the type of calculations you want to do, and it’ll write you a program to parse your sheet, calculate what you want, then output an result sheet.

The key to good results with minimal effort seems to be the planning modes to me, combined with the /goal command. A little bit of validation before implementation goes a long way.

They’re very much trained to do this, in fact. Even Step 3.5 Flash, a medium sized open model from way earlier this year is trained to use python to calculate answers on the fly :)

I've seen so many "truthy" results coming back that suggest it's not even remotely as smart as you might hope.

I'm mostly starting to have some success telling TPMs and engineers to ask chatgpt to give them appropriate code that generates more accurate answers.