undefined

points

[-]

I run a SaaS business on the side of my job and have been for 15 years. There’s a million questions I’ve never had the time to dig into although the data was there. Retention cohorts, free to paid tier conversions, subscription upgrades/downgrades and so much more. Just this week, I decided to just let an agent have access through psql and go nuts, writing all analysis to markdown files. Reading through it, there’s a few things it misunderstood and as a result, some of the analysis was flawed, but all in all I’m honestly mindblown. It would have taken me months to write queries and even just coming up with frameworks of how to think about these metrics.

by rick12901 hours ago|

prev|

[-]

Agreed. When I watch the llm start to explore the db - it really does impress me.

Can you expand on this:

You can even incrementalize the schema description process itself by way of the system tables. Intentionally not providing a schema description tool/document/prompt seems to perform better with the latest models than the other way around.

by bob10291 hours ago|

parent|

[-]

If you tell GPT5.x that there is a database it can query by calling ExecuteSql(query), but you don't bother explaining anything about the schema, it will try to figure things out ad-hoc. This has advantages for token budget because it will tend to only lookup the metadata for tables that seem relevant to the user's query.

If you have a gigantic data warehouse with 1000+ tables, there's no way you could fit all of that info into a system prompt without completely jacking something up in the blackbox. So, why bother trying?

Consider that the user's specific request serves as an additional constraint that can be used to your advantage to dramatically reduce the search space. Building a single prompt / schema description that will magically work for all potential user requests is a cursed mission by comparison.

by operatingthetan3 hours ago|

prev|

[-]

I think context windows are too small for an agent to actually do this properly yet. I have much smaller databases and with 1b context frontier models they still need reminders or nudging or come up with completely wrong stuff in response to basic queries.

Having the c-levels relying on this for off-the-cuff info seems ... dangerous?