> When I use LLMs I leave things as open as possible. I just give them the information they need and my ask.

How do you handle security?

Both “internally” against e.g. data loss, I’m assuming via limiting the harness, and “externally”, i.e. stuff like prompt injection risks?

This vibe people sentiment is not wrong per se.

If you want outlier performance from these models it is best to just ask in the most high level prompt of the most minimal harness and let them loose.

Any extra information reduces their performance.

However, as often as these models output masterpieces, they also produce utter garbage so our current choice is for them to have a process to follow that can be reviewed by humans and LLMs.

That works for well trod paths, e.g “fix ci” works exceedingly well. “why app slow” obviously doesn’t work because the task is underspecified. But in order to properly specify you either need an experienced engineer who knows how to narrow the problem domain, or you have to provide some template instructions/output formats (e.g, skills) which will invariably never fit the problem perfectly