Agree with your assessment of guardrails. They barely work on the best days. We need to flip the idea of “agent” on its head. The agent here is an agent of the user interfacing with GitHub. Not an agent of GitHub interfacing with the user. Prompts and guardrails cannot keep the agent loyal to the company. Stop giving these things any permissions the user doesn’t have, and recognize them for what they are: a different UI than web forms, but still the same security model.
replydeleted
replyThat last part is I think called negligence. And in some industries that becomes criminal negligence quite quickly.
replyMost companies I ever worked for inherently operate on criminal negligence, and even when addressed, have no interest in fixing it.
replyGuardrails are essentially part of the input. Saying "but we have guardrails" is like saying "but we do trust part of the input".
replyEither way, even if you trust 100% of the input, there is actually no way to guarantee that you can trust the output of the LLM. (Which, I guess, is also true for every dependency you pull in. But for those, you at least have ways to audit them.)
deleted
reply