upvote
>In this case, it is an advisory control only, because the LLM can also unhook that hook

you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.

reply
This appears to be what the claude code-review is doing:

https://github.com/anthropics/claude-code/blob/7ef6eec9d9ba8...

reply
I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`).

That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).

reply
So you are essentially saying "you can't, you can only safeguard from effects of LLM eventually ignoring it"
reply
I think things like CI, linters, hooks, etc started as defense against humans eventually ignoring it (low blood sugar, under pressure, never cared in the first place, etc).

These seem like they would naturally extend to LLMs as well.

(I'm not pro-LLM on balance, but the solutions of today feel similar for humans and LLMs.)

reply