Exactly, "you can't win" is grossly under-specified. The goal of the demo is just to show that laws can't be broken. Yet, if that's your only law, the AI can do whatever to protect it - including changing how the character moves, or even removing the flag entirely!

So, yes, that's the issue with laws: they only protect what you remember to write. They're not a silver bullet. But they can still be incredibly useful, because it takes a small law to protect against entire classes of bug, covering your entire codebase. For example:

LAW: "the sum of all balances in this contract must be zero"

This one liner would have prevented Ethereum's infamous "The DAO" hack, where millions were stolen and almost undermined the entire project. But if your laws are under-specifying or ill-specifying your intents, Bend can't do anything to protect you.

Yeah all of these attempts to try and control AI outputs by using language fail to grasp that language is itself the problem. It is a closed system with no ground truth. Words only point to other words in an endless self-referential chain. There is no "closure" in language, ambiguity cannot be erased.

Imo LLM researchers would benefit a lot by reading what continental philosophers have said on the topic of language. Barthes, Sarte, Deluze, Derrida, et. Al have a lot to say that explains why we're running into the same problems over and over.

> writing the laws becomes a bigger problem than writing the code itself.

But that's how it is anyway, no?

Defining clear boundaries and clear goals is the hardest part. If you get those right everything else is rather trivial.

Side thought - I like the idea of this as a game, where you’re essentially fighting a monkeys paw / tricky genie. Not totally sure it’d work but I like the concept of trying not to get caught out.

I got the same result when I tried the "remove the walls" example. I followed up by telling it to reimplement up/down/left/right movement without reinserting the walls and it basically made the square with the flag "unenterable". Like with a force field.

Respectfully I'm not sure if I share your worry though. You would have the same problem if you wrote extremely thorough / exhaustive unit tests or extremely precise types and didn't allow the LLM to amend them. You're basically shrinking the state space of what is considered a "correct" program per your spec. That the LLM has to get extremely creative to break your program is a _feature_; in my experience, an LLM does not have to get creative at all to break a typical program written in an enterprise setting, and that's unfortunate.

Heh, We're writing Asimov in real life after been warned to decades.

> writing the laws becomes a bigger problem than writing the code itself.

Of course because at its limit programming is basically defining desired behaviour under all circumstances and logical conditions.

Well described the problem scope.

I wonder if harness-hooks + Jev (equivalents) could semantically lint for `sloppy_law` etc when ever they are edited