lets say you have a table that is partitioned. how do you lint/format "any select into this table MUST include the partition key in the predicate and any join must include it in the on." I'm not personally familiar with any static analysis tool that does this but its trivial to implement with an llm prompt. trivially easy to add to your automated PR reviews.
replyI would tell the LLM to write a custom rule/check for whatever the scenario is. Then when the CI gate is run, all my custom checks get deterministically run.
replyElixir is where I prefer to build software, so it would be creating a custom Credo rule.
It's not always possible, or at least trivial. For example how do you enforce "prefer to reuse existing code over making a copy"? Is there a static analysis tool that will detect two pieces of code that do the same thing?
replyYes it’s possible:
replyWrong, custom "specs" i.e. schemas, are literally all we have for "real" guardrails with LLMs.
replyhttps://developers.openai.com/api/docs/guides/structured-out...
Nothing else operates on the logprobs level and literally bans continuations that fail your schema.
Enforcing structured outputs from LLMs is not the same thing as using linters, formatters, static analysis to control how an agent writes code.
replyNo, it's not. It's strictly better.
replyCan you share examples, links to Github of this approach? I'd like to learn.
reply