> give them a json specification, they're not always amazing at following it

Used correctly, this hasn't been true for a quite a while. Most inference engines have a form of grammar constrained decoding. See, for example: https://vllm.ai/blog/2025-01-14-struct-decode-intro