I think all of those are very naive approaches. What I currently do is script out screenshot + application state pipelines and then run those screenshots and states through clef with questionnaires. The scripts are deterministic, clef costs pennies or less and is also reliable, the questionnaires are derived from the script with escalation to stronger lmm's if necessary (historically very infrequent from my usage). It isn't a "drop in" solution, but it is fairly trivial to set up and you can attach it to CI. It isn't perfect, but I've found it to be more successful and way more cost effective than throwing Claude or GPT models at it regardless of harness.
replyThis sounds a lot like what we call an "AI check" in a regular Momentic test. These work great on known flows, and are actually one of the main reasons a lot of our customers love Momentic. Mo doesn't replace that; it builds on top of it by adding the ability to discover and test new flows that you didn't previously write scripts for, or don't want persistent scripts for. In the near future, Mo will also be able to propose durable tests that you can rerun later (essentially, exactly what you're describing), which will decrease cost and increase consistency significantly for already known flows.
reply