Sounds to good to be true, but it works for the given examples!
What is the scope though? It produces garbage with larger documents with tables and figures. For example this large document, even with removed menu and headings: https://docs.redhat.com/en/documentation/red_hat_enterprise_...
replyCould it be it has only been fed small documents?
All of the suite test cases are small fragments of HTML to test specific things, so I probably do need a way to test bigger documents. Thanks for the feedback. I’ll add that to the to-do list.
replyI’ll try the red hat example later this evening.
That's an interesting approach. I'm concerned about the use of LibreOffice as your source of truth. Would it be possible to swap out LibreOffice for actual MS Word in this workflow? This also could reveal some libreoffice rendering bugs/edge cases that are worth filing bugs over.
replyI don’t have MS word installed on my development machine, so I haven’t done much testing with Word, but that does sound like a good idea to run the suite using Word. For what it’s worth I did not notice any issues with LibreOffice, even after running many iterations and tests.
replyI would add a +1 for testing w/ Word - the official Office suite runs some validation where only Word will show a "broken file" popup, even when nothing else does.
replyIn our case, clients use only real Word, so any machine-generated/mutated files (excel/ppt as well) need a pass through the real office executable.
Thanks for the feedback, I’ll add Word validation to my todo list.
replyIn my experience a docx that looks good in libreoffice usually looks good in word, it's the other way around that is usually more of a problem.
replyInteresting but how is an "autoresearch loop" different than creating a spec and X number of testcases and letting an agent run against these testcases and the spec?
replyScoring would probably be the big difference because the outcome is to optimize performance or optimize a score or a metric versus just a pass fail.
reply