Yes, one of the obvious ways to me to use these models is to tell them to write such a program. It can then go figure out data extraction and normalization. This is "the harness improving itself". Have it write tools to do its tasks.
replyI think you missing the problem with composition itself. The tool generation is the easy part. Knowing which tool is needed, specifying it correctly, validating it against current context, knowing why it failed, and deciding when it needs replacement are separate tasks, and they compose, and composition compounds and fails.
replyI've never found that to be a problem (using codex since the beginning of the year and having it make dozens of shell scripts for itself). It adds usage/help without me ever having had to ask, and if the tool has an issue, I've found it will generally debug and fix it on its own.
replyI may not be personally familiar with the latest and greatest moving target in agentic AI, but I am familiar with AI-generated "slop code". What I see is consistently defective, to the point where I would not trust an AI-generated data pipeline to produce remotely accurate results, even though messing up a basic data pipeline that much is a difficult task for humans. (Then there's the tendency for an LLM post-processing the tool output to smooth over such flaws, massaging or fabricating conspicuously missing or corrupt data to hide what would normally be extremely obvious warning signs.) If it is, as the proponents claim, even possible to entice these systems to produce decent programs (other than by direct plagiarism), it must require skill way beyond that of the average prompter with 18 months' experience to explain my observations.
replyWhen I say "write an ordinary computer program". I mean just writing the program, in a programming language. Your comment expresses disagreement with me, so should have started with "No", not "Yes".
I totally agree - designing a competent AI agent with a fully customized harness to successfully pull off this task is a much more challenging engineering effort than merely creating an ordinary computer program. Had OP made chatgpt write an ordinary program instead, they likely would have succeeded in their task.
replyYou said "Had OP made ChatGPT write an ordinary program" but that assumes enough structure exists to specify that program. If so that may be the right architecture, but also demonstrates why the "agentic AI" does not automatically solve the original open ended task.
replyYou have converted a fuzzy task into a conventional software engineering problem, and then relying on conventional software for the reliability :-)
> converted a fuzzy task into a conventional software engineering problem, and then relying on conventional software for the reliability
replyWhat can I say? This is how I get results from AI. It also gives me context & tools to fight the AI when I have to.