Anything that can be verified mechanically should be code. Only use LLMs to fill in the gaps where things are fuzzy. Don't fall for the idea that those harnesses are general purpose, make your own fit to your task with the guards and verification steps you need. Make the LLM create the harness even.
There is no amount of markdown that can make a machine generating plausible text generate truthful text, it just happens to be truthful because of what it was trained on. Nothing coming out of an LLM should be taken at face value.
The propaganda about LLMs being intelligent and able to "reason" is only serving the companies selling you tokens to waste on "prompt engineering".
But the mathematicians here in this thread, are having a hard time with these clearly dumb models, doing so well in proving theorems in their domains :-)
On the other end of it, something like 80% of resumes I receive right now are clearly hallucinated -- referencing accomplishments that are copy-pasted from the novel-to-our-company thing in the job description a candidate will be working on, usually claiming they did XYZ at big tech a decade before the thing existed, or similarly with languages and skills. The resume "tailoring" process just manufactures lies rather than tailoring actual experience to the actual job.
I don't know your prompt and setup, but my claude had no problems doing that task. The search index isn't live, so it can't find current gigs, but that is a tooling problem.
Why do you think there's such a thing as too long for an LLM prompt? You'll run into context window limits at some point, but the more verbose you are with what you ask of it, the better the results will be.
Trivially falsifiable:
"Context Length Alone Hurts LLM Performance Despite Perfect Retrieval"
https://aclanthology.org/2025.findings-emnlp.1264/
"Large Language Models Can Be Easily Distracted by Irrelevant Context"
As a human, not an LLM, I could interpret "including maybe opportunities driven from temp agencies based in geographically close countries" as meaning "including opportunities in nearby countries outside of Ireland" (that happen to be driven by temp agencies).
Before writing off LLM as simply a "stochastic parrot" or a "parlour trick" remember it can't read your mind, not yet anyway.
So what happen is a prompt said for example, find freelance opportunities in Ireland but keep in mind some of these might be available via temp agencies in London.
If you offer me not freelance but permanent roles, and not in Ireland in London...that is a logic failure.
Its this type of complexity with the normal world, that these SOTA constructions so badly fail at, and so spectacularly fail at the margins... despite maxing all benchmarks...Parlour trick.
( I was trying to quote a single sentence and then realised it ran on for the whole paragraph. )
Given how difficult I found that to follow, are you sure your prompt is actually "Clear, not too long not too short"? We now only have your word for it. I too had assumed that was a prompt given to an LLM to further prompt agents.
It sounds like they're hitting a data source quality issue, which is hardly uncommon in scraping.
It's common for job boards to obscure who the real clients are, and if the scraping engine is LLM powered ( rather than LLM written ), then I would expect it to accidentally present agencies as the contracting organisation sometimes.
Breaking down the process so you can inspect the messy middle of a data pipeline is an important part of software engineering, but it sounds like they've tossed a messy task at an LLM and expected it to be proficient end-to-end.
"General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks" https://arxiv.org/abs/2604.11778
"...General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks..."
Question: Strangers A, B, C, D, and E line up from youngest on the left to oldest on the right. Their clothing
colors and shoe colors all differ, and they come from five different regions.
Known facts:
1. A is from Morocco.
2. D is five years older than B.
3. E is older than A.
4. C stands next to D.
5. A stands next to B.
6. The person in teal shoes is not adjacent to the person from Vanuatu.
7. One twelve-year-old wears yellow shoes.
8. The person in orange shoes wears white clothing.
9. The person in blue clothing is from Chile.
10. The youngest person wears red shoes.
11. Counting from the right, the fourth person comes from South Africa.
12. E wears yellow clothing.
13. The person in green shoes does not wear multicolored clothing.
14. Two people are twelve years old, ordered by birth month.
15. One adult is thirty-five years old, and that age is sixteen less than the combined ages of the other four.
If you multiply every possible age C might have, what number do you obtain?
Nowhere is "multicoloured" defined, are we to assume it should be treated as a colour and implied that someone else must be wearing "multicoloured clothing"? Because strictly that doesn't logically follow, and it ought to be phrased as "The person in green shoes is not the person wearing multicoloured clothing" if that is the case.
This is an extremely hard logic puzzle, especially since it's revealed at the end that there are multiple solutions.
I'd expect anyone to struggle unless armed with prolog.
It means that it is possible for someone to be wearing a white shirt and yellow pants (say), but the person in green shoes came from the set of a Wes Anderson film.
white, blue, yellow, and is otherwise undefined.
But we know from [1], [2], [4], [5] and [11], that the order must be:
A, B, C, D, E or A, B, E, D, C.
Which makes C either the older 12 year old or the 35 year old.
The key this is that D can't be a 12 year old without A being in slot 2, but A can't be in slot 2 because slot 2 is South Africa and A is Morocco.
Trying to reach the shoes + Vanuatu clue is a complete waste of time, paying any attention to shoes or clothing is a waste of time, it feels like there ought to be a way to narrow it down to one of those two configurations, but the clothing is too ambiguous, the shoes end up irrelevant.
What a frustrating puzzle, where half the clues are seemingly redundant.
I guess they were not optimising for a satisfying solving experience! (I'm not sure what 'expansion' means in this context, but it sounds like maybe the 'expansion strategy' referred to in the paper involved padding out problems with red herring premises?)
I particularly found the note about “local collapse” helpful (near the end of section 3). The idea is that even though benchmarks contain a wide variety of different reasoning tasks, each individual problem requires only a few skills - unlike this benchmark where they deliberately construct tasks that span many categories.
If a human misunderstood this, they'd be a dumb human.
"Frontier LLMs Still Struggle with Simple Reasoning Tasks" https://arxiv.org/abs/2507.07313
"General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks" https://arxiv.org/abs/2604.11778
"...General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks..."
You're clearly operating in bad faith, but just for the record: the General365 problems are very difficult as you can see from the examples at https://arxiv.org/html/2604.11778v1#A1. It's actually impressive that Gemini 3 Pro got 62%, and the strongest OpenAI and Anthropic models they tried were GPT-5.1 and Sonnet 4.5.
It seems to me that such simplified tasks tend work better. The rest of the loop is just scraping websites, which doesn’t really have a reason to rely on ai agents.
If AI is really that complicated, it sounds like it would be easier to write an ordinary computer program to aggregate job boards.
When I say "write an ordinary computer program". I mean just writing the program, in a programming language. Your comment expresses disagreement with me, so should have started with "No", not "Yes".
You have converted a fuzzy task into a conventional software engineering problem, and then relying on conventional software for the reliability :-)
What can I say? This is how I get results from AI. It also gives me context & tools to fight the AI when I have to.
Most of these are 2026....
Frontier LLMs Still Struggle with Simple Reasoning Tasks - https://arxiv.org/abs/2507.07313
General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks - https://arxiv.org/abs/2604.11778
LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening - https://arxiv.org/abs/2605.19597
LogicGraph: Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification - https://arxiv.org/abs/2602.21044
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models - https://arxiv.org/abs/2607.08317
Vision-Language Models Lag Human Performance on Physical Dynamics and Intent Reasoning - https://arxiv.org/abs/2601.01547
Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects? - https://arxiv.org/abs/2605.20448
The Reversal Curse: LLMs Trained on “A is B” Fail to Learn “B is A” - https://arxiv.org/abs/2309.12288
Large Language Model Reasoning Failures - https://arxiv.org/abs/2602.06176
This comes across, frankly, as either Dunning-Kruger (classic illusory superiority), or potentially as mental illness. The slop dump is highly reminiscent of how a schizophrenic friend of mine communicates.
Do you really think slopping down a bunch of random arxiv links "proves" that AI is a scam and you're so smart and everyone else isn't?
Most awkwardly for your arxiv slop -- most of this is irrelevant to your central claim, and you've missed papers that are much closer.
For example your LogicGraph paper: "Can't exhaustively enumerate all minimal proofs" is not "can't distinguish Ireland from London".
Or your "Do VLMs Understand 3D Scenes..." is nothing more than citation decoration, completely irrelevant to our discussion.
Or your "Frontier LLMs Still Struggle with Simple Reasoning Tasks" which is potentially your pièce de résistance, it supports brittle multi-step constraint handling, but isn't remotely an eval of a modern web-search agent.
For example, VibeSearchBench would have been far more relevant to your claims https://arxiv.org/html/2605.27882v1 (but still obviously not proof that AI is "a parlour trick")
Going further: my point that we need to discuss your beginner's approach to the harness is substantiated clearly here: https://arxiv.org/html/2605.23950v1
Finally, failure to exhibit human-like generality is not evidence of absence of intelligence. It is evidence that whatever cognitive machinery LLMs possess has a very different error distribution from ours. Your General365, LLMEval-Logic and the Reversal Curse are actually fascinating evidence for that jaggedness, rather than proof of your claim that AI is a scam.
Come on, that's being incredibly pedantic. If I say, your comment reminds me of someone being an asshole, I'm just using a circumlocution to call you an asshole. No?