Does the human find enough bugs that they stay on guard, or just rubber stamp everything without really looking at it? It’s hard to stay vigilant when stuff looks plausible.
replyComplete speculation: you could instruct Claude to hide one random mistake in every document.
replyAnd Claude should write down the mistake in a sealed envelope, so it doesn't make into the database.
A review that doesn't find the mistake counts as invalid.
This is what bag scanners at airports do - the hit rate is so low and the job so boring the software projects fake contraband onto the imagery. Fail to spot the knuckledusters and expect a chat with the manager.
replyDon't forget that humans have a not insignificant error rate when copy/pasting or copy/typing data.
replyAnd it's possible to run each document through the LLM pipeline multiple times, using different models and/or prompts each time, to check for errors and inconsistencies. That will take more time and cost more, but it can reduce the error and hallucination rate significantly.
replyI'm doing some public court records processing for bankruptcy cases (interested mostly to seek out corruption in big national cases), and yes, the "variousness" of random PDFs is exactly the issue. Trying to get the cost for a whole case down to a minimum.
replySample is around 300 court dates, shy under 1k files.
At best I'm building a claude skills file.
> Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
replyA better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.
Nobody working there has the requisite experience to do this in a reasonable amount of time. These are not particularly tech-savvy folks, Claude use aside.
replyYes. And it might not even be worth it, as the AI agents gets cheaper and cheaper.
replyKeep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.