It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).
Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.
Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
And Claude should write down the mistake in a sealed envelope, so it doesn't make into the database.
A review that doesn't find the mistake counts as invalid.
Sample is around 300 court dates, shy under 1k files.
At best I'm building a claude skills file.
A better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.
Keep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.
"Write a python script that breaks down this PDF by X feature" would not hallucinate anything in the PDF. Certainly you could trivially double check that all text in the extracted JSON was in the text layer of the PDF.
LLMs change not a whit, and there's no one to take responsibility for the failure (and thus no way to fix it).
As the new variation on the old theme has it, "A computer can never be held accountable, and so very many people are trying to get them make management decisions."
For instance, if you can say:
"Refer to the database schema in x.sql as your source of truth for the database structure we want to import into*. Do not invent data, tables or columns that do not exist. Carefully match all output against this database schema and do not create output that doesn't exist if it does not match the schema, simply skip it."
You will end up with a far better result in my experience.
Gotta treat it like a child.
"Sorry for that, Your Honour, but we gave that case prep work to a child."
But now it's comparing already filled columns on a spreadsheet, not copy-pasting every single thing from an (often uncopyable) PDF.
... with a PDF, right?
> not copy-pasting every single thing from an (often uncopyable) PDF
Obviously the PDF is copyable, else your bot would not be copying it.
Whether that is useful measurement I suppose depends on the circumstances.