That's a factor of 3 to 5 improvement reported on here. If you keep stacking these kinds of improvements, you end up at month to minutes pretty quickly.
I thought, something like a medication helping the women develop the baby in 30 days, but seriously damages the host in a couple of days, so the fetus gets transferred from host to host. Some die, but the evil rich don't care. Your idea could also work though.
I realize that this is totally the wrong corner of the internet to post this but I guess today I'll just go with it.
At my company we're also in the process of deploying a system that does exactly that. And what's interesting is that you absolutely do not need a frontier model for that, a small VLM (vision-language model), with optionally a little bit of fine-tuning, gives you the same output quality at a fraction of the latency and cost.
It's also not really clear if everyone is going to want a frontier model when the real implications sink in. Maybe we'll get sick of incomprehensible code optimizations and people wile tire of reading AI prose that feels ever-more-human. There might be a few use cases, but who is going to pay for this when providers start charging enough to be profitable.
But that's phase two, phase one is finding localized problems to solve using LLMs and productize them. I'm reminded of cloud tech, where phase one was changing software to run in the cloud, and phase two was optimizing costs.
But between 10 and 20 seconds to analyze a document with only a bunch of pages just isn't good UX.
The bottom line, though, is that PDF OCR is usually regarded as a solved problem. LLMs won't usually do the recognition itself. It will farm it out to established tools which are very good.
So in many cases the visual way is the only one that works correctly, the textual one is just a shortcut that may be walkable in certain cases.
Disclaimer: I used to work in legaltech, but not those two companies.
You’ve accurately stated that AI isn’t as rigorous as a trained attorney. Doesn’t that mean that every single datapoint must be confirmed by a human?
How is that quicker than just using a human to read the content and make the call? Data entry savings?
It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).
Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.
Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
And Claude should write down the mistake in a sealed envelope, so it doesn't make into the database.
A review that doesn't find the mistake counts as invalid.
Sample is around 300 court dates, shy under 1k files.
At best I'm building a claude skills file.
A better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.
Keep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.
"Write a python script that breaks down this PDF by X feature" would not hallucinate anything in the PDF. Certainly you could trivially double check that all text in the extracted JSON was in the text layer of the PDF.
LLMs change not a whit, and there's no one to take responsibility for the failure (and thus no way to fix it).
As the new variation on the old theme has it, "A computer can never be held accountable, and so very many people are trying to get them make management decisions."
For instance, if you can say:
"Refer to the database schema in x.sql as your source of truth for the database structure we want to import into*. Do not invent data, tables or columns that do not exist. Carefully match all output against this database schema and do not create output that doesn't exist if it does not match the schema, simply skip it."
You will end up with a far better result in my experience.
Gotta treat it like a child.
"Sorry for that, Your Honour, but we gave that case prep work to a child."
But now it's comparing already filled columns on a spreadsheet, not copy-pasting every single thing from an (often uncopyable) PDF.
... with a PDF, right?
> not copy-pasting every single thing from an (often uncopyable) PDF
Obviously the PDF is copyable, else your bot would not be copying it.
Whether that is useful measurement I suppose depends on the circumstances.
I see problems in LLMs doing research, making drafts, etc.
I see 3 kinds of potential users.
1. Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
2. Fresh out of law school graduates.
They think they can do fast work, draft quick replies, do fast case research, enh. They should go the traditional route of interning, learning the "art", they probably won't need it because they will learn "how the law works"
3. Established attorneys. They usually have templates made for most things, over their career, they have established routines of making things fast, maybe it can help them but to say replacing good chunk of work or offloading to LLMs isn't probably going to go well.
There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....
I used LLMs to replace a lawyer this year to great effect. I successfully advised myself as to the right strategy and drafted an immigration petition in the correct language. Granted it wasn't 100% LLMs (I also supplemented it with "traditional" online research and wrote the letter myself using the LLM version as rough guidance), but still, I certainly would've gone to a real lawyer in 2024 for this stuff but didn't.
Professional advisory work can be divided into two types:
1. Once you know the rules of the game, the next move is immediately obvious.
2. Knowing the rules is not enough to know the next move -- that requires judgement and experience.
LLMs in the hands of someone smart and savvy can easily replace the first type of work.
Lawyers nearly universally tend to: convince clients they need a lawyer; promote extreme views; charge legal fees (that ultimately result in financial damages to the client). There are no winners in this game, except for lawyers, who are having a lovely time. Even if you won the case, you've lost, because the pie got smaller due to lawyer fees.
"You can't trust the output, it doesn't understand bigger systems"
"Its an art, you need to learn the ropes of it to truly write good code"
Its a very dangerous line of thinking. Software engineering will never be the same, as writing code has basically vanished from the daily workflow. Not for every specialized usecase, but for many.
In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you. I think this is a valid use case to get at least some understanding what the next steps might be and if it even makes sense to seek legal representation.
There was a post here a few weeks ago about a UK court being inundated with some kind of claims (unfair dismissal?). To submit a claim you do not need a lawyer, but the process is (intentionally?) vague and very complicated, so it takes a lot of effort to figure it out yourself. But AI makes that kind of stuff easy.
AI is not going to replace layers billing big corporate clients millions a month, but it will replace things that are fairly standardized processes, that don't require a lawyer, but today are just too complicated for regular people.
I'm not sure there's quite accurate. The form is really not that difficult to complete.
It's basically: Party details, type of claim, particulars, damages
Obviously knowing these details requires some legal knowledge but I actually think ET claims are easy enough to bring if you've got the time to do a little research and build your case. Almost all ET judgments are public so there isn't the usual difficulty with research as you get with law reports being restricted either.
https://assets.publishing.service.gov.uk/media/65bcbd214a666...
On the margin, AI will drop these guys' billable hours.
Really? I find them quite talkative. But I'm not sure what kinds of questions you are pestering them with?
In any case, the whole discussion reminds me of people using LLM to replace medical professionals.
For both: unless you have a doctor or a lawyer always on standby, you have to make a decision on whether to even go and seek out one of these professionals. By definition, you make that decisions without professional help. I think LLMs can help you there with the initial research to decide whether it's even worth it to contact the professionals.
If there is no precedence for X happening when Y , Z, P, Q, R are special combinations , the judge decides and sets precedence. But if another judge interprets it differently with similar set of circumstances it is not considered incorrect I presume especially if in different jurisdiction unless overridden by a higher court. This is my layman understanding but I think this design is wrong and essentially where human subjectivity controls the system.
Subjectivity is not totally avoidable in law but the goal of the legal system should be reduce it as much as possible along with ambiguity.
Law is intentionally not code. The world is not something you can program.
Since this is a thread about using ChatGPT for law, I should note that this is actually one of the most important reasons you'd rather hire a lawyer than ChatGPT to represent you. A good lawyer or firm will know the local judges and how they tend to lean on ambiguities in their area of expertise, and will be able to use that information to both advise you and tailor their arguments to appeal to the judge's sensibilities. There is an element of unfairness to this, to be sure, but you'd find much more unfairness really quickly with machine-generated judgments because of how messy and non-compressable real world circumstances are.
That might be true, but that doesn't mean you benefit from training up juniors.
These situations usually are not ones that an individual can justify the time or money to contact an actual lawyer, but then if they do decide to contact one they will come in with better questions and more of a sense of what they are expecting.
This is similar to medical. Should you use LLM to diagnose yourself, treat yourself with prescription drugs you buy from shady gray market online sellers? No. But you can use it very well to know when it's time to go to the doctor and what to ask.
LLMs can help with that. I don't think they are worse at this than me trying to figure this out all by myself.
3 kinds of users that don't scare the shit out of you.
My buddy works with lots of folks that have gone all in on Harvey. There are other kids of users.
That very first image uses a different inflation indices for each of the two lines in the graph.
They also only post pictures without linking to sources.
You havent given any proofs or even comments that the work is the same level of quality or accuracy.
The statements in the post are opinions, there is no actual PROOF they are true, and thats my point.