upvote
> I work with domain experts often and it’s usually the most difficult part of the process. Now, I can ask an LLM the questions

And you will not notice the obvious errors in its answers to even basic questions, because you are no expert and should have really consulted with one.

When talking about their own area of expertise, almost everybody immediately notices the glaring faults. But then they are very impressed and take LLMs as gospel when talking about anything else. I do not get it.

reply
How would you know the agent is correct in domain expertise if you’re not possessing domain expertise yourself?
reply
Shallow generic domain knowledge is not really a helpful domain expertise, is it.

Lets take banks - knowing how banks work in general means little for that one specific bank who is paying you, which has different portfolio than most, different processes because everything is homebrew over decades of doing business and they differentiate from rest of the market in many ways and thats their key proposition, different data infra, set of internal apps, protocols and so on. There is no general llm knowledge you can distill in few minutes that would help you much.

You need to know your specific company, no way around it. In fact, to be proficient in those deliveries, you need to know those internal politics, processes, their quirks and so on, 100x more than some code fast delivery since this is majority of any bigger project. You need some reputation to get things done effectively.

This, for some time, won't be something llm can massively improve, unless those companies change themselves.

reply
I'm in the position of being shallow in the domain I code for--and I find it a very big help. Knowing what the real world implications of what the numbers become is a great help in understanding what truly is being requested. Just ran into that not too long ago--my code suddenly started slamming a side-cutting bit into the wood. Look into it....years ago there was a bug upstream of me that slammed the bit into the wood. Simple fix: spot the offending pattern, replace it with something sane. The upstream bug was "fixed" (not fully), my simple (lack of true knowledge of what was happening) fix proceeded to undo their fix. Then it got rewritten the way it should have been, a more general recognition of unacceptable movements and how to fix them.
reply
>> You need to know your specific company, no way around it

Of course there are ways around it. For example you can build the foundation yourself (which will cover the 80% that is shared across every company in that industry) and give users the ability to build the rest themselves inside the software. This type of abstraction-based architecture is how platforms like Salesforce took off and became ubiquitous.

reply