upvote
It could also be as simple as the technology adjacent people developing tools via LLM that somehow end up in the critical path of some business function and now it needs to be supported by someone.

I’ve seen the same happen with insane spreadsheets.

As empowering as something like Claude code can be, leaving development to developers means the output is inherently supported by the organization long term.

reply
Extending the thought, there’s also the risk it enters critical path but something upstream changes or Anthropic starts charging 1000x for the tokens and now someone needs to walk through god knows how many lines of never before reviewed code to figure out how it works.

By wholly depending on the assisted coding you are wholly dependent on Anthropic, forever, for something that is now critical.

There are probably thousands of these little microscopic dependencies that occur when you make these tools widely available. What happens when you don’t want to foot the bill anymore?

reply
You use OpenAI or Deepseek in that case.
reply
Capture is capture though. It’s not a matter of which AI provider you use. It’s a matter of losing agency over how an aspect of your business works.

Like going 100% into a single cloud provider. Or 100% into MSFT products. There’s legitimate friction in decoupling.

reply
This is what happens. We had this happen with end user scripting pre AI where these scripts became unknown production applications. It made the actual engineering impossible because you had no idea if anything you changed would impact the ??? processes run by users that have become critical. AI would amplify that greatly so we have an explicit rule that it can only be used for reporting and prototyping only and we restrict any write action
reply
I am co-teaching an AI course for college business students. We decided to use Claude Code and Visual Studio since the space is moving so quickly and we thought understanding some of the fundamentals is the most value they could get from the course.

It was the right choice. But many of the rough edges of the terminal do surface and confuse non technical people.

reply
Isn’t that a good thing? It would be irresponsible to introduce non-technical students to building software without at least covering some basics like a terminal and encouraging some understanding of the limits of their capabilities.
reply
> which leads to the actual problem I think.

Getting there faster with less knowledge of coding isn't the problem.

Pre LLMs, the main barrier to producing software for non-tech people was writing the code, literally. With LLMs, this barrier is now gone. This results in a much larger volume of slop than before. And then the real barrier shows up -- understanding, fool-proofing and maintaining that code. Code is cheap, show me how it runs in prod.

Like it's great that quants can iterate on their ideas much faster now. But GS is saying they shouldn't leave their sandbox.

reply
Elephant in the room is QA, the thing that disappeared when Agile took over the world.
reply
As much as I agree I think the leading comment is actually even more spot on. Understanding dies on the hill of having a tool build out the process for you, and not understanding, especially in a space as knowledge critical as finance/trading, is where you run into catastrophic levels of risk.
reply
> But GS is saying they shouldn't leave their sandbox.

Exactly

reply
Maybe there is an irony in that LLM with coding harness could be a step in the right direction for operational domain experts to solve business problems while maintaining high standards for legal/compliance/regulatory/reputational and prd-deployment concerns.

User scripts that turn into applications that need production support are often consequential and business-differentiating since they do the things that can't be bought off the shelf and effectively encode tacit knowledge.

reply