upvote

points

by bob10295 hours ago |
comments
upvoteby sowhat15 hours ago|
next
[-]
Not my experience at all. I pointed frontier models at some PyTorch code, told it to make it faster, and it optimized it 10x fold.

You’re probably limiting the frontier models by being specific.

reply
upvoteby vladms5 hours ago|
parent|
next
[-]
I am sure people could find code such that an LLM can optimize it 10x fold. I have seen many types of code in my life (well written, horribly written, fast, slow, etc.).

I do feel there is a limit to LLM-s today. I did not try, but I doubt it will work if I would tell it "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".

Knowing where that limit is, is as hard as it always was knowing how fast a team of engineers was going to do a project.

reply
upvoteby latentsea4 hours ago|
parent|
next
[-]
> I do feel there is a limit to LLM-s today.

I felt there were limits 4 years ago. I couldn't get even an 8k context window.

It's starting to feel like the important gaps are the only ones left that need to be closed before there aren't any left. It also feels like next year they will start meaningfully closing.

reply
upvoteby throwuxiytayq4 hours ago|
parent|
prev|
[-]
> I did not try, but I doubt it will work if I would tell it "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".

I do feel there is a limit to human-s today. I did not try, but I doubt it would work if I asked literally any programmer I know "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".

Hell, I'm willing to bet they'd fail at this task even with an unconstrained snacks and kombucha budget. Humans have a long way to go. My job is safe.

reply
upvoteby progval5 hours ago|
parent|
prev|
next
[-]
Not every optimisation is the same. LLMs can optimize code that is behind known practices, but won't usually invent new data structures or algorithms to solve a unique problem.
reply
upvoteby Zambyte5 hours ago|
parent|
[-]
While it's unlikely for LLMs to implement patterns not present in their training, current training techniques absolutely can produce practices and data structures and algorithms that humans have never done before. Modern LLMs are not just trained on text. They are trained on computer use, including - at a high level - solving programming problems with code. They are not just rewarded for producing the most likely next token. They are rewarded for producing tokens that solve problems. Patterns and techniques can emerge from this that are not present in any human behavior.
reply
upvoteby lifeisloving4 hours ago|
parent|
[-]
They are rewarded for code compiling, thats about it. Thats why the code is terrible. Ive yet to look at a project made by an llm whos user swore this "frontier model is the one", where the code wasnt utter garbage. The heavy llm users dont care as long as it compiles.
reply
upvoteby lifeisloving4 hours ago|
parent|
prev|
next
[-]
This doesnt even apply to his issue. What does PyTorch code have to do with complex concurrency or complex systems in general.

Theres lots of bad Pytorch code, just go ask george hotz. This isnt the magic you think it is. And nobody wants to hire the guy that just points llms at things and says make it faster. Things have value because talented humans make them. Its why a luxury coat is worth more than the linens that make it, or one from walmart made by a machine. This will 1000% apply to programmers. I think OP will be fine.

reply
upvoteby mjr005 hours ago|
parent|
prev|
next
[-]
The fact that you're pointing it to PyTorch code in the first place is being specific.

Pointing a frontier model to a 10 or 100MLoC codebase and saying "make this code fast, make no mistakes" doesn't work. As an experiment I recently tried this with a relative small (500KLoC) codebase and it got stuck on believing that the primary cause of slowdown was the database not using a connection pool. (Which was completely irrelevant for this specific code.)

reply
upvoteby xpct4 hours ago|
parent|
prev|
next
[-]
With how you presented this, it makes it sound like it was either a toy codebase or something outside of your domain already.
reply
upvoteby 3 hours ago|
parent|
prev|
next
[-]
deleted
reply
upvoteby intended5 hours ago|
parent|
prev|
next
[-]
The example provided covered both options.

In general the OP is right, the people who get the most value out of LLMs are veterans.

reply
upvoteby bob10295 hours ago|
parent|
prev|
[-]
How large is your codebase?
reply
upvoteby AlienRobot5 hours ago|
prev|
[-]
I don't know why but in Typescript when I tell an LLM to write code to check if a character is of a type, it often uses a whole Set() instead of just using a string and doing indexof. It's weird because not only should strings be faster, doing it with strings should also be the prevalent way in the dataset. Yet it uses a Set.
reply