upvote

points

by LeifCarrotson3 hours ago |
comments
upvoteby RussianCow3 hours ago|
next
[-]
Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.
reply
upvoteby ngcc_hk26 minutes ago|
parent|
[-]
In 1980s ibm has studied and said why sub-second response needed to maintain the mental flow. That time you send a whole screen unlike unix like character by character. This proves very true even when you deal with form processing. I think that we are dealing with the same issue here.

Keep your mental context in your brain is critical

reply
upvoteby Aurornis2 hours ago|
prev|
next
[-]
> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete.

If I spend 10 minutes reading an article, that would only generate 3000 tokens.

That’s not counting the prompt processing time.

We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.

> Yes, with top-tier GPU farms you can hit hundreds of tokens per second

My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.

> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.

If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.

I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.

reply
upvoteby zoobab2 hours ago|
parent|
[-]
You could suspend it to ram, and only wake it up on request, it takes 2 seconds on my box.
reply
upvoteby Aurornis1 hours ago|
parent|
[-]
It’s not a cost savings relative to paying API prices even if you’re suspending it.

This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low.

It’s not going to be cheaper than paying API prices for the model though.

reply
upvoteby allknowingfrog3 hours ago|
prev|
next
[-]
We clearly have different goals. I want an LLM to review my code, not the other way around.
reply
upvoteby pmarreck17 minutes ago|
parent|
next
[-]
I'm sure this exact topic has been argued hundreds of times already on HN, but I think I have a new "possibly agreeable to both sides" perspective on this after having lost man-years to retired corporate code aka "FAIAP, throwaway code"

Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.

Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway.

But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.

reply
upvoteby nolok3 hours ago|
parent|
prev|
[-]
It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back
reply
upvoteby Aurornis34 minutes ago|
parent|
next
[-]
> another task a lunch break whatever and it will be done when you're back

At 5 tokens per second and unknown prompt processing speed, you may need a very extra long lunch break depending on your codebase.

reply
upvoteby allknowingfrog2 hours ago|
parent|
prev|
[-]
How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.
reply
upvoteby vitally36431 hours ago|
parent|
next
[-]
This is like comparing a hammer to a screwdriver and feeling smug because you can hammer nails faster than someone else can drive screws.

These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.

reply
upvoteby 2 hours ago|
parent|
prev|
[-]
deleted
reply
upvoteby adastra223 hours ago|
prev|
next
[-]
We aren’t there yet. Not for frontier development work at least.
reply
upvoteby bredren1 hours ago|
prev|
next
[-]
Except often queued agentic flows must be checked in on. Or to use the comparison, 3D printers are not immune to making spaghetti all night when something goes wrong. (I’m not a 3d printing expert so maybe that is solved now)

It is common for agents to just stop because overload or some API error hijinks.

Or you get a TUI question that is blocking.

In general you’re right though, staring at tokens from agentic is not time well spent.

Some of these I’ve built custom harness around in iterm2 though.

reply
upvoteby bitpush3 hours ago|
prev|
next
[-]
is there a good tool to manage these workloads? batch process a bunch, handle failures, retry things etc?
reply
upvoteby varispeed1 hours ago|
prev|
next
[-]
Filament snaps at 1am and then you have to run print again. 10 hours turn into many days potentially.

I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.

reply
upvoteby ctoth3 hours ago|
prev|
[-]
[dead]
reply