But people are now individually acting this way on their desks on an hour by hour basis. LLMs make cargo-culting inevitable because they are inscrutable and opaque.

There is always this sense in the LLM-proponent world that LLMs are at any moment as bad as they are ever going to be; line goes up.

But it seems clear that the gap between perceived and measurable productivity is still likely spent in poking entrails with a stick.

We are so used to probabilistic tools that have significant setup time before they become valuable and save us loads of time that we're at risk of repeatedly writing off that setup time without seeing the rewards, believing that one day it will actually work out that way.

(Which is most recognisable from the early JS frontend frameworks era.)

Meantime here we have an article that shows that a thing (longer context windows) that people thought would functionally solve a problem so we would get the value from all that setup does not, in fact, very meaningfully kick it down the road, and the comments are still full of entrails-and-stick work.

I’m working on a tiny agent harness at home to learn and the process of taking human speech and turning it into agent tool calls that output something generally deterministic depending on how the tool is defined is so interesting.

One of the big takeaways is you really only have to rely on the non-determinism<->determinism translation layer once when you switch between the two domains. You can obviously rely on it more if you want, and that’s probably faster because determinism is hard, but you don’t need too do that.

Can't help but feel like a lot of people who are deep in IT made it there because they hated working with humans.

Have you ever tried doing evals on moderately complex but bounded tasks?

I spent some time doing it when testing these "token reducing" tools like Headroom, RTK etc. as well as customizing my Pi tools. What I found interesting was that despite LLMs being deterministic, for a given toolset and prompt, the results were highly consistent for a given eval, across multiple models (I tested at the time using GPT 5.4 mini, 5.5, 5.3 codex, Gemini 3 flash, initially running sets of 5 evals on each task but once I realized how consistent the results were, dropping to sets of 3.

Aside: in my tests, RTK and Headroom made the overall context use higher for roughly equivalent results. The context use for those specific toolcalls went down but the number of model turns and overall context use went up.

The LLM providers fine tune the models with some kind of information retrieval tasks, but to do so you must provide some non relevant context to bootstrap the session for the long context tasks.

It would be very easy to do this in ways that train the sequence model to treat early history as noisier than it really is, or to weaken its relationship to late context.

You’re also probably stacking more contexts together with long contexts (start with task A, then detour to solving B and C before you can complete A).

Training sequence lengths probably decay super linearly with length creating far fewer samples at long length during training.

I don't think OP is claiming that prior to LLM coding everything in the software development world was super rigorous (I assume that's effectively what you mean with the "rose-tinted glasses" comment). But rigor was actually possible and in a deterministic way too, which is fundamentally impossible with LLMs. You can build all kinds of guardrails and processes around LLMs that make it somewhat approach rigor again, but it's still fundamentally based on a bunch of statistical probabilities instead of deterministic, repeatable results.

All of the methods I see to mitigate the fundamental and inherent issues of LLMs seem roughly equivalent to the kind of crap you see in astrology groups or palm reading etc. You need Venus and Mercury to be in alignment while Mars is retrograde if you want to be able to get the right results from your token predictor.

But those technologies are layers, and there are reliable things that sometimes bubble across the boundaries — type hints, better code patterns to trigger compiler optimisation, interesting tricks with key column selection — and someone with expertise from that layer below can explain why, and their advice will always work in situations that are sufficiently similar.

You are right about AI personalities. Obvious even with the open weights models. Gemma and Qwen write code and documentation like people from different cultures. Because I guess they are a bit like that.

Because that's what OP is talking about. Superstition presented as factual advice instead of the technically rigorous and scientific fact that already exists.

You're being downvoted because you don't understand this fact, or indeed understand what you're saying at all.

I'll spell it out for you: technically and scientifically rigorous facts do actually exist, even in regards to LLMs. We can, in fact, obtain scientific and objective facts about how LLMs perform. It can be rigorously proven that certain context habits affect certain tasks positively or negatively. Your argument is that none of this matters more than superstition. And you're surprised that arguing to a room full of engineers and scientists that science is dead and superstition is the one true way forward gives you negative response.

I usually don't have to worry about compiler optimisations because compiler experts do that; sometimes they appear in a thread about code and say "compiler guy here — if you write your code like this the compiler can optimise it".

And that person will be provably right (or wrong), in that context. And it'll be the same each time you run the test!

I just… ehh. You make a good point and I worry you are not wrong. It's all so different.

I like my 3D printing analogy much more than I wish I did.