I wonder if a productive avenue might be "doing aspect-oriented programming in reverse": asking the LLM not to "summarize" the code per se, but rather to "clarify" it by transforming it into what a programming blog post would call a "toy example" of what the code is doing, by stripping out all the (non-semantic) error-handling, logging, metrics incrementing, etc — all the things that you might treat as their own "aspects" under AOP.

AI summaries are great for learning which parts of the codebase are load bearing.

I've found that I don't mind getting a lot of text back from an LLM, when I was the one who prompted it. I can easily enough let my eyes flit around in the text and figure out what I need to, and I expect that putting all that text in the context window will help with the rest of the conversation.

It's when that text gets copied and pasted into a blog, or a PR, etc. that it really galls.

I don't like the default explanations. But I prompt for small code snippets with explanations of a problem and the solution. This is in the context of extending features, fixing bugs, reviewing new code etc. I still skim the code, but it's nice to have a somewhat thoughtful overview of the key points like database schema, API spec, algorithm or abstraction. It makes it easier to skim a large diff without feeling lost. It also quite often catches some weird choices that might slip through it not carefully reasoning about the code. For me it is hard to understand code I didn't write myself so I have landed on this workflow.

Yeah, I have a rule in my agents MD that essentially says the code doesn't need a comment because it should be readable. And ask me in the exceptionally rare instances where it needs a summary or a comment. I am a Ruby and JavaScript developer, so I suppose that is a lot more readable than some other languages, but broadly, I've found code summaries and code comments to not be necessary if the code is very elegantly written.

The "language model" aspect shows IMO, at least for someone who grew up with the material of 90s and 2000s where we describe things more in mechanical, engineering terms. When gpt4 landed it was somehow amazing to see the output, but yeah nowadays I keep skimming through the explanation. It's like an intermediate dev who has nothing else to do but create long sentences to describe what could be simpler.

Right, the value is not so clear of having multiple people wade through lengthy LLM responses. Part of the value that a human in the loop adds is figuring out what's correct and important in the response and passing on a more polished result.

There is a reason Caveman (https://github.com/JuliusBrussee/caveman) is as popular as it is

I wish LLM summary of code was put aside the actual code, like the Jasmine documentation[0].

[0]: https://jasmine.github.io/tutorials/your_first_suite

I'm a bit confused by your use of 'pithy, but at least its terse'. Don't pithy and terse mostly mean similar things? Here it seems like you're contrasting them. Not trying to be pedantic, apologies if it comes off that way.

I use AI explainers a lot in my own projects and have skills that give outputs I like. I'll even pipe the output into NotebookLM to make a video sometimes. This is actually a decent way to grok complex subsystems (YMMV).

What I don't care for is the pretty-yet-verbose HTML outputs that I see from some teams. They can be way too long and it's often the case that the engineer who made it didn't read the whole thing.

We need a new specification language for summaries. I think just a (pseudo?-)programming language’s API (modules, interfaces, method signatures), but more concise and descriptive than Java, maybe Haskell or OCaml minus quirks (…the academics were this close to an ideal syntax, maybe they should’ve gave up those languages’ LR parsers…)

Your AI summary is raw output, like uncooked mince meat.

Ask for a rewriting pass! Get it to edit for word count, remove redundancies, remove run-on sentences, and change sentence structure to use active voice.

You wouldn't ship your first draft - why let AI do it?

Ask for a walkthrough of the highlights (with '...' omissions so that the most important parts are quickly visible) instead of a verbal summary.

I use caveman to handle this. Not because to I want to spent less tokens, I want to spent less time reading.

I hate fighting with the AI to get a good summary. Reading the good summary is great. Getting to it is the challenge.

First attempt: mountains of meta conversation and not answering the original question.

Second attempt: shorter and more concise, cuts out some of the nonsense, but terribly written.

Third attempt: goes off the rails, misunderstands what you are asking and tries pushing the work onto you.

Fourth attempt: finally something that's concise, reasonably well written, to the point and passable.

It’s an LLM, it does what you want. If you don’t like it verbose tell it you want it to be succinct and it’ll do exactly what you ask of it. What sort of complaint is this?

Yes this is why I ask the agent to summarize the code as a comic of a lizard explaining it to a cat.

[dead]

You're absolutely right!