However, I have it on for ChatGPT (use it mostly for on the go queries via mobile) and their memory experience is significantly better. I haven't enabled Claude's memory in a long time so things may have changed though.
I often see tool call failures because the agent invokes the tool with wrong parameter names, e.g. for file creation it provides "file_content", but the tool expects "file_text" instead. My colleague told me to put this in a pre-prompt file, but I don't understand why the models make these kinds of mistakes - the model should have the available tools with their parameter names in context, should it not?
It would be useful if they could scope them properly. Then it would be useful to keep around. But I haven’t seen memory systems be able to do that.
How did you determine this?
Do you mean that you were giving them tasks that were too small or you were telling them to split the tasks up or something?
When analyzing my session logs, I tried to look at how story size relates to how much work the AI had to do to fully implement it (made easier by my sessions always having epic#.story# - like 2.6 - in their name). Neither "story size" nor "AI work" have fully obvious metrics, so I looked at a variety of things, like length of story plan, number of lines of codes written for the story, number of different files read/written (and for AI work: number of tool calls, number of tokens for main thread / implementer / reviewer subagents, etc.).
The pattern is always pretty much the same: smaller stories are less token-efficient. One major factor for this is that in bigger stories, the average number of lines of code written per tool call is significantly higher (and not just a little bit, if I remember correctly it's a factor of 1 to 3 between my smallest and largest stories).
None of this is scientifically rigorous, but it's consistent over a fairly significant sample (about 60 stories up to this point). I'm also only measuring amount of work and not quality, but when I find things wrong with a story, I generally ask the AI to fix the issues in the same session, so the two should be correlated (sometimes the issues are large enough that they require new, separate stories to fix, but this is normally caused by problems in ideation and/or planning, not implementation). This is enough to encourages me to ask for more rather than less per story. It might also not apply to everyone, I'm sure this is dependent on everyone's workflow and way to work with the AI. But yeah, that's the sort of stuff you can explore when analyzing your session logs.
I also never use memories. I don't see the usefulness above project-specific prompting and a good knowledge of your own codebase. Am I missing something?
The thing you're asking it for might be intent, which is something the technology fundamentally lacks.
Which.. is something that has been discussed very extensively, which makes one wonder why Anthropic still keeps trying to push that square peg through a round hole.
And in the Claude Desktop UI I manually curate the memories.
Works well enough, I don’t like the concept of dreaming as it quickly leads to memory bloat and/or outdated stuff.
So the main improvement to memories I'd like to see is better judgment on which ones to create.
I started with [proactive post-hoc analysis], and ended up following some threads to interesting insights.
I partly say this because I also finally pulled the trigger on deprecating the last ClaudeCode memories on my system in favor of a personal system[1] recently, and the issues I noticed were things a casual user/'vibecoder' might appreciate, but any engineer working fulltime on multiple projects via multiple harnesses would find counterintuitive. E.g.:
- The memories were hidden away and clearly not written for human eyes; useful for low-stakes ledgers, frustrating when the right kind of errant note can derail a longrun 2 weeks later.
- The memories seemed to err towards the side of remembering everything and recording incidental details; useful for remembering the user is gluten-free, but not for tracking spec changes over time.
- The memories could be specific to a subagent or the project or the whole system; an obvious & elegant system for toy examples, but a terribly unweildy one for anyone working at scale over the span of months.
[1]: if curious, a somewhat-outdated version of it is here: https://gitlab.com/doering-ai/libs/fleet TL;DR: Friendship ended with ClaudeCode memory files; my new best friends are `BULLETIN.md` for transitory alerts, `campaigns/[slug]/LEDGER.md` for basic episodic memory, and `reports/[date]-[slug].md` for writeups polished specifically for posterity.