upvote

points

by tosh4 hours ago |
comments
upvoteby bisonbear3 hours ago|
next
[-]
This is the right direction (Anthropic removing 80% of the system prompt https://claude.com/blog/the-new-rules-of-context-engineering...)

However what I wonder how we make these decisions on what to remove. Certainly some subset of the skills / AGENTS.md / prompt / tools do actually help agent performance, the problem is in identifying those, throwing away the rest, and validating that this didn't actually hurt the agent performance

reply
upvoteby radlad4 hours ago|
prev|
[-]
'better results' in terms of what though? A benchmark, or code that I would actually click "approve" on in a pull request scenario?
reply
upvoteby tosh4 hours ago|
parent|
[-]
apologies, I should have clarified the 'better' claim

  - same task result (passed)
  - finished faster
  - fewer tokens, less cost
  - fewer requests for inference
  - fewer tool calls
  - less peak RAM
reply
upvoteby radlad3 hours ago|
parent|
[-]
Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable?

As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible.

My CLAUDE.md, memories, and skills are all about either (a) adherence to project standards and guidelines, (b) product decisions which impact future code, and (c) instructions on how to prototype and work in my environment.

Removing these instructions would mean more turns with the AI to get the desired result.

reply
upvoteby tosh3 hours ago|
parent|
[-]
in this case there was a hidden grader that checked if the implementation was correct (because that was the easiest thing to check), all 3 agents cleared this hurdle in all 9 runs

I agree, next it makes sense to try more open ended tasks + have humans (and/or multiple models) grade the runs and their results

reply