upvote

points

by alexhans5 hours ago |
comments
upvoteby stingraycharles4 hours ago|
next
[-]
People always say this about the evals, but I find it hard to have a practical implementation of such a thing where you won’t end up spending 100x the amount of time on the evals than building the skill itself.

Like, ok, I have a debugging skill, now how do I make evals except for the most trivial things?

reply
upvoteby jurgenburgen2 hours ago|
parent|
next
[-]
Why do you have a debugging skill? Just tell it to read the docs.

Skills are for packaging instructions for how to interact with your organizations homebrew process and tools. By definition skills shouldn’t be useful outside of your org because they’re just docs and third party tools already have them for humans.

reply
upvoteby RicDan3 hours ago|
parent|
prev|
next
[-]
You don't. If you're using skills to force the AI to fullfill some must criterias, it's not going to work. Must criterias need deterministic checks -> be it hooks or what not.

This is also my biggest gripe with AI. I.e. for specifications, no matter what hype machine I tried, it never fulfilled my criterias, which are: easily verifiable, concise, small specs. Hence I built https://github.com/RicardoMonteiroSimoes/Yamlet initially for claude code, but then decided to use extend it for pi.dev. I now have a dedicated docker image for pi.dev, that only contains Yamlet plugin, and whenever I work on spec I spin it up.

The end result is a .yaml file that easily works in git + git diff, so that I can then proceed with the technical specs-

reply
upvoteby TobTobXX4 hours ago|
parent|
prev|
[-]
A skill should only document behaviour the LLM didn't/couldn't exhibit on its own.

So you take your failed case (eg. working with gdb or whatever), write a skill and then test for that failed case.

reply
upvoteby hakunin4 hours ago|
parent|
[-]
There are also skills that help LLM do the thing it can do without the skill, but faster (by cutting out unnecessary discovery). I guess for such skills the fail case is "being slow"?

I imagine many fail cases can burn a lot of tokens/usage/time because failing LLMs can be very persistent. Maybe some upper bound (turn count, timeout) would help too.

reply
upvoteby resonious2 hours ago|
prev|
[-]
Big yes on this. I do not understand the appeal of skill shopping. The one exception I have is things like the Axiom Apple development skills and e.g. the official Flutter skills. At that point the skills are just docs though. It's either I remember to paste a URL to the official docs or I just install the skill. But shopping around for random skills just sounds extremely unappealing.
reply
upvoteby flurdy1 hours ago|
parent|
[-]
I do both, or rather I do 'skill browsing', for new ideas to then evaluate the skill with my agent if they are useful. Most are not, but some I extract ideas from to augment my own skills https://github.com/flurdy/agent-skills/tree/main/skills#shar...

Though most of the time my skills are just things I found useful and could avoid repeating myself by having as a skill.

That I also use it to route model used with https://github.com/flurdy/pi-skill-model-router is also a reason

reply