undefined

points

[-]

Would it be correct to say you have experience building LLM based workflows like this? I'm guessing so, given by your critiques and suggestions of better approaches. Can you recommend any books/sites/other for learning these kind of dos and donts?

by YossarianFrPrez19 hours ago|

parent|

[-]

While I have some experience, I'm afraid I am learning mostly through a) trial and error, and b) using an llm chat interface to double check my thinking.

This being said, here are my top recommendations:

1. Build your system against real targets. Had HackerRank continually tested their prompt against 2-3 real resumes that were scored by hand, I think some of the issues would have immediately popped out. The people who built the prompt thought they could magically skip the hard part of articulating a preferred decision making process by having the LLM do it. But LLMs are much better at scaling a pre-existing decision-making process rather than inventing one, let alone the same one, from scratch, every time.

2. Think about what it would take to get motivated undergraduate interns to do the task from end to end, step by step. That's essentially what your workflow will need.

3. If the LLM can't do a step or sub-task reliably, then it's time to decompose those subtasks into even smaller chunks.

I'm sorry I can't be more helpful!

by WickyNilliams13 hours ago|

parent|

[-]

No apologies necessary, that's perfectly helpful. Appreciate the response and the candour!

by prime_ursid21 hours ago|

parent|

prev|

[-]

This is a technique called LLM-as-Judge. Well-studied at this point. Here’s a good intro: https://www.evidentlyai.com/llm-guide/llm-as-a-judge

I recommend reading Hamel.dev posts. Here’s an example: https://hamel.dev/blog/posts/evals/

by WickyNilliams13 hours ago|

parent|

[-]

Awesome, thank you! I am starting to do some work building LLM workflows and would like to stand on some giant's shoulders to skip the initial flailing :)