Just point an agent at your llm logs and ask it to generate a dataset of questions and answers from the problems you solved already.