> And this was a debug session from hell, enormously helped by an AI doing much of the grunt-work.
> I'd like to call it my tireless helper, but the AI several times stated flat out that this was impossible and unsolvable and that we should just write a report about it.
> I suspect those things have been trained by people who may not be quite as stubborn as I am.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
On the data analysis side, something specific I’ve noticed is an (understandable) bias towards computing numerical statistics, which they do very well and reading the post-analysis report has significantly improved my own “statistical thinking” approach overall. Numerical statistics are cool and understandably what a text-based LLM is going to want to work with, but asking the model to produce time-domain and frequency-domain plots of, say, specific events has multiple times resulted in “trying to plot this out has shown the opposite of what I concluded numerically… recalculating…” There’s still a pretty significant review and critically assess step for me, especially since the actions I take as a result of the analysis are pretty expensive, especially if they steer the next data collection run in a useless or harmful direction.
https://arxiv.org/abs/2309.11495
A RL pipeline can reinforce verification behaviour even better than simple prompting.
This is says more about humans tendency to pattern match than anything else.
X works better than Y only is only a useful observation if we are using the same X and Y in a similar context, with similar parameters. Kind of goes out the window without it and I think this is part of why people have such vastly different opinions about the same technologies. We’re all talking past each other.