upvote
That mirrors my experience exactly. I keep a couple of older models (and much larger ones as well) for when I want to do stuff that is unrelated to computer programming because the models that are really good at that tend to fare less well at other stuff. I suspect that there is a boundary where maximizing benchmarks for coding at the expense of general knowledge is going to give you a model that is really fast and good but 'narrow' and that this may also lead to loss of performance when applied to coding tasks that are less well represented in the training set (such as, for instance, DSP code). This is a variation on 'the bitter lesson', short term you can make gains like this but long term leveraging compute and data will give you something better. I'm really happy I didn't waste half a life on engineered AI because I would be really quite miserable if I had.

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

reply