True. I'm a big fan of Cactus's work on the needle family of Simple Attention Network: https://github.com/cactus-compute/needle
replyThat looks like a really nice and well put together library.
replyBut it feels like it sort of backs up my point about there being good models at every size class. Fine tuning Needle looks automated. Yes, you need to know basics like what validation loss means and how to use Python, but otherwise it's all about creating the dataset.
Someone had to come up with needle in the first place though. And it's the kind of thing that's going to come from a big lab with an AGI monopoly ambition.
reply