The problem is that it's too expensive. It can technically do parts of my job for me, but at a variable cost that's higher than mine right now.
replyEfficiency is an angle of research that's quite active, see quantization, MoE, speculative decoding, QKV cache optimizations, sparse attention, etc
reply