I agree 100%. This field is not amenable to progress from people with a pen sitting in a corner proving theorems. The math is mostly uncertain vibes and to test it you need millions of dollars of compute. Smart loners just can't.
replyreplace they key-query-value mechanic by just dropping it while making the entire context the latent space.
replyWhat do you mean by this? Like concatenating all token embeddings into one large vector?