Unless they are using a linear architecture, the compute cost still scales O(n²) for n tokens, and nemory cost scales O(n).
reply>the compute cost still scales O(n²) for n tokens,
replyThat is never the cost, it's a common misconception.
Cost scales linearly per tokens. Unless you are sending one token at a time and avoiding using the same machine or cache.
Just look at api charges, they are charged by token, not by token squared.
Which seems to contradict the usual consensus that purely linear architectures are not sufficiently capable and unsuited for frontier models.
reply