Self-attention is an O(n^2) operation. You don't want to train on individual characters, or bytes.