Yes, although then people came up with reasonable ways of parallelizing RNNs entirely or in part anyway. There's also the need for lookups in transformers. Queries still need to be multiplied with old keys, all of them, so RNNs don't have to be totally incompatible with parallelism. It's just that you can't have a matrix inside them like in the linear RNN h_t = Ah_t + Bx_t that you have to multiply together step after step. If A is an input-dependent scalar or something you can multiply together easily you're fine.

mLSTM layers are completely parallel (the update rule for the cell state is C_t = f_t C_{t-1} + i_t x_t where f_t and i_t are gates that can be computed from x_t alone, x_t is the input at time t and this means that you can compute F_s = \prod_{s<t} f_s as fast as a cumsum and an exponential and then compute C_t = \sum_{s<t} F_s i_t x_t, again as a cumsum, so it's as good as if though it were parallel). I think the sLSTM layers that are the other component of the xLSTM have something else like this and presumably there's some trick also to training the Kimi "delta attention" RNN.

I'm not sure whether this is hardware or optimization dependent to some degree, but I get the impression that good custom kernels are an important part of this kind of thing.