This is an efficiency improvement that significantly lowers the amount of RAM you have to look at, on average, during decode.

It should improve performance on most hardware because most LLMs are memory bandwidth bound during decode.

Maybe somaday an 8gb videocard can be used for coding...