But everything is fast enough, so moving to the M5 seems like a downgrade, as it is heavier

Well when macOS feels heavier now than before so is that 2x really a 2x?