Just because a driver's license isn't proof of citizenship doesn't mean that it isn't sufficient, since it will have some identifier connected to the identity of the voter. You would show the driver's license, then the registrars would find your name on the list, strike it out, give your voting envelopes, and then you go wait for a private booth to become free and then you go into that booth and put your votes in those voting envelopes.
A Swedish driver's license doesn't mean Swedish citizenship. It can still be used to check somebody's identity. Citizenship is checked by seeing whether the person is on the voter rolls, not by their identity card.
The point I tried to make is that with the system we have here it's much less of a hassle to prove you're eligible to vote. The system wants you to be able to vote, and to be able to do that easily.
Nitpick: E.g. Norwegian ID cards can be had by any legal resident. The card will say that you're not a citizen though, so your point about that stands.
Re driving license: Driving license is a valid form of ID for voting here. All eligible people are on the voting rolls automatically (unlike the absurd american system), and the driving license is merely used to prove you are who you say you are.
Your eligibility to vote and have your name entered into the poll book is determined by providing a Social Security number or birth certificate.
So yes, a driver’s license is sufficient ID to vote in many states.