That's basically how radio is accounted for in royalties, as well.
With Spotify knowing exactly who listened to what, it could be more precise (and arguably more susceptible to the fraud), but tbh what they do is standard (compulsory licensing) industry practice.
If either of those was true with spotify, the unfairness would go away.
But when different listeners are paying very different amounts per hour, any correlation between payment amount and preferred content causes problems.