That's a fairly new thing. Growing up, I never heard anyone use "ginger" in that way. It was always "redhead." Or sometimes "strawberry blonde" for lighter shades.
It's true. A person is ginger, but nobody ever says things are ginger coloured. Ginger isn't even really one colour - it's a spectrum of orange-containing hair colours.
Were there many reds around when you were growing up? My family is full of redheads, and ginger was a well known term to me. I'm much older than South Park