We'll probably never know how they were perceived in society back then, if it was e.g. like Asians versus westerners now.
Of course, it depends on who you are reading. There are plenty of disagreements on taxonomy among the published academics and 19th-century literature is still influential, and pop-sci writers and their AI pets rarely do thorough research.
Just last week I was in the Coa valley in the iberian penninsula observing upper paleolithic rock art that could only be credited to Neanderthals (because there is zero evidence sapiens has migrated to Portugal 35,000 years ago; Spain was still covered in ice sheets and provided a physical barrier). Excavations had found stone tools and even a burial at nearby sites. Chances are really really good you wouldn't be able to pick out a Neanderthal if one were wearing modern dress walking down the street. Those academics who classify them as a separate species are merely propagating old imperialistic beliefs from the 19th century.