It's a distasteful, but relevant, aspect of vikings that they were slavers as well as raiders. If you went viking, a large part of the booty you brought back walked on two legs and had genes to pass on. Perhaps the Norse liked their neighbours just enough not to make many of them "visitors".