I think the explanation is much simpler, we know the Norse were a bit afraid of the Sami. They viewed them as a weird non-threatening neighbour people who had a weird language and weird magic. So you traded with them, you respected them, you said please and thank you, and then you were happy to see them gone because you didn't want them to curse you. (And I would assume the Sami were very happy to foster this belief since they were much weaker militarily)

Unlike the fat and rich continental Europeans that the Norse viewed as ripe for plunder, they did not fear them at all.

doubtful given histories of the area, however maybe they disliked getting retaliation for a raid.