1. Neanderthal woman "Ann" mates with Human man "Hugh"
2. Ann gives birth to son "Ander", who is then raised with neanderthals. Notably, Ander has human Y chromosome via Hugh, but Neanderthal X chromosome via Ann.
3. Ander mates with human woman "Uma". They have a daughter, passing Neanderthal X chromosome into human population.
I realize this is a very specific path, but it would _only have to happen once_ for the neanderthal X chromosome to be introduced into the human genome. I think it is very unlikely that such a path would simply never happen across the thousands of interactions we had. And therefore I think the observed fact (no impact of neanderthal x chromosome in modern genes) can't _just_ be explained by the proposed behavior (neanderthal mothers raise their children in their neanderthal tribe)
I think there does actually have to be some sort of incompatibility, or some other very-very-high failure rate, something like 99.99%.