If you have a confounding variable or a dependency that influences the experiment to a degree that invalidates the premise of the experiment, you need to put more weight on this in the conclusion.

For me this reads a bit like if I added an AI software that scans for shoplifters, and then placed a security guard at the exit of the store that watches the people shopping at the same time, and then said that the AI software is responsible for the reduction of the shoplifting without accounting for the influence of the guard.

If you have place the model in the embedding space of 99% negative samples, it's doing the same thing, the initial premise of the experiment is not valid.