Singers drift because they use relative pitch, because most musicians dont have perfect pitch.

With relative pitch music sounds the same even if you deviate from the original equal temperament pitch of the key you started singing even changing the key.

For the same reason if there is a fixed instrument playing at the same time, like a piano accompaniment, it's sound would be used as a reference and the singers would not drift