They are trained with one, and when you look at DPO you can say they contain an implicit one as well.