The layer of CdTe is thinner than a hair, is hermetically sealed inside and CdTe is practically insoluble in water. It is also much less toxic than elemental cadmium.
Even if the panel cracks and a torrent of rain flushes through it, the runoff would not dissolve/transport any to the soil.
Sure, dust and shards shouldn’t be ingested, but I think that applies to all types of panels.