Massively simplifying here but when doing one rasterization pass you process a list of triangles, in order, once. When raytracing a camera view you query a database of triangles, at least once for each pixel. To avoid scanning the whole list of triangles for every pixel, the database needs a fancy 3D spatial index whose structure depends on the exact position of each vertex. When you modify the position of vertices you have to fix up or rebuild the fancy spatial index too.

To make ray tracing practical, you need to update the ray-tracing accelerating structures when you animate.