Are you talking about the secret shared between the NAS and the RADIUS server? It's only used to scramble some attributes (like MS-MPPE-Send-Key), but not all of them. Message-Authenticator is one that's not scrambled. Looking at this FreeRADIUS dictionary file I have, I see 42 out of ~6000 attributes that are scrambled.
Anyway, yeah, if you have a bigass shared secret, it's going to be infeasible to guess. I'm pretty sure that the long-standing very, very strong suggestion for operators has been something like "If you don't co-locate your RADIUS server and your NAS, then you really need have a bigass shared secret, and probably want to be using something like IPSec to secure the connection between the two." [0][1]
[0] <https://datatracker.ietf.org/doc/html/rfc3579#section-4.3.3>
[1] <https://datatracker.ietf.org/doc/html/rfc3579#section-4.2>