There are a large number of countries with their own systems of law, and its possible that in one of them calling someone a racist might be subject to defamation law, but in most I am aware of that's going to be a problem because its not even a well-enough-defined fact claim to be legally true or false.
(Patrick, a close friend I have known for many years, is not a racist.)