There are a large number of countries with their own systems of law, and its possible that in one of them calling someone a racist might be subject to defamation law, but in most I am aware of that's going to be a problem because its not even a well-enough-defined fact claim to be legally true or false.
(Patrick, a close friend I have known for many years, is not a racist.)
Like complimenting someone for being ‘articulate,’ or ‘one of the good ones.’ Sounds innocent enough, until you understand the racist underpinnings.