This is eerily similar to something I do with Hacker News stories that hit the front page. I run the post against a couple of LLMs (Mixtral, GPT-OSS, Qwen3, etc.) with the directive to produce a set of 20 of the most likely top-level replies.

I then wait a few days, and then use a couple of systems (embeddings, deBERTa, etc.) to rank comments by novelty against the LLM-produced replies.