That's becoming less common. The issue is that, if training only contains examples of CVE reports, then asking for one will get you one, even if it's not grounded in reality. But if training consists of many negative cases with the response, "I can't find one," them that will become the default unless evidence pushes it towards a report. It's a classic class size problem. SOTA models seem to have this largely solved, though a determined person (just like a determined boss) can prompt one into giving a false answer just to satisfy the request.

Two different points.

First, is there any reason they have to? The LLM makers are aware of this problem and doing things to stop it. Maybe they will start saying "I can't find anything". Already they refuse to provide some exploits.

Second, most people are not prompting for a CVE, they are asking to review and find issues if any. Which sometimes finds a CVE, but in my experience it sometimes does say "looks good".

Hence future CVEs on a lot of projects will likely demand working exploit code for testing. This at least attempts to lock the issue down to something more tangible.