So somewhat unlikely to be able to exploit it but have at it hass.
I'd imagine it'd be easier to exploit the server side, actually.
The client on the other hand I would guess is running it's code as root, or at least something with full GPU access.
There are extensions to RTMP to use encryption or even just TLS. But do you mean that the risk of fragments of audio and video bitstreams going out unprotected presents a remote code execution risk? That seems less a problem of vulnerability and more one of privacy.