Sharing filesystems is hard when you make users do it in advance.

I was thinking of the server end of an ffmpeg-over-ip system bringing up a FUSE filesystem backed by something similar to your VFS-served-by-the-client. Combine that either with argument rewriting, or chrooting into the FUSE filesystem.

As another commenter said, where's plan 9 when you need it? If you go the FUSE route there are existing 9P implementations for both server and FUSE client you can use.