Dfu-util is one of those "user space drivers", so if you have a nonstandard protocol you'd have to add it directly to dfu-util. There's no intermediate API.

It's not easy to set up a fake or "remapped" USB device on most OS as far as I'm aware, if you were trying to write an adapter program that modified USB packets.

dfu-util actually also just uses libusb under the hood! Any class or device that doesn't have a driver baked into the OS can be implemented like this. And if you'd need the DFU functionality in a different application, you may be able to just simply link parts of the dfu-util tool into your application