My biggest complaint about the fish shell is the lack of true vi mode. They attempt to emulate it and it works to some degree, but it's no comparison to readline's implementation.
Just pressing `xp` to swap two characters does not work in fish. Combining deletion with a movement also does not work (e.g. `d3w` to delete three words).
You can always use Alt-E to open the command line in $EDITOR if you need more powerful commands. I find it better to use readline for small changes and jumping to vim for bigger ones.
Have you tried a recent version? An issue I opened about this years ago was finally closed, they claim it’s fixed now. I haven’t tried the purported fix, though.