You probably can — by using neovim:

https://wiki.archlinux.org/title/Neovim#Use_as_a_pager

https://neovim.io/doc/user/filetype/#_man

I've also been running (neo)vim as a manpager. You get the same features as with vim (like easily copying text or opening referenced files/other manpages without using the mouse), but neovim also parses the page and creates a table of contents, which can be used for navigation within the page. It doesn't always work perfectly, but is usually better than nothing.