I keep those rarely used ones in there as a useful record of the syntax that I created the alias to get around in the first place. Fish shell's abbreviations are even more useful in this respect as they expand to the full command.
Also, if one is not sure what's behind an alias, M-C-e will expand it inline. If it turns out to be not the one you hoped for, C-/ (undo) is one keypress away.