I've been thinking of just using sandpaper stuck to a block of wood, though I imagine that might be slower.

Heck, a little part of me is tempted to try the smallest radius round-over router bit I have in a trim router, but the odds of that going horribly wrong are just way too high.

Or get a Dremel.