I think if you were going to "downsample" for the purpose of creating a color set you could just scan through the picture and randomly select 10% (or whatever) of the pixels and apply k-means to that and not do any averaging which costs resources and makes your colors muddy.
I should probably add this nuance to the post itself.
Edit: added a footnote
[0]: https://dgroshev.com/blog/okmain/img/distance_mask.png?hash=...