upvote
Let's say you have a way of encoding the properties of something using a vector of numbers. But two things might be very similar and yet still produce vectors that are not identical to each other. Indeed, the vectors are nearly guaranteed to always be at least slightly different. Also, assume there are a large number of dimensions.

You want to assign things to a set of clusters so that things that share similar properties are grouped closely together. But, we also want those clusters to be as big as possible. We don't want to leave little islands that only have a few things in them.

Take, for example, classification of email messages. Let's say you have a billion email messages and you want to find a way to cluster them automatically into 10 categories. You don't know what the categories are ahead of time. You just have a way of making sure that an email can be represented by a vector of numbers such that two emails that are close together in their meaning and characteristics will have similar numbers.

K-Means clustering solves that problem but is intractable to calculate for large datasets having high dimensionality (such as the email example). My understanding is that E-Means is away to achieve this spectacularly more efficiently than the best alternative previously known.

reply
It's clustering. You have a bunch of data points and you want to group them into k clusters such that all the data points in a cluster are similar. Like dividing customers into high/medium/low spenders, or whatever (usually you'd have more dimensions than 1D though).
reply
This is what I was looking for. I appreciate all the responses.
reply
Faster vector calculations. What help for example, helps with ivfflat or prism index building. It cut CPU (or GPU) usage, and time needed to build. Seen index build times drop by 50 to 80%, and at the same time CPU usage drop by 50%.

Impressive results ...

reply
I found this unhelpful, sorry. "Faster vector calculations" sounds like a faster way to add or multiply vectors, which TTBOMK k-means doesn't help with at all. There are hundreds of types of things called an "index"; I'm familiar with ordinary RDBMS indices but haven't heard of ivfflat or prism indices. Gemini tells me PostgreSQL uses them to support approximate nearest neighbour queries on geometric data -- e.g., finding the nearest train station to your current position.
reply
Well, thanks for the downvote.
reply