upvote
Re: Max dimensions: You can truncate your input vector specifically for the index, and then when querying, you can query the index via a truncated vector and perform an additional full-length comparison as well.

For example: (taken from a project I'm currently working on)

```

SELECT *

FROM report_embeddings a

INNER JOIN report_embeddings b

    ON (

        subvector(a.embedding, 1, 768)::vector(768) <-> 
subvector(b.embedding, 1, 768)::vector(768)

    ) < $1

    AND b.id != a.id
WHERE a.id = $2

    AND a.embedding <-> b.embedding < $1
```

In this example, I truncate the embedding to 768 dimensions for the index and the lookup, (see the INNER JOIN) but do an additional full vector comparison as well. (in the WHERE clause)

This performs a fast but less accurate lookup into the index, and then only does the non-indexed full vector check for the matched results.

reply
That’s only going to work on an MRL embedding.
reply
I have used pgvector on an Azure DB. It did a "reasonable" job. I am not an expert on HNSW vs PRISM but because it is all backward compatible, I am curious why they didn't just send a PR to the pgvector folks?

Like, I would love to learn more about the design process that led to a whole new thing. I am sure the decision was not taken lightly, but the README is very thin on the details for why anyone should go through the troubles above, at least in the making of it.

reply
Fair question. When we built our first vector extension, pgvectorscale, we actually reached out to pgvector's author to see if they'd be interested in supporting additional index types (at the time, DiskANN-based), so we could contribute our work directly to pgvector. At the time, their preference seemed to be to focus on the existing index types rather than expand the project in that direction. We respected that decision and went ahead with a separate extension.

But one of the great things about open source and PostgreSQL's extension architecture is that these projects can coexist! pg_vectorsearch can work directly with pgvector's existing data types, so developers can use either index.

reply
Main pg_vectorsearch contributor here. There are multiple reasons why simply “sending a PR” is more difficult than it sounds.

As a company, we want to move at a different pace and try things that a project like pgvector wouldn’t necessarily allow (and rightfully so). pg_vectorsearch is alpha-level software, while pgvector is stable. Releasing early is a way to make it available to the community sooner and getting feedback.

Then there are all kinds of technical reasons: different architectures, different implementation choices, etc. There's also the freedom to try new things and going in completely different directions.

reply
"Just sending a PR" is not that easy
reply
[flagged]
reply
Points 2 and 3 would probably apply to every interesting bleeding-edge Postgres extension out there.
reply
so basically you should use Qdrant anyway lmao :p
reply