It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.
Demo (2k React docs, fully on-device): https://ternlight-demo.vercel.app
Two tiers on npm: - @ternlight/base (7 MB, ~5 ms/embed, more capable embedings) - @ternlight/mini (5 MB wire, ~2.5 ms/embed).
Bundled for Node and browsers.
Repo - see technical details (MIT, training pipeline included): https://github.com/soycaporal/ternlight
Curious if this is something useful, what are the use cases for on-device embeddings.
Do you think your work could help us let users type "pancake" and get "crêpe" without writing an explicit "pancake = crêpe" dictionary entry ?
In practice : if I understand well, your lib would first need to download 5 Mb, once and for all, and would then be used as we use Fuse.js right now ?
How well does it handle languages other than English ?
Could it be "trained" on the OpenStreetMap tag wiki ?
Thanks a lot for your work.
Ideally you would have real query data (e.g. from cartes.app telemetry), then you could get a LLM to write a bespoke Overpass query for each one and use that as the ground truth. Alternatively, start from the list of OSM tag values used in the wild and ask an LLM to list possible reasons to visit that POI.
You could then use that data to finetune an embedding model for your use case. But, you know, somewhere in that model there's going to be a token vocabulary that the model knows about and at the other end you get a similarity score for each tag value. If you don't need to support complex queries where interactions between words matter ("any restaurant that is NOT Korean"), you could get away with a simple list of words and tags that they match to. Which is right where you started, except it could be more exhaustive. Why limit yourself to two Korean dishes when you can have a LLM list many more for you?
Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)".
But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 400/sec. Is there an issue with falling back to a non-SIMD path? I'll try a native Rust binary next.
bge-small-en-v1.5 is one that is comparable and what we’re working with for now.
We've just used it to embed the entire django doc + our private knowledge base, allowing us to search in the 2 sources instantly!
The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.
but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.
This way on the frontend you can lazily load this. Maybe you could even store the HNSW in chunks and just load the pieces you need for your specific search query.
i.e. like https://pagefind.app/ but to get fully static vector search.
If anybody knows of a good solution in this space, or if I’m wrong about SQLite-vec, please let me know. For our own SSG we’ve basically decided that we’ll give it a couple months while we work on other infra we want, then if they’re still not done we’ll just do it ourselves.
What I think is really cool is that the search happens using http range queries across statically hosted parquet files.
I think things like this could bloom into a relatively open and distributed search ecosystem that isn’t controlled by major corporations.
It’s advertised 7MB, but also comes with a 5MB mini version.
Looks like mini saves space by using 256 element vectors internally instead of 384, but then projects it up to 384 at the end for compatibility.
It’s a third smaller, but the loss is not linear, looks like you give up less than 1/3 of information with the smaller data path.
So I pull ONNX weights from HuggingFace (MPNet, MiniLM), use Transformers.js to embed, and use a clusterer from scikit-learn (running on pyiodide - it was a surprise to me that this worked flawlessly) on the page - all client-side.
Inference is nice and quick after that.
First search downloads the model from the internet and subsequent runs are from the cache.
The model is very small so it's not the best for everything but it's good for basic math and coding.
Give it a try.
Loading model... + Loading search results...
Or sometimes "Service Worker API is available and in use." + "Loading search results...".
Tested on Macbook Pro M1 8gb RAM and Macbook Air M1 8gb RAM. Mostly likely because of M series of chips. All tests were done on Brave/Chrome.
Does not work on iPhone 11 Pro Max and iPhone 16 Pro. Mostly likely because of A series of chips. Tests were done on Safari and Chrome and it crashes on both.
The workaround is to unregister/stop the service worker from the DevTools > Application tab > Service workers.
"Hmm, 7MB would barely make a dent in the size of the app and allow us to do some of our basic ML without calling the backend"
Probably a lot more practical to use this though: https://developer.apple.com/apple-intelligence/
Language.complete('the quick brown fox jumped over the lazy')
Image.generate('a spaceship flying toward a planet')Keep up the great work!
Re excessive browser memory use: Yes, it adds non-negligible weight, but again, you could already achieve excessive browser memory usage before this. For comparison, a true color 1080p image, uncompressed (which is needed for actual display on screen) is only slightly smaller at 6.22Mb.
There are JS files larger than 7MB in the wild. They run on JIT engines that displayed severe CVEs over the years. PDFs, video running directly on special hardware encoders. That's the web now.
A WASM model is not that offensive.