Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD.
It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.
gte-small outscores all-MiniLM-L6 on MTEB (~61 vs ~56 avg per the GTE paper). MiniLM is ternlight's teacher (ternlight holds 0.84 Spearman fidelity to teacher).
I haven't run a head-to-head yet; STS-B/MTEB numbers are on the roadmap. Also on the roadmap is to distill gte-small as teacher.
but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.
This doesn't add any malware risks beyond what a JavaScript-enabled browser already allows.
Re excessive browser memory use: Yes, it adds non-negligible weight, but again, you could already achieve excessive browser memory usage before this. For comparison, a true color 1080p image, uncompressed (which is needed for actual display on screen) is only slightly smaller at 6.22Mb.
That's... how the web works? You download things on demand.
There are JS files larger than 7MB in the wild. They run on JIT engines that displayed severe CVEs over the years. PDFs, video running directly on special hardware encoders. That's the web now.
Very cool! I'd love to point it at my own corpus to index/embed. Would be cool if you could give it a link to a markdown file or even a website to crawl.
It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.
Demo (2k React docs, fully on-device): https://ternlight-demo.vercel.app
Two tiers on npm: - @ternlight/base (7 MB, ~5 ms/embed, more capable embedings) - @ternlight/mini (5 MB wire, ~2.5 ms/embed).
Bundled for Node and browsers.
Repo - see technical details (MIT, training pipeline included): https://github.com/soycaporal/ternlight
Curious if this is something useful, what are the use cases for on-device embeddings.
but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.
First search downloads the model from the internet and subsequent runs are from the cache.
The model is very small so it's not the best for everything but it's good for basic math and coding.
Give it a try.
Loading model... + Loading search results...
Or sometimes "Service Worker API is available and in use." + "Loading search results...".
Inference is nice and quick after that.
"Hmm, 7MB would barely make a dent in the size of the app and allow us to do some of our basic ML without calling the backend"
Probably a lot more practical to use this though: https://developer.apple.com/apple-intelligence/
Re excessive browser memory use: Yes, it adds non-negligible weight, but again, you could already achieve excessive browser memory usage before this. For comparison, a true color 1080p image, uncompressed (which is needed for actual display on screen) is only slightly smaller at 6.22Mb.
There are JS files larger than 7MB in the wild. They run on JIT engines that displayed severe CVEs over the years. PDFs, video running directly on special hardware encoders. That's the web now.
A WASM model is not that offensive.