In September 2025, two of the most widely used packages in the JavaScript ecosystem were hijacked. Attackers phished a maintainer, pushed malicious versions to the public registry, and for a window of time anyone who installed them pulled code designed to steal cryptocurrency. Within days a self-propagating worm known as Shai-Hulud tore through the npm registry, harvesting credentials, cloud tokens, and API keys from developer machines and build pipelines, then using the stolen tokens to publish infected versions of other packages automatically. The first wave compromised hundreds of packages. A second wave that November reached more than twenty-five thousand code repositories in a matter of hours.
I am not telling this story to scare anyone off open source. I am telling it because it reframes a decision that most teams make on autopilot. The reflexive plan for adding semantic search to a product is to call a hosted embedding API, push the resulting vectors into a managed vector database, and pay per request forever. Every part of that plan is a third party you depend on, a credential you have to hold, and a place your data travels to. The npm attacks of 2025 were a loud reminder that every one of those is attack surface. So before reaching for the default, it is worth asking a question almost nobody asks. Do you even need it?
The number that decides everything
The single most important question is how many documents you are searching over. The honest answer, for a great many internal tools and product features, is not millions. It is hundreds, or low thousands. A search feature I built recently runs across roughly three hundred records, short titles and descriptions of a couple of hundred words each.
At that scale, the entire premise behind a managed vector database evaporates. You do not need an approximate nearest neighbour index to search three hundred vectors quickly. You can hold them in memory and compare against all of them on every query, and it is still effectively instant. The infrastructure that exists to make billion-vector search tractable is pure overhead when your corpus fits comfortably in memory.
What a small local model actually buys you
The model I run is a compact, quantised sentence encoder. It is about twenty-five megabytes on disk and produces 384-dimensional vectors. It downloads once, lives on disk, and never phones home again. Here is what that changes in practice. On cost, a hosted API charges a fraction of a cent per query, every single time, forever, while the local model costs nothing once it is on disk. On latency, the hosted call is a network round trip you do not control, while the local model settles at roughly 80 to 120 milliseconds once it is warm. On data exposure, the hosted route sends your content to a third party, while with the local model nothing leaves your server. And on failure modes, the hosted service brings rate limits, outages, and key rotation, while the local model is, in the end, just a function call.
The cold start is real and worth naming. The first query after a restart takes a few seconds while the model warms up. After that, queries sit in that 80 to 120 millisecond range, which is below the threshold where a person perceives a wait in a search box.
On quality, the worry people carry is that a small open model must be far behind the proprietary giants. For short text at small scale, that gap is much narrower than the marketing suggests. Compact sentence encoders score competitively with the large hosted models on short-text retrieval. The proprietary edge shows up on long documents and subtle, paragraph-length similarity, which is exactly the workload most internal search features do not have.
What this looks like in practice
The implementation is less code than the hosted version, not more. You load the model once at startup. For each document, you compute its vector when the document is created or its text changes, and you store that vector on the document itself, alongside the text, rather than in a separate database. At query time you embed the search text once and compare it against the stored vectors. There is no separate vector store to provision, secure, and keep in sync, because the vectors live next to the data they describe. Keeping them fresh is a matter of recomputing when the source text changes, with a periodic sweep as a safety net for anything the normal path missed.
The cost difference compounds in a way that is easy to underestimate. A hosted embedding call is cheap per request, but a search feature embeds on every query and often re-embeds documents as they change. Multiply a fraction of a cent by every search and every edit across a busy product, every day, for the life of the feature, and the running total is real money for something a twenty-five megabyte file on a server you already pay for does for free. The local version front-loads a small amount of engineering and then costs nothing to run. The hosted version is effortless to start and then bills you forever.
The security angle nobody counts
Here is where the npm story comes back. When you call a hosted embedding API from your application, you are holding an API key, and that key sits in your environment, your configuration, and very often your continuous integration pipeline. The Shai-Hulud worm spread precisely by harvesting credentials of that kind from build environments and developer machines. Every external service you wire into the hot path of your application is one more secret that can be stolen and one more dependency whose compromise becomes your compromise.
Running the model locally removes that entire category of risk. There is no embedding API key to leak, because there is no embedding API. There is no customer text travelling to a third party that might log it, get breached, or change its retention policy. There is one fewer external call in the request path and one fewer vendor in your threat model. In a year when the biggest software supply chain stories were about stolen tokens and poisoned dependencies, fewer credentials and fewer third parties is not a minor convenience. It is a genuine reduction in how much can go wrong.
Do not throw away keyword search
The mistake that follows the decision to add embeddings is treating semantic search as a replacement for keyword search. It is not. A pure vector search will happily fail to find a document when the user types an exact term that is sitting right there in the title, because the vectors landed slightly apart. Users find that maddening, and they are right to.
The version that actually feels good is a blend. Run a deterministic keyword layer and a semantic layer, then combine the scores. The weighting I settled on, after testing against real queries, leans toward meaning but keeps exact matching with real influence.
// when an embedding exists for the document
combinedScore = 0.4 * lexicalScore + 0.6 * semanticScore;
// when it does not, fall back to keyword only
combinedScore = lexicalScore;The keyword layer guarantees that an exact match always surfaces. The semantic layer catches the cases where the user describes a thing in words that never appear in the document. Neither alone is enough. The fallback line matters too. A document that has not been embedded yet is still findable by keyword, so the feature degrades gracefully instead of going silent.
The part that protects you later
The honest objection to running things locally is that you might outgrow it. The corpus could grow tenfold, or you might one day need long-document quality. This is a real risk, and you handle it at the seam. Keep the scoring and the search interface independent of which embedder produced the vectors. The rest of the system should ask for a vector and a score and not care where the vector came from. When you genuinely need to swap in a hosted model or a managed index later, you change the embedding step and leave everything downstream untouched. You get the cheap, private, simple version today, and you keep the door open to the heavier version for the day you actually need it, which for many features never arrives.
The takeaway
Before you reach for a hosted embedding API and a managed vector store, count your documents. If the number is in the hundreds or low thousands, you can almost certainly run a small model on your own server, blend it with keyword search, and ship something faster, free at query time, and private, with less moving infrastructure and fewer credentials than the default plan. Reserve the heavy machinery for the scale that genuinely demands it. Most features do not, and quietly paying as though they do, in money and in attack surface, is a cost you chose without meaning to.
Sources
npm supply chain compromise of popular packages, September 2025 (Sonatype, CISA advisories).Shai-Hulud self-propagating npm worm, September 2025 and November 2025 (Wiz, Palo Alto Networks Unit 42, Check Point Research).