Hugging Face released @huggingface/kernels at the start of September, a library of 207 reusable WebGPU kernels for running inference in the browser. The headline is performance. The more interesting part is what it does to your architecture.

Most teams route every AI feature through the same path: user action, API call, model provider, response. It works. It also means you are paying per token for a spellcheck, and sending a user's private notes to a third party to classify them into two buckets.

Not every inference call deserves a network hop

There is a whole tier of AI features that are small, high frequency, and low stakes. Semantic search over a user's own documents. Intent classification on a search box. Summarising a note as the user types. Live translation. Embedding generation for client-side filtering. Background removal on an uploaded image.

These share three traits. They run constantly, they operate on data the user already has locally, and a 300ms round trip is genuinely noticeable. That is exactly the profile that argues for running the model on the device.

The benchmark numbers back this up. Depending on model size and task shape, WebGPU can be 10 to 15 times faster than the WebAssembly fallback. For a small embedding or classification model, that is the difference between a feature that feels instant and one that feels laggy.

The real decision criteria

Stop asking whether a model is small enough to run in the browser. Ask two better questions.

How often does this call fire? A feature that runs on every keystroke has a completely different cost curve to one that runs when a user clicks "generate report". Per-token pricing punishes frequency. On-device inference has a fixed cost paid once at download.

Does the data have any reason to leave? If you are classifying something the user typed, about themselves, for their own benefit, the network trip adds risk and a compliance conversation for no product benefit. Keeping it local removes an entire category of problem.

Anything that needs a frontier model, shared state, or server-side data stays where it is. This is not an argument for moving everything.

What it actually costs you

Browser inference is not free, it just moves the bill.

Download weight. A quantised model is somewhere between 20MB and 200MB. That is a real hit on first load and you need a caching and versioning strategy for it, not a hopeful fetch.
Device variance. WebGPU support sits around 70% globally and skews toward newer hardware. You need a WASM fallback path, and you need to test on the low end, not just your MacBook.
Two code paths. The local model and the server model will not give identical output. Decide up front which one is authoritative and where the seam sits.

None of this is exotic. It is the same work you would do for any progressive enhancement, just with a bigger asset.

The takeaway

The default of "AI feature means API call" was a reasonable shortcut when browser inference was a demo. It is not a demo any more. Go through your AI features and mark the ones that fire constantly on local data. Those are the candidates. Move one, measure the latency and the bill, then decide about the rest.

We're here to help founders and teams design and build digital products that are built to scale with you, not slow you down. If you're looking to build something, get in contact with us today!