Every AI feature I'd built before this ran the same way: my server calls a hosted model, pays per token, and hopes the rate limit holds. Chrome's built-in AI breaks that pattern entirely. Gemini Nano runs inside the browser itself, and a website can call it with a plain JavaScript API — window.LanguageModel. No key, no network round-trip to run a prompt, no bill that grows with traffic.
Three things this isn't
- It isn't a hosted API call. No request leaves the device to OpenAI, Claude, or Gemini's cloud endpoint. No latency waiting on a network hop, no quota, no per-token invoice.
- It isn't "run Ollama yourself" local AI either. That's a great pattern for developers, but it means asking every visitor to install and run their own model server first — dead on arrival for a public website. Nobody does that to read a portfolio.
- It's a browser capability. Chrome downloads and manages Gemini Nano once, on-device, shared across every site that uses it. Your code never touches the model file — you just ask the browser to run a prompt against it.
How it actually works
The API is a small, session-based surface:
const availability = await window.LanguageModel.availability();
// "unavailable" | "downloadable" | "downloading" | "available"
const session = await window.LanguageModel.create({
initialPrompts: [{ role: "system", content: "..." }],
});
for await (const chunk of session.promptStreaming("your question")) {
// stream tokens as they generate — entirely on-device
}
The first call on a fresh Chrome install triggers a one-time model download, handled entirely by the browser — not your app, not your infrastructure. After that, every session on every site reuses the same on-device model.
The catch that isn't really a catch
Right now this is desktop Chrome only — not mobile Chrome, not Safari, not Firefox. That's a real platform limit, not a bug to route around. The honest move is to detect it and degrade gracefully: show the feature where it works, show a plain explanation where it doesn't, rather than a broken button or a silent gap. availability() exists precisely so you never have to guess.
The actual engineering lesson: small models need scaffolding, not longer prompts
Running fully on-device means the model is small by design — that's the tradeoff that makes zero-cost, zero-latency AI possible at all. A small model doesn't reliably obey every instruction the way a giant hosted one might, and building a real feature on it means writing code that doesn't just trust the prompt:
- Chrome's streaming API has shipped two different chunking behaviors across versions — some builds stream the cumulative answer so far, others stream only the new delta. Assuming the wrong one silently truncates every response. The fix is to detect which mode a session is in from its own output, not hardcode an assumption.
- A single "stay on topic" line in a system prompt is not a boundary — it's a suggestion a small model will sometimes ignore. The reliable fix lives in code, checked before the model is ever called, not in wording harder inside the prompt.
That's the real shift this API asks of you: less "prompt engineering," more "treat the model like an unreliable narrator and build the guardrails around it in code."
Why it's worth caring about now, not later
Zero marginal cost per visitor, at any scale. Nothing typed ever leaves the device — genuine privacy, not a policy promise. No backend to run, patch, or pay for. This isn't a future roadmap item; it's shipping in Chrome today, and any site can use it.
I built exactly this into my own portfolio — an assistant, bottom-right, that runs the way described above. Try it live in Chrome.