On-Device AI: When Your App Shouldn't Call the Cloud

The default way to put AI in a mobile app is to call a cloud API. The phone sends text up, a big model in a data center thinks about it, and the answer comes back. It is simple, it works, and for a lot of features it is now the wrong default. In 2026 a capable language model runs on the phone itself. Gemma 3 ships in sizes down to 270M parameters, Llama 3.2 comes in 1B and 3B, and Phi-4 mini fits comfortably on a mid-range Android. The neural engine in a modern phone runs them fast enough that the first token lands in under 20 milliseconds, with no network round-trip at all.

That is a real architectural fork, not a gimmick. Cloud inference adds 200 to 500ms of latency before you see anything, costs you per token forever, and sends your users' data to a server you then have to secure and explain in a privacy policy. On-device inference is instant, free after the model ships, and keeps the data on the phone where regulators increasingly want it. The catch is that it is not free of tradeoffs, it just moves them. This is a guide to when on-device AI is the right call for a mobile app, what it takes to build, and the costs the demos never show you.

The shift: the model moved onto the phone

Three things changed at once. Models got small enough that a sub-billion-parameter model handles real tasks like summarizing, classifying and rewriting text. Phones got Neural Processing Units that are now standard on flagships and common on mid-range. And the runtimes matured: ExecuTorch hit 1.0, and most popular edge models run out of the box across a dozen hardware backends.

The result is that "AI feature" no longer has to mean "API call". A note-taking app can summarize on the device. A camera app can describe a scene offline. A keyboard can rewrite a sentence without a single byte leaving the phone. None of that was practical two years ago and all of it is shipping now.

When on-device wins, and when it doesn't

The decision comes down to a few honest questions, not to whether on-device sounds modern.

Most apps end up hybrid. The small on-device model handles the common, latency-sensitive, privacy-touching work, and it hands the rare, hard requests up to a bigger cloud model. That routing is where the real engineering is, and it is worth getting right rather than picking one side on principle.

What it costs to build

The model is free to run; the engineering is not. Budget for the work that cloud APIs hide from you. You will spend time picking and quantizing a model so it fits in memory and stays fast, wiring in a runtime like ExecuTorch or a platform framework, and testing across a spread of devices where the same model behaves very differently on a three-year-old phone than on this year's flagship.

There is also app size. A quantized small model adds tens to a few hundred megabytes to your download, which you manage by fetching it on first launch rather than bundling it. None of this is exotic, but it is real mobile engineering, and a team that has only ever called an API will under-estimate it. For a sense of the broader number, see what a mobile app really costs to build in 2026.

The tradeoffs nobody puts in the demo

Constant local inference generates heat and drains battery. Run a model in a tight loop and the phone gets warm and the battery graph drops, so you cache results, run on-demand rather than continuously, and lean on the NPU instead of the CPU. You also give up central control: once a model ships inside the app, you cannot hot-swap it the way you edit a server prompt, so improving it means an app update and a slower feedback loop.

And quality has a ceiling. A 270M model is genuinely useful and genuinely limited. The teams that succeed are the ones that scope the on-device feature to what a small model does well, then measure it on real devices instead of trusting a benchmark run on a workstation.

A practical starting point

Pick one feature where privacy, offline support or instant response actually matters to your users, and prototype it on-device before you decide anything global. Measure the real latency, the battery cost and the output quality on the cheapest phone you support, not the most expensive. That single honest test tells you more than any trend piece.

On-device AI is not a replacement for the cloud, it is a second tool that happens to be cheaper, faster and more private for a specific and growing set of jobs. If you are weighing where AI belongs in your mobile app, talk to us and we will help you draw the line in the right place.