Before you build a distributed system

“The Buddha, the Godhead, resides quite as comfortably in the circuits of a digital computer or the gears of a cycle transmission as he does at the top of the mountain, or in the petals of a flower. To think otherwise is to demean the Buddha – which is to demean oneself.”

― Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance

Most AI features in mobile apps today run server-side, on “frontier models” (or cloud LLMs). These models offer scale and worldwide accessibility you don't get anywhere else. Plus, they're easy to integrate and faster to use for most prototyping.

But that doesn't make server-side models the only choice for every app or feature. Some apps can be served through on-device models that don't require making API calls to a frontier lab, but efficiency here requires balancing tradeoffs for your use case.

When you build a feature on a server-side model, it streams your user information offsite, and changes your app's character – from a native app into a distributed system with a heavy outside dependency. Once your app is a distributed system, data privacy becomes your problem – especially where strict data regulations bar you from sending user information to remote data centers. It also adds a layer of security risk, internet dependency, and latency.

That new server-side “AI feature” is also a significant cost item, at least for the time being. Maybe your app doesn't need to be a voracious API subscription consumer. Sure, you can pass the subscription cost on to your users, but cost pass-throughs don't last forever.

It's also important to understand how the decision is ultimately influenced by your combination of data, algorithms, and machine: the quality, volume, and arrival rate of your data, the arithmetic intensity of the model you're working with, and the compute and memory status of the typical user's hardware.

Balancing these factors against your product niche tells you where the model should be deployed. Together they set the level of feature processing, monitoring, and resource management available to you. And they answer one straightforward question: “Can my workload run on the user's mobile phone?”

If your planned AI feature doesn't need the entire knowledge base of the internet, why serve it with a gigantic model with hundreds of billions of parameters?

For comparison, Apple's on-device AFM 3 Core Advanced model has 20 billion parameters, but uses a sparse architecture that activates only the relevant fraction of the model's total network, which then reduces parameters usage to 1-4 billion per request. The smaller, baseline AFM 3 Core model has just 3 billion parameters and it executes cleverly for lightweight tasks, offering maximum privacy, offline capability, no per-token cost, and low latency. Right now, the 3B model is the one you can actually ship against; the 20B one is how Apple runs Siri and dictation.

Mobile ML suits personal AI – especially where privacy is the pitch, or where the feature has to work offline. Where user data is already sitting in your app, an on-device model might be appropriate. It suits applications like Signeasy for contracts management and MoneyCoach for transaction categorization – both of which built those features on Apple's on-device model – and other uses like inferring input from free text, providing a knowledge base, voice-to-task, and generating or managing data in a nutrition app.

Sizing this is arithmetic. At 4-bit quantization, weights cost roughly half a byte per parameter – about 500MB for a 1B model, 1.5GB for a 3B – plus a KV cache that grows with context length. And the number that matters isn't the phone's RAM but the per-app budget the OS will let you hold before it kills you; a 12GB device doesn't mean 12GB for you. The good news is how small small can get: Liquid AI reports its 350M-parameter model running at 88 tokens per second on an iPhone 13 Mini in 56MB.

A fine-tuned, small-parameter, context-dependent model with enough coverage brings sufficient intelligence to a mobile app built around that work.

First aid and emergency service apps especially need low model latency. A cloud call has to clear the radio, the handshake, and the provider's queue before it returns a single token. Published benchmarks put time-to-first-token between 450ms and 1.1 seconds – measured from a wired server, before your user's network gets a vote. Add the phone's own network and you're budgeting 600ms to 1.3 seconds for that round trip on a good 5G connection, and several seconds on a weak signal. Right now, for an emergency services app this server trip takes too long.

A small on-device model answers in 10 to 100 milliseconds and doesn't care whether there's a signal at all. For a first-aid app, that isn't a performance preference. It's whether the feature exists in the basement, elevator, on the subway, a remote town, or on an airplane – this is what decides whether the AI feature in your app is useful.

However, on-device does not win everything. Cloud throughput is far higher (200+ tokens/sec for a cloud model versus roughly 10 tokens/sec for a 3B model on mobile silicon), so for long generated output requests the cloud can finish first even though it starts later. For work that needs sustained throughput – long-form generation, code, math – the cloud is usually the right call.

Sustained inference is the hard part. A mobile phone has no fan: under continuous load an iPhone 16 Pro has been measured dropping from 40.5 to 23.7 tokens per second – a 41% drop – and it starts throttling by the third iteration. Any benchmark from a cold device is just marketing. Budget for the throttled number.

There are two on-device deployment strategies, and they fail differently. Call the OS model – Apple's AFM 3 Core, Android's Gemini Nano – and you pay nothing in memory or download size, but you only reach devices new enough to have it. Ship your own weights and you reach almost everything, at the cost of owning the bundle, the quantization, and managing the memory budget yourself.

Fragmentation is the other constraint. Google's on-device stack wants 12GB of RAM and a current flagship chip, which excludes most of the Android installed base, so in practice you ship the on-device path and a cloud fallback.

Running a model on-device is an engineering commitment rather than pure cost optimization: you're trading a subscription for a training set, an eval loop, a thermal budget, and the fact that every model change now ships through App Store review. For a bounded feature running on data already in your app, that's a trade worth making. For an open-ended one on arbitrary hardware, it isn't. Knowing which you have is the whole job.

Still, deploying a model isn't a clear-cut choice between cloud and on-device. Some designs call for a hybrid workload: an on-device model sitting close to the user data, paired with a much larger cloud LLM. You can have on-device handle the common case and anything touching private data; the cloud handles the long tail and long output tasks.

The industry has been shipping on-device ML for a decade. Google Pixel's Now Playing and Recorder have run fully on-device speech recognition since 2019; Snapchat runs real-time on-device vision at enormous scale; Brutalist Report is a working implementation whose creator has written about why Local AI needs to be the Norm. LLMs are the new arrival to an old practice, not the other way round. The Buddha that lives on the mountain lives just as comfortably in the circuits of the phone in your pocket – if you respect its limits.