Mobile App + AI Agent: 7 Architectural Patterns for Embedding LLMs in iOS and Android

The seven patterns are: Cloud Round-Trip, Edge-Cached Cloud, Hybrid Local-Cloud, On-Device Small Model, Federated Inference, Agent Sidecar, and Streaming Speech-to-Speech. Each has a clear best-fit use case. Choosing wrong is the most common architectural mistake we see in AI-enabled mobile apps and the one that usually kills the user experience first.

Pattern 1: Cloud Round-Trip

The app sends a request to your backend, which calls a frontier LLM, and returns the result. Best for: deep reasoning tasks where 1–3 second latency is acceptable — long-form generation, document analysis, complex planning. Tradeoffs: lowest implementation complexity, highest per-request cost, full server-side control of prompts and tools, requires connectivity. Default starting point for most apps.

Pattern 2: Edge-Cached Cloud

Same as Cloud Round-Trip but with aggressive caching at edge nodes — Cloudflare Workers, Vercel Edge, AWS Lambda@Edge. Common queries are served from cache; novel queries fall through to the model. Best for: apps with high query repetition (FAQ-style assistants, common-query helpers). Tradeoffs: dramatic cost reduction (50–85% in some workloads) for repeated queries, slight increase in implementation complexity, careful invalidation needed when underlying data changes.

Pattern 3: Hybrid Local-Cloud

A small on-device model handles classification, intent detection, or preprocessing; a cloud model handles the heavy reasoning. Example: on-device classifies a user query as "schedule," "summarize," or "search," and only the latter goes to the cloud. Best for: apps where many user requests do not actually need a frontier model. Tradeoffs: lower cost and faster response on simple cases, more complex codebase, more model drift to manage.

Pattern 4: On-Device Small Model

A 1B–8B parameter model running entirely on the device — Phi-3, Gemma 2B, Llama 3.2 1B/3B, Apple Foundation Models, Google AICore. Best for: privacy-sensitive use cases, offline capability, latency-critical UX (autocomplete, suggestions, summaries while typing). Tradeoffs: model quality is meaningfully below frontier models, cold-start memory pressure, app size grows by the model size, careful battery management on Android. We use this for assistants that must work offline or on regulated data.

Pattern 5: Federated Inference

Inference runs on a fleet of devices for collaborative learning or shared knowledge updates without raw data leaving devices. Best for: privacy-preserving personalization, especially in healthcare and financial apps. Tradeoffs: significantly more complex than centralized training, longer iteration cycles, mature only for narrow use cases. Niche but important when data residency requirements rule out everything else.

Pattern 6: Agent Sidecar

A separate process — often packaged as a system extension on iOS or a foreground service on Android — runs the agent loop independently of the main app. The main UI sends intents to the sidecar; the sidecar runs the long-running plans. Best for: apps where the agent needs to keep working when the user navigates away (background research, scheduled tasks, long-running automations). Tradeoffs: meaningful platform-specific work, battery and memory governance is harder, lifecycle complexity. Worth it when the value is high.

Pattern 7: Streaming Speech-to-Speech

Audio in, audio out, with the model embedded in the audio pipeline rather than treated as a discrete request-response. Implementations use real-time WebRTC or WebSocket transports, server-side STT-LLM-TTS pipelines (Deepgram + Claude + Cartesia is a common stack), or model-native voice integrations. Best for: voice assistants, accessibility features, hands-free use cases. Tradeoffs: highest implementation complexity, best UX when done right, latency sensitivity is extreme — anything over 800ms feels broken.

Choosing between the patterns

Three questions decide most cases. Does the app need to work offline? If yes, you are in Pattern 4 or 5. Is the use case privacy-critical or regulated? Pattern 4 (on-device) is preferred; Pattern 1 with the right BAA can work. Is latency under 500ms required? Pattern 4, Pattern 7, or aggressive caching in Pattern 2. Most apps end up with two or three patterns coexisting: a small on-device model for autocomplete and quick suggestions, plus a cloud round-trip for the heavy lifting.

Implementation gotchas we see most often

iOS background execution: agent sidecars on iOS face strict background rules. Plan around them or use silent push notifications carefully. Android battery and Doze mode: foreground services are visible to users; design the UX so that visibility is justified. App size: shipping a 2 GB on-device model in a 30 MB app is jarring; use on-demand downloading and clear progress UI. State sync between local and cloud: when both have memory of the user, conflict resolution becomes a real engineering problem. Eval coverage: mobile agents have additional eval requirements — battery, thermal throttling, network variability, OS version compatibility — that web-only agents do not.

Cost patterns to plan for

Cloud round-trip dominates cost in chatty apps; aggressive caching in Pattern 2 is the easiest 50%+ cost cut you will find. On-device models shift the cost from per-request inference to one-time download bandwidth and a higher device-class minimum spec. Hybrid patterns are the most efficient at moderate scale because most queries do not need frontier reasoning.

What is changing in 2026

Three structural shifts. First, on-device models above 4B parameters now run usably on flagship phones, which expands Pattern 4's range significantly. Second, Apple Foundation Models and Google's AICore both shipped APIs that reduce the cost of doing on-device LLM features by 60–80% versus rolling your own. Third, network-side voice integration shrank Pattern 7's latency floor to ~300ms, making voice agents finally feel real.

The studios shipping the best AI-enabled mobile apps in 2026 do not pick one pattern — they pick the pattern per surface within the app. Voice assistant: Pattern 7. Quick search: Pattern 4. Document summarization: Pattern 1. Background research: Pattern 6. Treat the architecture as a portfolio decision per feature, not a global one for the app.