“What if your AI feature worked at 30,000 feet, in a tunnel, with zero signal — and nobody could ever steal your users’ data because it never left the device?”
You’ve shipped a Flutter app. It’s fast. It’s beautiful. And somewhere in your roadmap, there’s an AI feature — a smart reply, a document summarizer, a local assistant.
So you do what everyone does: you call an API. You send user data to a cloud server. You pray the latency doesn’t ruin the UX. And every month, that API bill quietly climbs.
But there’s another way.
On-device LLMs in 2026 are no longer a research experiment. They’re production-ready, and the model landscape has exploded — Gemma 4, Qwen3, Phi-4 Mini, DeepSeek R1 distills, SmolLM. Flutter, through a single package, can now run all of them. Locally. With GPU acceleration. With multimodal support. With thinking mode.
Let me show you the full picture.
Why On-Device? (The Real Reasons, Not the Marketing Ones)
Before we touch code, let’s be honest about why this matters:
- Privacy by default. User data never leaves the phone. No GDPR nightmares. No breach liability. No “we store your queries for 30 days” footnote.
- Zero latency (seriously). No network round-trip. Inference on a modern phone can feel near-instant for small models.
- Works offline. Airports, subways, rural areas — your AI feature keeps working.
- No API costs. The model runs on the user’s GPU/NPU. Your infra bill doesn’t move.
- No rate limits. One user can hit the model 10,000 times a day. You pay nothing extra.
The tradeoff? Model size and capability. You’re not running Gemini 2.5 Pro on a phone. But for focused tasks — classification, summarization, Q&A over a small context, code completion, smart suggestions — 2026’s small models are genuinely impressive.
The 2026 Landscape: What Can Actually Run on a Phone?
This is where things have changed dramatically. A year ago, your only realistic option was Gemma 2B. Today:
Model Size RAM Best For Flutter Support Gemma 4 E2B ~1GB ~2GB General chat, multimodal, audio ✅ flutter_gemma Gemma 4 E4B ~2GB ~3GB High-quality reasoning, vision ✅ flutter_gemma Gemma 3n E2B/E4B ~1–2GB ~2–3GB Multimodal (text + image + audio) ✅ flutter_gemma Gemma 3 1B (INT4) ~600MB ~1.2GB Lightweight chat, mid-range devices ✅ flutter_gemma Gemma 3 270M ~150MB ~400MB Ultra-lightweight, background tasks ✅ flutter_gemma Qwen3 0.6B ~400MB ~800MB Reasoning + thinking mode, multilingual ✅ flutter_gemma Phi-4 Mini ~2.5GB ~3.5GB Coding hints, strong reasoning ✅ flutter_gemma DeepSeek R1 Distill (1.5B) ~1GB ~2GB Chain-of-thought, step-by-step reasoning ✅ flutter_gemma SmolLM 135M ~90MB ~200MB Edge devices, classification, autocomplete ✅ flutter_gemma FastVLM 0.5B ~300MB ~600MB Fast vision-language tasks ✅ flutter_gemma
💡 The big shift: flutter_gemma v0.15.0 (May 2026) now supports ALL of these under a single package. One dependency. Every model. GPU-accelerated on Android, iOS, and even desktop (macOS, Windows, Linux via LiteRT-LM).
The One Package That Rules Them All: flutter_gemma
Stop reaching for MediaPipe directly. The flutter_gemma package (by Sasha Denisov at mobilepeople.dev) now abstracts everything:
# pubspec.yaml
dependencies:
flutter_gemma: ^0.15.0
path_provider: ^2.1.0
http: ^1.2.0
What this single package gives you in 2026:
- 🖼️ Multimodal — text + image + audio input (Gemma3n, Gemma 4, FastVLM)
- 🧠 Thinking mode — visible chain-of-thought (Gemma 4, DeepSeek R1, Qwen3)
- 🛠️ Function calling — native tool use without prompt engineering
- ⚙️ CPU/GPU switching — choose your backend per model
- 🔍 On-device RAG — text embeddings + vector store built in
- 🖥️ Desktop support — macOS, Windows, Linux via FFI (no JVM)
Setup: Android & iOS
Android (android/app/build.gradle)
android {
defaultConfig {
minSdkVersion 24
ndk {
abiFilters "arm64-v8a" // GPU acceleration requires 64-bit
}
}
}
iOS (ios/Runner/Runner.entitlements)
For larger models (Gemma 4 E4B, Phi-4 Mini), you’ll need to request extended memory entitlements:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"> <plist version="1.0"> <dict> <key>com.apple.developer.kernel.extended-virtual-addressing</key> <true/> <key>com.apple.developer.kernel.increased-memory-limit</key> <true/> </dict> </plist>Minimum iOS 16.0 required for the larger models.
Choosing Your Model (Decision Tree)
What does your feature need?
├── Text only
│ ├── Smallest possible footprint? → SmolLM 135M
│ ├── Good reasoning on tight budget? → Gemma 3 270M
│ ├── General chat + multilingual? → Qwen3 0.6B (thinking mode!)
│ ├── Balanced quality + speed? → Gemma 3 1B INT4
│ ├── Strong coding / step-by-step? → DeepSeek R1 1.5B distill
│ └── Best quality, high-end devices? → Phi-4 Mini or Gemma 4 E4B
│
├── Vision (image + text)
│ ├── Fast image Q&A? → FastVLM 0.5B
│ └── Rich multimodal understanding? → Gemma3n E2B / Gemma 4 E2B
│
└── Audio (voice input)
└── Voice → AI response, on-device? → Gemma3n E2B/E4B
Step 1: Smart Model Download
Never bundle the model in your APK/IPA — it will kill your install conversion rate. Download on first launch with progress:
import 'dart:io';
import 'package:path_provider/path_provider.dart';
import 'package:http/http.dart' as http;
class ModelManager {
// Model registry — swap modelUrl to change models
static const models = {
'gemma3-1b': {
'url': 'https://huggingface.co/litert-community/Gemma3-1B-IT/resolve/main/gemma3-1b-it-int4.task',
'filename': 'gemma3_1b_int4.task',
'sizeMb': 600,
},
'qwen3-0.6b': {
'url': 'https://huggingface.co/litert-community/Qwen3-0.6B/resolve/main/qwen3-0.6b-int4.task',
'filename': 'qwen3_0.6b_int4.task',
'sizeMb': 400,
},
'smollm-135m': {
'url': 'https://huggingface.co/litert-community/SmolLM-135M-Instruct/resolve/main/smollm-135m-int4.task',
'filename': 'smollm_135m_int4.task',
'sizeMb': 90,
},
};
static Future<String> getModelPath(String modelKey) async {
final model = models[modelKey]!;
final dir = await getApplicationDocumentsDirectory();
final file = File('${dir.path}/${model['filename']}');
}
static Future<String> _downloadModel(
File destination,
String url, {
void Function(double progress)? onProgress,
}) async {
final client = http.Client();
final request = http.Request('GET', Uri.parse(url));
final response = await client.send(request);
}
}
Step 2: Initialize with the Right Model Type
import 'package:flutter_gemma/flutter_gemma.dart';
class OnDeviceLLM {
final FlutterGemma _gemma = FlutterGemma.instance;
Future<void> initialize({
required String modelPath,
required ModelType modelType,
bool useGpu = true,
bool isThinking = false, // Enable for DeepSeek, Qwen3, Gemma 4
}) async {
await _gemma.init(
modelPath: modelPath,
modelType: modelType, // ModelType.gemma3, .qwen3, .deepSeekR1, etc.
maxTokens: 1024,
temperature: 0.8,
backend: useGpu ? Backend.gpu : Backend.cpu,
);
}
// Available ModelTypes in flutter_gemma 0.15.0:
// ModelType.gemma4 → Gemma 4 E2B / E4B (thinking + function calling)
// ModelType.gemma3n → Gemma3n E2B/E4B (vision + audio)
// ModelType.gemma3 → Gemma 3 1B
// ModelType.gemma3_270m → Gemma 3 270M (ultra-light)
// ModelType.functionGemma → FunctionGemma 270M (tool-use specialist)
// ModelType.qwen3 → Qwen3 0.6B (thinking mode)
// ModelType.qwen25 → Qwen 2.5 1.5B
// ModelType.phi4 → Phi-4 Mini
// ModelType.deepSeekR1 → DeepSeek R1 1.5B distill
// ModelType.smolLM → SmolLM 135M
// ModelType.fastVLM → FastVLM 0.5B (vision)
}
Step 3: Generate Responses (With Thinking Mode)
The biggest new capability in 2026 — thinking mode. Models like Qwen3, DeepSeek R1, and Gemma 4 can show you their reasoning chain before the final answer. It’s like having console.log for the model's brain.
class LLMService {
final FlutterGemma _gemma = FlutterGemma.instance;
// Standard streaming response
Stream<String> chat(String userMessage) {
return _gemma.streamChatResponse(
messages: [
Message(role: Role.user, content: userMessage),
],
);
}
// Thinking mode — get reasoning + final answer separately
Stream<InferenceToken> chatWithThinking(String userMessage) {
return _gemma.streamChatResponseWithThinking(
messages: [
Message(role: Role.user, content: userMessage),
],
isThinking: true,
);
// InferenceToken has:
// .text → the token content
// .isThinking → true if this is part of reasoning chain
// .isFinal → true when generation completes
}
// Vision input — send image + question (Gemma3n, Gemma 4, FastVLM)
Stream<String> analyzeImage(Uint8List imageBytes, String question) {
return _gemma.streamChatResponse(
messages: [
Message(
role: Role.user,
content: question,
image: imageBytes,
),
],
);
}
}
Step 4: The Full Chat UI with Thinking Blocks
class ChatScreen extends StatefulWidget {
const ChatScreen({super.key});
@override
State<ChatScreen> createState() => _ChatScreenState();
}
class _ChatScreenState extends State<ChatScreen> {
final LLMService _llm = LLMService();
final TextEditingController _controller = TextEditingController();
final List<ChatBubble> _messages = [];
bool _isGenerating = false;
String _thinkingBuffer = '';
String _responseBuffer = '';
bool _showThinking = false;
Future<void> _sendMessage() async {
final input = _controller.text.trim();
if (input.isEmpty || _isGenerating) return;
}
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(
title: const Text('On-Device AI 🧠'),
actions: [
IconButton(
icon: Icon(_showThinking ? Icons.psychology : Icons.psychology_outlined),
onPressed: () => setState(() => _showThinking = !_showThinking),
tooltip: 'Toggle Thinking',
),
],
),
body: Column(
children: [
// Live thinking stream during generation
if (_isGenerating && _thinkingBuffer.isNotEmpty && _showThinking)
Container(
color: Colors.amber.withOpacity(0.1),
padding: const EdgeInsets.all(12),
child: Text(
'💭 ${_thinkingBuffer}',
style: const TextStyle(
fontStyle: FontStyle.italic,
fontSize: 12,
color: Colors.grey,
),
),
),
}
Widget _buildBubble({
required Role role,
required String content,
String? thinking,
bool isLive = false,
}) {
final isUser = role == Role.user;
return Align(
alignment: isUser ? Alignment.centerRight : Alignment.centerLeft,
child: Column(
crossAxisAlignment:
isUser ? CrossAxisAlignment.end : CrossAxisAlignment.start,
children: [
if (thinking != null && thinking.isNotEmpty)
Container(
margin: const EdgeInsets.only(bottom: 4),
padding: const EdgeInsets.all(8),
decoration: BoxDecoration(
color: Colors.amber.withOpacity(0.15),
borderRadius: BorderRadius.circular(8),
),
child: Text(
'💭 $thinking',
style: const TextStyle(
fontSize: 11,
fontStyle: FontStyle.italic,
color: Colors.grey,
),
),
),
Container(
margin: const EdgeInsets.symmetric(vertical: 4),
padding: const EdgeInsets.all(12),
decoration: BoxDecoration(
color: isUser
? Theme.of(context).colorScheme.primary
: Theme.of(context).colorScheme.surfaceVariant,
borderRadius: BorderRadius.circular(16),
),
child: Text(
content.isEmpty && isLive ? '▌' : content,
style: TextStyle(
color: isUser
? Theme.of(context).colorScheme.onPrimary
: Theme.of(context).colorScheme.onSurfaceVariant,
),
),
),
],
),
);
}
}
class ChatBubble {
final Role role;
final String content;
final String? thinking;
ChatBubble({required this.role, required this.content, this.thinking});
}
Bonus: On-Device RAG with flutter_gemma
One of the most underrated features added in 2026 — flutter_gemma now ships a built-in vector store + text embeddings API. You can do full RAG (Retrieval-Augmented Generation) without any external database:
// Embed your documents locally — no cloud needed
final vectorStore = await _gemma.createVectorStore();
await vectorStore.addDocument(
id: 'doc_1',
content: 'Your Flutter app security checklist...',
);
await vectorStore.addDocument(
id: 'doc_2',
content: 'How synapse-cortex manages persistent memory...',
);
// At query time, retrieve relevant context
final results = await vectorStore.query(
query: 'How do I prevent reverse engineering?',
topK: 3,
);
// Inject into your prompt
final context = results.map((r) => r.content).join('\n\n');
final prompt = '''
Based on the following context:
$context
Answer: How do I prevent reverse engineering in my Flutter app?
''';
// Then stream normally
_llm.chat(prompt).listen(print);
Your docs never leave the device. Your embeddings are local. Your retrieval is instant. This is the stack that makes a real offline AI assistant possible.
Real-World Performance (2026 Devices, GPU backend)
Device Model Tokens/sec First Token iPhone 16 Pro Gemma 4 E2B ~22 t/s ~400ms Pixel 9 Pro Gemma 4 E2B ~18 t/s ~500ms iPhone 15 Gemma 3 1B INT4 ~20 t/s ~500ms Samsung Galaxy S24 Qwen3 0.6B ~28 t/s ~300ms Mid-range Android (2023) SmolLM 135M ~80 t/s ~80ms Mid-range Android (2023) Gemma 3 270M ~40 t/s ~150ms
The story in 2026: GPU backend is a must. CPU fallback is fine for async tasks but not interactive chat. flutter_gemma defaults to GPU and falls back to CPU automatically.
The Architecture I Recommend: Adaptive Tiering
Don’t kill the cloud entirely. Think in three tiers:
User Input
│
▼
┌──────────────────────────────┐
│ TIER 1: On-Device (instant) │ ← SmolLM, Gemma 3 270M
│ Intent detection │ Always-on, <100ms, private
│ Autocomplete, triage │
└─────────────┬────────────────┘
│ Needs reasoning or context?
▼
┌──────────────────────────────┐
│ TIER 2: On-Device (quality) │ ← Gemma 4, Qwen3, DeepSeek R1
│ Full chat, summarization │ ~500ms first token, private
│ RAG over local docs │ Works offline
└─────────────┬────────────────┘
│ Needs real-time data / heavy compute?
▼
┌──────────────────────────────┐
│ TIER 3: Cloud (escalation) │ ← Gemini, Claude, GPT-4
│ Complex reasoning │ Only when explicitly needed
│ Real-time lookups │ User-aware, opt-in
└──────────────────────────────┘
Handle 80% of queries in Tier 1 and 2. Your users get speed + privacy. Escalate to cloud only for the 20% that genuinely need it.
What NOT to Do in 2026
🚫 Don’t bundle the model in your APK/IPA. A 600MB+ model destroys install conversion. Always download post-first-launch.
🚫 Don’t run inference on the main thread. It freezes your UI. flutter_gemma handles isolation internally, but if you wrap it yourself, use Isolate.run().
🚫 Don’t skip INT4 quantization. The INT4 .task files run 3–4x faster with negligible quality loss on chat tasks. Always prefer the quantized variant.
🚫 Don’t ignore thinking mode for your use case. For a casual chatbot — skip it (latency overhead). For a “why is my code wrong?” feature — enable it. The reasoning trace is genuinely useful.
🚫 Don’t hardcode one model. The ecosystem is moving fast. Build a ModelType enum into your settings so you can swap models without a full release cycle.
The Bigger Picture
In January 2025, running a capable language model on a phone was a party trick.
By May 2026, a single Flutter package (flutter_gemma: ^0.15.0) gives you: Gemma 4 with multimodal + audio + function calling + thinking mode + on-device RAG — across Android, iOS, and desktop — in one dependency.
The developers who figure out the on-device + cloud hybrid pattern now will have a serious moat in 18 months. Faster apps. Better privacy stories. Zero API dependency risk. Users who trust you more because you never touch their data.
Your users’ data deserves to stay on their devices.
And honestly? Your app will feel better for it. 🧠
What’s Next
In the next post, I’ll show how I wired synapse-cortex (my local-first MCP memory server) into an on-device Flutter app — so the model not only runs locally but remembers across sessions, builds a knowledge graph of your usage patterns, and gets smarter over time. All without a single cloud call.
Follow along if that sounds interesting.
Tags: Flutter, AI, On-Device AI, Machine Learning, Mobile Development, LLM, Gemma, 2026