A quick reminder of my POV (in case you're new here): I'm a Flutter & AI Engineer. I'm comfortable on both sides: I fine-tune, train and quantize my own models, and I wire them into real apps. I'll always walk you through my entire journey in achieving the goal - what worked and what didn't. The "didn't" part is where I learn the most, and I'm sure you will too.

If you want my full argument for why you'd want to run a model on-device (offline, private, free, independent), I explained that in my previous article. On desktop, the same reasons apply, except that you typically have more RAM, a real GPU, and no strict app-store size limits to fight with. Desktop is infact, the better place to run local AI models.

This article is an end-to-end, standalone one. By the end, you'll have gotten an LLM running on a Flutter Windows/MacOS/Linux app (on CPU and/or GPU), and you'll know what's left to do to distribute it to end users. I still have other articles coming in this series to show alternative and better engines and methods, but everything here works on its own.

Note: I focus primarily on Windows in this article as it is the most popular laptop/desktop OS in the world right now, however, the process, as I explain it here (specifically from part 2), is the same for MacOS and Linux except for one thing which I shed more light on briefly later in the article.

Part 1: Turning a Flutter App Into a Windows App

I'm assuming you already have a Flutter mobile app project or know how to create one.

What you need first

• The Flutter SDK (of course, you already have this if you've built ever for mobile). Install / upgrade instructions here.

• Visual Studio (not VS Code. You definitely have this already too if you've built for mobile) with the "Desktop development with C++" workload installed. This is non-negotiable as Flutter compiles the Windows shell with MSVC and CMake. Download Visual Studio here and tick that C++ workload during install or go back to edit the install and tick the workload in case you had Visual Studio without the workload.

Run flutter doctor after and make sure the "Visual Studio" line has a green check.

Next: Enabling Windows in your Flutter Installation

This step is usually only needed once, and mostly unnecessary on modern Flutter (Flutter 3.x and above).

flutter config --enable-windows-desktop

If you created the project before you thought about desktop, it won't have a windows/ folder yet. Generate it without touching the rest of your app by running:

# Run this from your project root. The "." is intentional.
flutter create --platforms=windows .

That creates the native Windows runner side-by-side your already existing android/ and ios/ folders. Your Dart code is not tampered with.

To run it and/or build it

# Run in development
flutter run -d windows

# Build a release .exe 
flutter build windows --release

The release build can be found in:

build\windows\x64\runner\Release\

Open that folder and you'll see your_app.exe, flutter_windows.dll, a data\ folder, and a few other DLLs. **Note: That whole folder is your app** - the executable (the .exe file) depends on those other files and generally won't run correctly if you move it by itself. This becomes important when we talk about shipping (in a future article).

That's pretty much what it takes to create a Windows app in Flutter. Now to the more interesting parts about running an LLM in it.

Part 2: The Methods That Didn't Work

On Android, as I explained in my previous article, I eventually found packages (flutter_gemma and llama_flutter_android) that wrapped all the native complexity I was having issues with. So of course, on Windows, I looked for packages too. I assumed it would be a solved problem.

Well, it wasn't -- at least, not at the time.

Attempt 1: flutter_gemma on Windows

flutter_gemma is excellent on Android. On Windows, at the time I tried it, it just wouldn't get the job done. I followed the documentation to the letter, made sure the runtime requirements were in place, and still kept hitting errors around model activation and installation. The model simply would not initialize.

Attempt 2: llama_cpp_dart

Next I tried llama_cpp_dart, which binds to llama.cpp directly. This one got further, but it kept throwing errors about missing DLLs. I resolved one missing dependency after the other on end. It became like a game of whack-a-mole with native libraries. Long story short, it didn't work.

The conclusion I reached

After going through every resource I could find, my conclusion at the time was that the available Flutter packages did not yet have full, reliable Desktop support for running LLMs locally. (TBF, they may have improved since then or maybe I didn't research well enough - actually I think I did - but I had a fast approaching deadline so I had to move quickly.)

To be clear, there is a "proper" way to do this: write your own native C++ bridge and call into it with Dart FFI. But on Android I'd already learned the hard way that FFI is a deep, sharp pit if you're new to it. I wasn't ready to jump into it on Desktop just yet. (Actually, I did eventually, and it's the subject of my next article, so if FFI is what you're here for, just wait for it.)

So I paused and thought about it differently.

Part 3: Thinking Outside the Box - Bundling the llama.cpp Server

Hmm, what if I don't embed the model in my app at all?

llama.cpp isn't only available as a library, it also provides a ready made program called llama-server.exe. That's a small, fast HTTP server that loads a model and exposes an OpenAI-compatible API (/v1/chat/completions and other related ones). It's the same server several people use to run models locally.

So instead of struggling with FFI and DLL perpetually, I figured I could:

1. Bundle llama-server.exe (and my model file) alongside my Flutter app.

2. Launch it as a subprocess ("sidecar") when the app starts.

3. Communicate with it over http://127.0.0.1:<port> like any other normal API.

From my app's perspective, the LLM became just another endpoint, except that this one runs on the user's own machine, fully offline.

Smart innit? 😌

Step 1: Download the server binary file

Download the prebuilt (ready made) Windows/MacOS/Linux release from the official llama.cpp releases page. The files have a clear naming pattern:

llama-b<build>-bin-win-cpu-x64.zip          ← CPU build (start here)
llama-b<build>-bin-win-sycl-x64.zip         ← Intel GPU build (we'll discuss this in Part 5)
llama-b<build>-bin-win-cuda-12.4-x64.zip    ← NVIDIA build
llama-b<build>-bin-win-vulkan-x64.zip       ← cross-vendor GPU build
llama-b<build>-bin-macos-arm64.tar.gz       ← Apple Silicon Macs (Metal GPU on by default)
llama-b<build>-bin-macos-x64.tar.gz         ← Intel Macs (CPU build)
llama-b<build>-bin-ubuntu-x64.tar.gz        ← Linux CPU build
llama-b<build>-bin-ubuntu-vulkan-x64.tar.gz ← Linux cross-vendor GPU build

Choose the CPU zip first to get things working. Inside, you'll find llama-server.exe and its DLLs (its supporting runtime libraries).

Step 2: Get a model (GGUF format)

llama.cpp only works with the GGUF format. You can use a model you fine-tuned and exported to GGUF (I showed how to do that in my previous article), or just find a public one. For these examples I'll use a small instruct model: Qwen2.5-1.5B-Instruct-GGUF. Download the q4_k_m (4-bit quantized version), it's a good size/quality balance for desktop.

Step 3: The sidecar service

"Sidecar" is really just a fancy word for any helper process that runs side-by-side a main application. The helper process in this case is the llama.cpp server. Here's what the service that manages the server process and streams responses looks like. A real one should obviously have more retry logic, but this is the shape of it:

import 'dart:async';
import 'dart:convert';
import 'dart:io';
import 'package:flutter/foundation.dart';
import 'package:http/http.dart' as http;

class LlamaServerService {
  static const String _host = '127.0.0.1';
  int _port = 8088;
  Process? _process;
  bool _useGpu = true;
  bool isRunning = false;

  Future<void> start({
    required String serverBinaryPath, // path to llama-server.exe
    required String modelPath,        // path to your .gguf
  }) async {
    _port = await _findAvailablePort(_port);

    // OpenAI-compatible server arguments
    final args = <String>[
      '-m', modelPath,
      '--host', _host,
      '--port', '$_port',
      '-c', '4096',   // context window
      '-n', '-1',     // no hard cap on generated tokens
    ];

    if (_useGpu) {
      // Offload all model layers to the GPU (see Part 5)
      args.addAll(['-ngl', '99']);
    } else {
      // CPU: use about half the cores, clamped to a sane range
      final threads = (Platform.numberOfProcessors ~/ 2).clamp(2, 8);
      args.addAll(['-t', '$threads']);
    }

    _process = await Process.start(serverBinaryPath, args);

    // Pipe the server's logs to your debug console — invaluable for debugging
    _process!.stdout.transform(utf8.decoder).listen(debugPrint);
    _process!.stderr.transform(utf8.decoder).listen(debugPrint);

    await _waitForReady();
    isRunning = true;
  }

  Future<void> stop() async {
    _process?.kill(ProcessSignal.sigterm);
    await Future.delayed(const Duration(milliseconds: 500));
    _process?.kill(ProcessSignal.sigkill);
    _process = null;
    isRunning = false;
  }
}

The server takes a few seconds to load the model, so we check its health endpoint to know whether it is ready (and we watch for the process dying early, which usually means a bad model - or on the GPU build, maybe a driver problem):

Future<void> _waitForReady({
  Duration timeout = const Duration(seconds: 60),
}) async {
  final stopwatch = Stopwatch()..start();

  // Different server versions expose different "I'm alive" endpoints
  const endpoints = ['/health', '/', '/v1/models'];

  while (stopwatch.elapsed < timeout) {
    for (final endpoint in endpoints) {
      try {
        final res = await http
            .get(Uri.parse('http://$_host:$_port$endpoint'))
            .timeout(const Duration(seconds: 3));
        if (res.statusCode == 200) return; // ready!
      } catch (_) {
        // not up yet, keep polling
      }
    }
    await Future.delayed(const Duration(seconds: 1));
  }
  throw TimeoutException('Server did not become ready in time.');
}

For text generation, it's just streaming a Server Side Event (OpenAI style) response:

Stream<String> chat(List<Map<String, String>> messages) async* {
  final uri = Uri.parse('http://$_host:$_port/v1/chat/completions');

  final request = http.Request('POST', uri)
    ..headers['Content-Type'] = 'application/json'
    ..body = jsonEncode({
      'messages': messages,          // [{role: 'user', content: '...'}]
      'stream': true,
      'cache_prompt': true,          // reuse KV cache across turns — big speedup
    });

  final response = await http.Client().send(request);

  await for (final chunk in response.stream.transform(utf8.decoder)) {
    for (final line in chunk.split('\n')) {
      if (!line.startsWith('data: ')) continue;
      final payload = line.substring(6).trim();
      if (payload.isEmpty || payload == '[DONE]') continue;

      final json = jsonDecode(payload) as Map<String, dynamic>;
      final delta = (json['choices'] as List).first['delta'];
      final content = delta?['content'] as String?;
      if (content != null && content.isNotEmpty) yield content;
    }
  }
}

A small helper to prevent port collisions (in case you already have something on 8088):

Future<int> _findAvailablePort(int start) async {
  for (var port = start; port < start + 10; port++) {
    try {
      final socket = await ServerSocket.bind(_host, port);
      await socket.close();
      return port;
    } catch (_) {/* busy, try next */}
  }
  return start;
}

Then wire chat() into your Cubit/Bloc/Provider/Riverpod just like the streaming pattern from my previous article, and there you've got an offline AI chatbot.

Why this method is actually good (not just a workaround)

To be clear, I didn't eventualy settle for this method on Windows, but it's a strong approach regardless, and here's why:

• Simplicity. No FFI, no ABI issues, no manually chasing missing DLLs. Process.start and http are typical, everyday Dart stuff.

• Reliability & isolation. The model runs in a separate process. If inference crashes (which can happen), it doesn't take the Flutter UI down with it. You can detect the dead process and restart it.

• You use optimized binaries. The llama.cpp team are defolutely better at squeezing performance out of hardware than I am. Every llama release I download is faster than the last, for free.

• A standard API. Because it's OpenAI-compatible, If you're familiar with OpenAI style APIs, it feels very natural. You can even point it at the real OpenAI API as a fallback by changing just one URL.

• Easy CPU/GPU switching. If you want to use GPU on Windows, just swap the binary and add one flag. That is the subject of the next section.

• Cross-platform for free. This same approach works on macOS and Linux with zero code changes. Process.start and http don't really care what OS they're running on. Just put in the right llama.cpp build (bin-macos-arm64 for Apple Silicon Macs, bin-macos-x64 for Intel Macs, bin-ubuntu-x64 for Linux) and everything just works. Even better: on Apple Silicon (Mac M1 and above), that binary has Metal GPU acceleration enabled by default, so Macs get the GPU path without needing any server swap at all.

The only main trade-off I'd say exists is that you're bundling and launching a separate executable, and then of course there's an HTTP roundtrip. In practice on localhost though, that overhead is pretty much negligible compared to the actual inference time.

Part 4: What Is GPU Inference, and Why Bother?

So far, everything I've explained runs on the CPU. And that's fine, but for anything bigger than a tiny model, performance may begin to decline (inference or token generation would be slower).

Generating text with an LLM is basically a huge pile of matrix multiplications. A CPU has a few powerful cores that do things one after another. A GPU on the other hand has thousands of small cores built primarily to do a lot of math in parallel (at the same time). So for AI inference (process of generating a response to your prompt), a GPU is many times faster than a CPU.

Also, another advantage of using GPU is, while the GPU does the heavy inference lifting, the CPU is free to keep your UI smooth and handle everything else the app is doing.

Unfortunately, not all computers have a GPU. So, in production, I run GPU with an automatic CPU fallback. It gives me the best of both worlds. How we set that up is explained below.

Part 5: GPU Inference with the SYCL (Intel) Binaries

If the hardware you're building for is Intel-based (Intel integrated graphics and Arc GPUs), the relevant llama.cpp build is the SYCL one. SYCL is the open standard Intel uses to run compute on its GPUs (via oneAPI). The beauty of the sidecar approach is that switching to GPU is basically a matter of swapping the binary.

Step 1: Download the SYCL build

From the same llama.cpp releases page, download:

llama-b<build>-bin-win-sycl-x64.zip

This contains a llama-server.exe built with the SYCL backend.

Step 2: Tell the server to use the GPU

You must have seen it in the service code - the not-gonna-lie flag -ngl (just kidding 😅, it's actually 'number of GPU layers'):

args.addAll(['-ngl', '99']); // 99 = "offload all layers to the GPU"

99 is really just a comfortably large number that means "put the whole model on the GPU (because many models have fewer than 99 transformer layers)." If the model has fewer layers, it offloads all of them; if it had more, well that's unlikely but it means 99 would be on the GPU and the rest will be on the CPU. Really, what this does is to split the model's transformer layers (its literal building blocks) between the GPU and CPU.

Step 3: The runtime dependency that makes it work

The SYCL build will not run on a Windows computer just because you copied the .exe. It needs Intel's GPU runtime present on the machine. Be sure to check for these two things:

1. Up-to-date Intel GPU drivers. Modern Intel graphics drivers include the Level Zero runtime that SYCL talks to. On most computers, this is already there, but you never know. You can get the latest drivers from the Intel Download Center.

2. The oneAPI / DPC++ runtime libraries. Depending on the particular build, the SYCL llama-server.exe may also need a few Intel oneAPI runtime DLLs (the SYCL/DPC++ redistributables) sitting next to it. The best way to handle this in production may be to bundle those DLLs along with the binary so you don't have to bet on the user computer having the Intel oneAPI runtime installed. (I'll talk more about bundling "seemingly unrelated dependencies" in a future article.)

Step 4: GPU first, CPU fallback

Attempting to use GPU can fail for many reasons outside your control (old drivers, no Intel GPU like I mentioned earlier, a bad machine), so don't put all your eggs in that basket. Try to start in GPU mode, but in the event there's any server crash error or startup delay, restart it in CPU mode. Here's the logic for it:

Future<void> startWithFallback({
  required String gpuBinary,
  required String cpuBinary,
  required String modelPath,
}) async {
  try {
    _useGpu = true;
    await start(serverBinaryPath: gpuBinary, modelPath: modelPath);
  } catch (e) {
    debugPrint('GPU startup failed ($e). Falling back to CPU...');
    _useGpu = false;
    await start(serverBinaryPath: cpuBinary, modelPath: modelPath);
  }
}

You can also scan the server's stderr for specific keywords like sycl, level_zero, gpu not found, etc., so you can log why it fell back, but the try/restart skeleton above is sufficient. The user will never see a failure; worst case scenario, it'll just take longer and then startup.

For non-Intel hardware

If you don't use an Intel computer or machine, the exact same pattern applies — you just download a different binary set of files and the GPU is activated with -ngl:

• NVIDIA: download llama-b<build>-bin-win-cuda-12.4-x64.zip (CUDA). Needs an NVIDIA driver / CUDA runtime.

• AMD: llama-b<build>-bin-win-hip-radeon-x64.zip (ROCm/HIP).

• Anything (cross-vendor): llama-b<build>-bin-win-vulkan-x64.zip (Vulkan). Probably the best option if you don't know your users' hardware.

All from the same releases page. Nothing changes in the code, just the binary files you bundle and you're done.

Part 6: The Single-File Problem (and What's Next)

Now, step back and just negodu what you have to hand to a client:

• your_app.exe and its Flutter DLLs and data\ folder

• llama-server.exe (GPU build) plus its Intel runtime DLLs

• llama-server.exe (CPU build) for fallback

• one or more large GGUF model files

A folder full of files, some of them large, with hidden dependencies (the VC++ runtime, the Intel runtime). You cannot reasonably tell a client "unzip this and find the actual app executable file to open." You need to give them a single file that installs everything correctly, pulls in the unrelated dependencies, and even cleans up itself when uninstalled. There are native solutions for this on MacOS and Linux, but on Windows, it requires a third party.

That's going to be the subject of the next article in this series: Bundling a Flutter Windows App and Everything It Needs Into a Single Installer.

And, that's it.

I'd recommend this route if you want the shortest, most reliable path to a having local LLM on a Flutter Desktop app, as it avoids native code entirely and leans on compiled, ready to run packages/binaries.

But it's not the only way, and it's arguably not the neatest. In a future article, I'll write about how I finally did the thing I avoided here: writing a native C++ FFI bridge, using Intel's OpenVINO GenAI, which runs the model in-process and gave even better results on Intel hardware. After that, we will make the app see and hear with vision and speech recognition models all offline.

Got questions, or want to go deeper on any part? Visit my website for my contact details. Watch out for the rest of this series.

Before we begin, you should know my POV: Although I have some experience on Machine Learning, I am primarily a Flutter Mobile/Desktop app developer. Infact, when I started working on this, I did not know the difference between a base model and an instruct model; terms like "QLoRA" and "quantization" meant nothing to me.

What I can tell you is that I figured it out anyway (learning a lot on the way), and if you're a mobile developer who wants to run AI locally inside your app without needing a backend or an internet connection, this article is for you. By the end, you'll understand how to fine-tune a model if you need to, and more importantly, how to actually get one running inside a Flutter Android app. In a future article, I'll write about running a model locally in a Flutter desktop app.

One more thing: you don't need to fine-tune a model to follow along with the integration section in this article. Pre-trained models are available for free in formats that work directly with the integration approach I'll show you. Fine-tuning is an option, not a prerequisite. So if you're here only for the Flutter integration, feel free to skip Part 1 entirely and jump directly to Part 2 (the integration section).

Why Offline-First AI?

Let me make a case for why you'd want to run an LLM locally on a device in the first place.

The most obvious answer is connectivity. I don't know about the rest of the world, but where I come from, mobile phone users live with unreliable, slow, and expensive internet. If your AI-powered feature requires a connection to a cloud API, you've excluded all those people by default for the most part. But guess what? connectivity is actually the least interesting reason.

Here are some reasons I think matter more:

Privacy. Every message a user sends to a cloud API leaves the device. With a local model, the data never leaves. For applications that deal with sensitive information like health, education, finance, legal stuff, this is particularly important. It is a trust-building architectural choice.

Cost. Cloud AI APIs are not free. The cost increases as you scale. A local model however incurs zero cost. You don't even have to pay for the compute to produce or fine-tune a decent-sized model (that's if you don't want a pre-trained model). Considering the recent news about a company's $500M AI bill, I think you'd find this reason very compelling 😂

Ownership. Your app will work on a plane, in a village school, underground, anywhere.

Independence. You're not at the mercy of OpenAI's pricing changes, Claude's rate limits, or changes in policies. The model runs on your terms.

I believe offline-first AI is the future for apps. The models are getting smaller and better at an impressive rate, and it can only get better.

Part 1: Fine-Tuning a Model with QLoRA and Unsloth

Again, you can skip this section if you have a pre-trained model and you just want integration. But if you want to customise a model's behaviour, teach it a specific domain, or adapt it to a particular style or language, fine-tuning is how you do it.

What Is Fine-Tuning Exactly?

A large language model in its base form has been trained on pretty much all the text on the internet. It knows a lot of general things. What it doesn't know is your specific context i.e. the domain-specific WAY you want it to respond, the format you want its answers in, or knowledge about specialised areas that may have been very little in its training data.

If you want a formal definition, Fine-tuning is the process of taking a pre-trained model and continuing to train it on a smaller, more focused dataset that's specific to your needs. You're not training from scratch since that would require a very large computing power, you're just nudging the model in a specific direction.

The catch is that fine-tuning a model requires quite a bit of memory. A 7B (7 billion) parameter model in full precision (no quantization) can't even fit on a standard GPU, let alone be trained on one. This is where QLoRA comes in.

QLoRA: The Magical Shortcut

QLoRA (Quantised Low-Rank Adaptation) is the reason fine-tuning became accessible to people like me and probably you. Here's a brief overview of what it does:

• The model weights are quantised i.e. compressed to 4-bit precision. This drastically reduces memory usage.

• Instead of updating all the model's weights during training, LoRA adds small trainable "adapter" matrices alongside the frozen original weights, so you're training a tiny fraction of the total parameters.

• This allows us fine-tune any model up to about 7B parameters on a FREE Google Colab T4 GPU.

That's amazing isn't it? Something that would have required expensive hardware a few years ago can now be done in a browser tab for free.

The Tool: Unsloth

Unsloth is the library that makes QLoRA training fast and practical. It wraps Hugging Face's transformers and trl libraries with optimised kernels, makes the model setup nearly one-liners, and is actively maintained. I used it for everything in this section.

Setting Up: Dataset Preparation

The most challenging part of fine-tuning is not the training, it's getting your dataset right. This surprised me. I thought training was the hard part. It isn't. The data is.

I learned a few important things in this aspect:

Your data determines your model. If your training examples are low quality, inconsistent, or too narrow, your model will mirror that. It's basically garbage in, garbage out. Sometimes, the garbage is even subtle. The model will happily train on it and produce confident-sounding but wrong and/or unnatural output.

You need more data than you think. A few hundred examples will make the model learn the format you want, but it won't meaningfully change its underlying knowledge or reasoning capability. For real domain adaptation, you need thousands of diverse examples.

Format consistency matters. Every example in your dataset has to follow the same instruction-input-output structure. If it's inconsistent, the model will produce inconsistent outputs.

Curriculum order matters too. When you're combining multiple datasets, the order you train on them matters. You should train on simpler, more general examples first before moving to harder or more specific ones.

This is how I structured a dataset combining multiple sources. Regardless of the domain you're working in, the pattern is pretty much the same:

import re

CANONICAL_INSTRUCTION = "Solve the following mathematics problem."

def clean_text(text):
    text = re.sub(r'\s+', ' ', text).strip()
    return text

def format_canonical(question, solution):
    return {
        "instruction": CANONICAL_INSTRUCTION,
        "input": clean_text(question),
        "output": clean_text(solution)
    }

The idea here is a canonical (or standard) format: every example has an instruction, an input, and an output. This consistency is what the model will learn to expect and will replicate when processing an actual query (at inference time).

For dataset sources, personally, I pulled from:

• Public HuggingFace datasets - there are thousands of datasets there, covering most domains. For anything general purpose, there's almost surely a dataset someone already prepared.

• Custom documents (PDFs and Word files) - for domain-specific content that isn't in any public dataset. I extracted text from these using pymupdf and python-docx.

Here's the PDF/document extraction approach:

import fitz
from docx import Document

def extract_text_from_pdf(pdf_path):
    doc = fitz.open(pdf_path)
    text_blocks = []
    for page in doc:
        text = page.get_text()
        if text.strip():
            text_blocks.append(text)
    doc.close()
    return text_blocks

def extract_text_from_docx(docx_path):
    doc = Document(docx_path)
    text_blocks = []
    for para in doc.paragraphs:
        text = para.text.strip()
        if not text:
            continue
        if para.style.name.startswith('Heading'):
            text_blocks.append(f"\n## {text}\n")
        else:
            text_blocks.append(text)
    return text_blocks

After extracting, I filtered through it for quality, skipping blocks that were too short, had a low volume of actual letters (which indicated garbled formula extraction since I was working in the mathematics domain), or had long strings of special characters:

def filter_quality(text):
    if len(text) < 100:
        return False
    alpha_ratio = sum(1 for c in text if c.isalpha()) / max(len(text), 1)
    if alpha_ratio < 0.35:
        return False
    if re.search(r'[^a-zA-Z0-9\s]{10,}', text):
        return False
    return True

For ordering the curriculum, I combined datasets in a special sequence - starting from general reasoning examples, moving to more structured problem-solving, and ending with domain-specific material as the final stage:

curriculum_data = []

# Stage 1: General reasoning (30k examples)
curriculum_data.extend(general_samples)

# Stage 2: More structured QA (50k examples)
curriculum_data.extend(structured_samples)

# Stage 3: Domain-specific content (~25% of total, upsampled)
# Added last so the model adapts to domain style without losing general capability
curriculum_data.extend(domain_samples)

# DO NOT SHUFFLE — preserve curriculum order

One important note: I did not shuffle the combined dataset. The curriculum order was clearly intentional. Shuffling would have mixed the stages and ruined the progressive learning structure.

Loading the Model and Applying LoRA

Once you have your dataset ready, the actual training setup with Unsloth is quite direct:

import torch
from unsloth import FastLanguageModel

MODEL_NAME = "unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME,
    max_seq_length=1024,
    load_in_4bit=True,
    dtype=None,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    use_gradient_checkpointing="unsloth",
)

The r parameter (LoRA rank) controls how many trainable parameters you're adding. A higher rank gives it more capacity to adapt, but also increases the risk of overfitting on the dataset. Setting r=16 is considered a generally reasonable starting point for most tasks. I used r=32 for a more complex domain that needed deeper reasoning, but for a language or style adaptation task, r=16 is likely enough.

Formatting Examples for Training

The tokenizer takes text, not a dictionary. So I converted each dataset example into a single formatted string:

def format_example(example):
    prompt = (
        "### Instruction:\n"
        f"{example['instruction']}\n\n"
        "### Response:\n"
        f"{example['output']}"
        f"{tokenizer.eos_token}"
    )
    return {"text": prompt}

dataset = dataset.map(format_example, remove_columns=dataset.column_names)

The eos_token at the end is very important, as it teaches the model when to stop generating. Without it, the model will likely keep generating past the end of a reasonable response.

For examples with both an instruction and a separate input:

def format_example_with_input(example):
    if example["input"]:
        prompt = (
            f"### Problem:\n{example['input']}\n\n"
            f"### Instruction:\n{example['instruction']}\n\n"
            f"### Solution:\n{example['output']}"
            f"{tokenizer.eos_token}"
        )
    else:
        prompt = (
            f"### Instruction:\n{example['instruction']}\n\n"
            f"### Response:\n{example['output']}"
            f"{tokenizer.eos_token}"
        )
    return {"text": prompt}

Training

from transformers import TrainingArguments
from trl import SFTTrainer

training_args = TrainingArguments(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,   # Effective batch size = 16
    warmup_steps=50,
    max_steps=1000,
    learning_rate=2e-4,
    fp16=not torch.cuda.is_bf16_supported(),
    bf16=torch.cuda.is_bf16_supported(),
    logging_steps=10,
    save_steps=100,
    output_dir="outputs",
    optim="adamw_8bit",
    weight_decay=0.01,
    lr_scheduler_type="cosine",
    seed=42,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=1024,
    args=training_args,
)

trainer.train()

Let me explain the hyperparameters:

• Gradient accumulation this simulates a larger batch size without needing more GPU memory. With per_device_train_batch_size=2 and gradient_accumulation_steps=8, you're basically training with a batch size of 16.

• Cosine learning rate schedule gradually reduces the learning rate, which generally produces a smoother convergence than keeping it constant.

• max_steps instead of epochs: for large datasets, it's generally more practical to set a step limit than train for full epochs.

After training, we save the model:

model.save_pretrained("final_model")
tokenizer.save_pretrained("final_model")

Exporting to GGUF

This step connects the fine-tuning to integration. There are several formats you can save your model in. GGUF is the file format used by llama.cpp and any library built on top of it. It supports quantization at export time, so the full-precision fine-tuned model can be compressed to a fraction of the size without losing much quality, and size is a really big deal when running a model on device.

Unsloth helps like so:

model.save_pretrained_gguf(
    "gguf_output",
    tokenizer,
    quantization_method="q4_k_m"
)

q4_k_m is the quantization type I recommend for mobile. It compresses weights to 4-bit using k-means clustering, with slightly higher precision retained for the most important layers. It produces a model that is about 4 to 6 times smaller than the original float-16 version, with performance that is typically within a few percent of full precision on most tasks.

Once you have the .gguf file, you are ready. And if you don't want to fine-tune at all, many models are freely available in GGUF format on HuggingFace, just search for the model you want (append "GGUF" in your search to get tailored results).

Before we move on from finetuning....

Before we move on from finetuning, let me talk a bit about some limitations to finetuning.

I previously explained that fine-tuning gives you:

• A model that reliably follows a specific output format, a particular tone, language style or persona. It also gives you improved performance on a specific domain that probably wasn't well represented in the base model's training (assuming good data).

But here are some limitations to finetuning and what finetuning can cost you if not done well:

• Catastrophic forgetting. If your training data is too narrow, the model can lose general capability. Some of the models I initially fine-tuned learned my specific domain format so aggressively that they started responding poorly to basic English questions. This is a real risk with small models and small datasets.

• Bad data = bad finetune. A bad dataset will produce mediocre results and degrade a model that was previously good.

• Finetuning doesn't give you new facts. Fine-tuning teaches the model how to respond, not what to know. If what you want is for a model to know facts it wasn't trained on at all, RAG (Retrieval Augmented Generation) may give you better results than fine-tuning.

• Small models hit a ceiling. A 1B parameter model fine-tuned on domain data will still struggle with complex multi step reasoning as all 1B models do. Fine-tuning cannot quite change the model's capability. Higer parameter models always do better than smaller ones.

From my experience, I found that a properly crafted system prompt (or RAG at most) with a good base model achieved similar results to fine-tuning with none of the downsides. Fine-tuning is mostly worth it when you have a very specific format requirement, a language or domain that was under-represented in the base model's training, or if you're optimising for size (a fine-tuned smaller model can sometimes match a larger base model on a specific task as general knowledge is essentially trimmed off in favour of the emphasised domain).

Part 2: Integrating a Local LLM in Flutter (Android)

Now to the more fun (harder maybe 😅) part!

My First Attempt: Native FFI (didn't work)

I tried integrating llama.cpp into Flutter the standard way. I compiled llama.cpp for Android ARM64 using the Android NDK, wrote a C++ bridge, and then wrote Dart FFI bindings to call into it.

I had C++ files in a native/ directory, Dart binding files, a shell script that called CMake targeting the NDK toolchain for arm64-v8a and compiled a .so file which I bundled into the APK.

The Dart code looked like this:

// llama_bindings.dart
class LlamaBindings {
  late final DynamicLibrary _lib;

  late final int Function(Pointer<Void>) _llamaDartInit;
  late final int Function(Pointer<Utf8>) _llamaInit;
  late final Pointer<Utf8> Function(Pointer<Utf8>) _llamaGenerate;

  void init(String modelPath) {
    final Pointer<Utf8> cPath = modelPath.toNativeUtf8();
    try {
      final int rc = _llamaInit(cPath);
      if (rc != 0) {
        throw Exception('llama_init failed (code=$rc)');
      }
    } finally {
      malloc.free(cPath);
    }
  }
}

And the C++ bridge had to handle Dart ports, the isolates, and the native threading just to stream tokens back to Flutter:

// llama_bridge.cpp
extern "C" {
int32_t llama_dart_init(void *init_data) {
    void *init_sym = dlsym(RTLD_DEFAULT, "Dart_InitializeApiDL");
    // ... 325 more lines of this
}

My git commit message on October 30th after a full day of back and forth was:

"lots of failed attempts. llama model initialization keeps failing"

The model would compile, the library would load but the model initialisation would fail every time. I never got it to work.

I figured bridging native C++ into Flutter via FFI is not trivial. It was my first time working with native FFI. Memory management, ABI compatibility, initialising the Dart API correctly from native code, and handling the threading model correctly are things that can go wrong in many ways that are difficult to debug. Except you have a lot experience working with c++ and integrating it into Flutter, you probably shouldn't take this route. P.S.: I later got the hang of bridging native C++ into Flutter, but I did it for a Flutter Windows app I made. I'll write about it in my next article.

Success: Finding the Right Flutter Packages

I'm not sure why it didn't occur to me to search for existing packages initially (I actually just wanted to figure it out myself 😅), but it turned out there were a few Flutter packages that wrapped the native complexity. Depending on the model format you're working with, there are two excellent options I ended up using: flutter_gemma and llama_flutter_android.

I used both of them extensively in my work, as they cater to different model formats and offer different levels of control.

Working with LiteRT Models: flutter_gemma

If you're working with LiteRT (formerly TensorFlow Lite) models, flutter_gemma is a fantastic, straightforward package built directly on Google's LiteRT runtime. It works with .bin, .tflite, .task and .litertlm model files

Add to pubspec.yaml:

dependencies:
  flutter_gemma: ^0.11.12

Using it was also very intuitive:

final model = InferenceModel();
await model.init();

final chat = await model.createChatSession(
  systemPrompt: "You are a helpful assistant.",
);

await for (final token in chat.sendMessageStream("Hello!")) {
  // stream tokens to UI
}

This package handles model loading internally and abstracts away most of the resource management, which makes it easier to get up and running quickly. It's a great choice if the model you want to run is available in one of its supported formats which I earlier mentioned.

Working with GGUF Models: llama_flutter_android

While flutter_gemma was perfect for LiteRT models, most models you will find or fine-tune yourself (as demonstrated in part 1) will actually be in GGUF format. Many of the community fine-tunes and newer open weights on HuggingFace are in the GGUF format.

For this, I found llama_flutter_android to be perfect. This package uses llama.cpp under the hood but packages everything needed- the native library, the GGUF loading, and the generation logic - as a proper Flutter plugin. So no NDK, CMake or C++ bridge problems anymore, which were the things I struggled with when trying to work with llama.cpp directly.

Now, while flutter_gemma does allow you to configure basic sampling settings (like temperature, topK, and randomSeed), llama_flutter_android allows for more control over both the generation process and hardware execution. This distinction became important as I started optimizing the app for performance on lowerend devices.

Let me walk you through the detailed process of using llama_flutter_android for GGUF models:

1. Add the Dependency to pubspec per usual

dependencies:
  llama_flutter_android: ^0.1.1

Don't forget to run flutter pub get.

2. Add the Model to Your Project

You have two options: bundle the model as an asset, or download it (once throughout the lifetime of the app) at runtime. I recomment downloading it at runtime, but your use case may require bundling which is okay.

For bundling:

# pubspec.yaml
flutter:
  assets:
    - assets/models/your_model.gguf

Remember, AI models are large (even the most quantized ones with <500M parameters can be up to 300MB-600MB). It will bump up your APK size. If this is a concern for you, you should host the model (you can create a HuggingFace account and upload your fine-tuned model there securely) and download at first launch instead.

For download-on-first-launch:

Future<String> downloadModel() async {
  final dir = await getApplicationDocumentsDirectory();
  final modelPath = '${dir.path}/your_model.gguf';

  if (File(modelPath).existsSync()) return modelPath;

  final response = await http.get(Uri.parse(MODEL_URL));
  await File(modelPath).writeAsBytes(response.bodyBytes);
  return modelPath;
}

If you chose to bundle the model as an asset, you'll need to copy it out to a writable directory first before loading (Flutter assets are read-only):

Future<String> extractModelFromAssets() async {
  final dir = await getApplicationDocumentsDirectory();
  final modelPath = '${dir.path}/your_model.gguf';

  if (File(modelPath).existsSync()) return modelPath;

  final data = await rootBundle.load('assets/models/your_model.gguf');
  await File(modelPath).writeAsBytes(
    data.buffer.asUint8List(data.offsetInBytes, data.lengthInBytes),
  );

  return modelPath;
}

3. Next, build the Service class

I wrapped everything in a singleton service class like this:

import 'dart:async';
import 'dart:io';

import 'package:flutter/foundation.dart';
import 'package:llama_flutter_android/llama_flutter_android.dart';

class LlamaService {
  static final LlamaService _instance = LlamaService._();
  LlamaController? _controller;
  String? _currentModelPath;

  factory LlamaService() => _instance;
  LlamaService._();

  LlamaController get _assertController {
    final existing = _controller;
    if (existing != null) return existing;
    final recreated = LlamaController();
    _controller = recreated;
    return recreated;
  }

  Future<void> loadModel(String filePath) async {
    _currentModelPath = filePath;

    final cpuCores = Platform.numberOfProcessors;
    final ramGB = await _getTotalRamGB();

    // Adapt context size and threads to available device memory
    int threads;
    int contextSize;

    if (ramGB >= 12) {
      contextSize = 6144;
      threads = (cpuCores ~/ 2).clamp(4, 8);
    } else if (ramGB >= 8) {
      contextSize = 4096;
      threads = (cpuCores ~/ 2).clamp(3, 6);
    } else if (ramGB >= 6) {
      contextSize = 3072;
      threads = (cpuCores ~/ 2).clamp(2, 4);
    } else {
      contextSize = 2048;
      threads = 2;
    }

    debugPrint('Loading model: threads=$threads, context=$contextSize, ram=${ramGB.toStringAsFixed(1)}GB');

    try {
      await _assertController.loadModel(
        modelPath: filePath,
        threads: threads,
        contextSize: contextSize,
      );
    } catch (e) {
      // Retry with conservative settings if adaptive settings fail
      if (contextSize > 2048) {
        debugPrint('Retrying with conservative settings...');
        await _assertController.loadModel(
          modelPath: filePath,
          threads: 2,
          contextSize: 2048,
        );
      } else {
        rethrow;
      }
    }
  }

  Stream<String> chat(List<ChatMessage> messages) {
    return _assertController.generateChat(
      messages: messages,
      template: null,
      maxTokens: 1024,
      temperature: 0.7,
      topK: 40,
      topP: 0.9,
    );
  }

  Future<void> clearContext() async => _assertController.clearContext();

  Future<void> dispose() async {
    await _controller?.dispose();
    _controller = null;
  }

  Future<double> _getTotalRamGB() async {
    try {
      final memInfo = await File('/proc/meminfo').readAsString();
      final match = RegExp(r'MemTotal:\s+(\d+) kB').firstMatch(memInfo);
      if (match != null) {
        final kb = int.parse(match.group(1)!);
        return kb / (1024 * 1024);
      }
    } catch (_) {}
    return 4.0; // safe default
  }
}

Let me explain a few key things from the code:

What (or why) /proc/meminfo? Android's Platform.numberOfProcessors returns CPU core count, but there's no standard Flutter API for available RAM. On Android, /proc/meminfo is a virtual file that the kernel populates with current memory info. Reading it allows us to dynamically set the context window based on the actual device the app is running on, instead of just hardcoding a value that either runs out of memory on low-end phones or under-utilizes high-end ones.

Why adapt context size and threads? Text generation (or inference) with large language models uses the CPU and so, running more threads can make it faster, but it can only do so up to a point. After that, too many threads start competing for the same CPU power, which begins to slow things down. Using cpuCores ~/ 2 is a safe starting point as it gives a good speed boost while still leaving enough CPU for the app UI and other background work. We clamp it based on available RAM because more threads also means more memory pressure.

Why the fallback retry? Well, model loading on devices sometimes fails at higher context sizes due to failure to allocate adequate resources rather than any other other fundamental issue. Retrying at lower tokens can succeed where higher ones failed. It's just a fail safe. Better to degrade than crash, isn't it?.

Now to generation parameters: Tuning the model generation parameters influences how the model chooses its next tokens and manages resources. A badly tuned model can effectively render it useless. I played around with the parameters extensively to find the right balance. I'll explain them and give my recommendations:

• maxTokens: This defines the limit on how many tokens the model can generate in one turn. My Recommendation: Keep this between 512 and 1024 for mobile apps. Anything higher risks draining the battery and holding up the device for too long.

• temperature: This controls randomness. A lower value (say 0.6) makes the model more direct or deterministic, while a higher value (say the max, 1.0) makes it more creative but prone to hallucination. My Recommendation: You may start at 0.7 for general chat. If you need strict formatting and reliable responses, drop it lower.

• topK: Restricts the model's token choices to only the top K most likely next tokens. My Recommendation: 40 is the standard default, that works fine for pretty much every task.

• topP: (also called nucleus sampling) This restricts the model's choices to a smaller group of tokens whose combined probability exceeds the value P. My Recommendation: 0.9 works well with a 0.7ish temperature.

• minP: Sets a relative minimum probability threshold. Tokens that fall below a the specified percentage of the most likely token's probability are jettisoned. This can be instrumental in ensuring quality. My Recommendation: 0.05 to 0.1 is usually perfect.

• repeatPenalty: This parameter penalizes the model for repeating the same words or phrases. If you discover your model repeats its output, this could solve that. Recommendation: 1.1 to 1.2. Smaller models usually have this looping or repeat issue, so this is crucial.

• penalizeNewline: This specifically penalizes the generation of newline characters (\n) to prevent formatting loops or overly spaced out text. My Recommendation: Set to true for general text tasks, but if your model needs to generate code or formatted lists, set to false.

• Thread count: Already explained this. It determines ow many CPU threads to allocate to generation. My Recommendation: Make it adaptive based on device cores (as shown in the code).

• Context size: The maximum number of tokens (both prompt and output) that the model can hold in memory. This directly determines how much RAM is required. My Recommendation: 2048 for low-end devices, 4096 for 8GB RAM devices, 6144 for 12GB+ RAM devices.

4. Handling Chat Format

It turns out different models actually expect prompts in different formats. The generateChat method handles this automatically for most models using their built-in chat template. However, some models may need manual formatting.

For example, Gemma 3's format uses special turn tokens:

String _formatGemma3Prompt(List<ChatMessage> messages) {
  final buffer = StringBuffer();
  String? systemText;

  for (final msg in messages) {
    if (msg.role == 'system') {
      systemText = msg.content;
    }
  }

  for (final msg in messages) {
    if (msg.role == 'system') continue;

    buffer.write('<start_of_turn>');
    if (msg.role == 'user') {
      buffer.write('user
');
      if (systemText != null) {
        buffer.write('$systemText

');
        systemText = null; // inject once only
      }
      buffer.write(msg.content);
      buffer.write('<end_of_turn>
');
    } else if (msg.role == 'assistant') {
      buffer.write('model
');
      buffer.write(msg.content);
      buffer.write('<end_of_turn>
');
    }
  }

  buffer.write('<start_of_turn>model
');
  return buffer.toString();
}

The important thing is that the system prompt is inserted into the first user turn, not as a separate turn. This is how Gemma 3 was trained to expect it. Getting this wrong can produce garbled output as the model will receive your instructions but won't follow them reliably.

If you're using a different model, check its model card on HuggingFace for the expected chat format. Most instruct models document this properly.

5. Using It in Your UI

Once you're done with the service, wiring it into a Cubit or Bloc is straightforward:

class ChatCubit extends Cubit<ChatState> {
  final LlamaService _llama = LlamaService();
  final List<ChatMessage> _history = [];

  ChatCubit() : super(ChatInitial());

  Future<void> sendMessage(String userText) async {
    _history.add(ChatMessage(role: 'user', content: userText));
    emit(ChatGenerating(messages: List.from(_history)));

    final responseBuffer = StringBuffer();

    _llama.chat(_history).listen(
      (token) {
        responseBuffer.write(token);
        emit(ChatStreaming(
          messages: List.from(_history),
          currentResponse: responseBuffer.toString(),
        ));
      },
      onDone: () {
        _history.add(ChatMessage(
          role: 'assistant',
          content: responseBuffer.toString(),
        ));
        emit(ChatIdle(messages: List.from(_history)));
      },
      onError: (e) => emit(ChatError(e.toString())),
    );
  }
}

The streaming approach is important as LLM generation produces tokens one by one unlike an HTTP response that arrives all at once. Streaming the tokens to the UI as they arrive makes the app feel responsive and natural. If you wait for the full response before displaying anything, your users may quickly get bored staring at a loading spinner for several seconds.

6. Initialisation and Model Loading

Load the model once ideally when the feature is first needed or maybe just at app startup:

void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  final modelPath = await extractModelFromAssets();
  await LlamaService().loadModel(modelPath);

  runApp(const MyApp());
}

P.S.: If you're loading at startup, do it before runApp so that the UI doesn't appear before the model is ready. Model loading for a quantized small model can take 2–5 seconds on a mid-range device.

You're all set!

Remember:

The model generation speed is primarily determined by:

• Model size -- smaller = faster

• Quantization level -- lower bit = faster, lower quality

• Context size -- larger context = more memory consumed (slower)

• Device -- RAM and CPU core count, can be properly handled per the adaptive logic above

Top level Summary

1. Prepare your dataset - pull from HuggingFace or extract from your own documents, clean it, format it by a specific standard, and order it.

2. Fine-tune with Unsloth - load a quantized base model, apply QLoRA, train with SFTTrainer.

3. Export to GGUF - model.save_pretrained_gguf("output", tokenizer, quantization_method="q4_k_m").

4. Bundle or download the GGUF in your Flutter project.

5. Add llama_flutter_android to pubspec.yaml.

6. Build a service using LlamaController, with adaptive threading and context sizing.

7. Connect it to your UI via your state management, stream the tokens as they arrive.

If you're skipping fine-tuning, just start from step 4 with a pre-quantized GGUF from HuggingFace.

Well, that's it.

Have questions, or want to go deeper on any part of this? Find me on LinkedIn. Watch out for my next article which is likely going to be on running AI models (LLMs & VLMs) on Flutter Windows Applications (that's a lot more fun than this one)