Every generative AI product roadmap eventually runs into the same question: how big does the model actually need to be? The instinct is often to reach for the largest model that will fit — more parameters feels safer, like buying more headroom than you think you’ll need. At the edge, that instinct is expensive in a way it isn’t in the cloud. Every extra billion parameters costs real memory bandwidth, real power, real thermal budget, and real BOM dollars — on hardware that has to run fanless, on a few watts, inside an enclosure you can’t redesign every six months.
Small Language Models (SLMs) are language models — typically under 8 billion parameters, often well under 3 billion — distilled or quantized specifically to run on local hardware instead of a cloud server. They’re not scaled-down chatbots trying to do everything a frontier model does; they’re purpose-built for narrow, well-defined tasks: generating a short piece of text, classifying intent, summarizing structured data, following a fixed instruction format. That narrower scope is exactly what makes them practical to run on a few watts of edge silicon.
The good news: most edge AI products don’t need a large model at all. 2026’s shift toward SLMs — sometimes called “micro LLMs” — means the right question usually isn’t “which frontier model can we fit,” it’s “how small a model can we get away with for this specific task.” Getting that answer right determines your entire hardware spec.
Model Size Is a Design Decision, Not a Default
A model’s parameter count should be driven by the task, not by what’s trending. A few practical anchors:
- Sub-1B models handle narrow, well-defined tasks well: keyword spotting, short-form text generation from a template, simple classification-adjacent language tasks, basic summarization of structured data.
- 1.5B–3B models are the current sweet spot for genuinely useful on-device assistants — capable of multi-turn interaction, reasonable reasoning on everyday tasks, and instruction-following, without needing datacenter-class memory bandwidth.
- 7B-class models are where you start getting closer to general-purpose conversational quality and more complex reasoning — but at a real cost in memory, power, and hardware complexity that most fixed-function edge products don’t actually need.
The costly mistake isn’t picking a model that’s too small — that failure mode is obvious immediately, in testing, and gets caught early. The costly mistake is over-provisioning: shipping 7B-class hardware into a product that only ever needed a 1B model, and eating that BOM cost across an entire production run.
Why the Constraint Is Memory Bandwidth, Not Just Model Size
As covered in more detail in Beyond the Datasheet: How to Choose the Right Hardware for On-Device LLM and Transformer Inference at the Edge, the number that actually determines whether a model runs usably at the edge isn’t parameter count or peak TOPS — it’s sustained memory bandwidth. Every token generated requires reading the model’s full weight set from memory; a larger model means more bytes to move per token, and if the memory bus can’t keep up, a fast compute core just sits idle waiting for data.
This is why model size and hardware tier are so tightly coupled. A platform with excellent peak compute but modest memory bandwidth will comfortably run a 0.5B model and then fall off a cliff on a 3B one — not because the compute ran out, but because the memory subsystem did.
Matching Model Size to Hardware Tier
Here’s how that plays out across real edge AI hardware, using Geniatech’s own SoM and accelerator lineup as a concrete reference:
| Tier | Model Size | Primary Hardware | Also Worth Considering | Typical Models |
|---|---|---|---|---|
| 1 | Sub-1B | RK3588 (6 TOPS onboard NPU) | i.MX95, QCS6490, Renesas RZ/V2N | Qwen2.5-0.5B, Llama 3.2-1B |
| 2 | 1.5B–3B | Hailo-10H (40 TOPS INT4, dedicated GenAI) | RK1820 (2.5GB DRAM, 20 TOPS INT8) | Qwen2.5-1.5B/3B, Qwen3 |
| 3 | 7B-class | NXP Ara-240 (16GB memory) | RK1828 (5GB DRAM, 20 TOPS INT8) | Qwen2.5-7B, Llama 3-8B family |
Tier 1 — Sub-1B Models: General SoC with Onboard NPU
For lightweight, well-optimized tasks, a general-purpose ARM SoC’s onboard NPU is usually sufficient — no dedicated accelerator required.
Primary recommendation: RK3588 (Rockchip) — 6 TOPS onboard NPU, the most mature ecosystem and best-documented toolchain (RKNN-Toolkit2) of any option in this tier, at a strong price-to-performance ratio.
Also worth considering: i.MX95 (NXP) — 2 TOPS onboard NPU, expandable via M.2 if needs grow later; QCS6490 (Qualcomm) — 12 TOPS AI Engine, strong choice if you also need integrated cellular/wireless; Renesas RZ/V2N — DRP-AI3, up to 15 TOPS, a good fit where Renesas is already the incumbent platform.
Typical models: Qwen2.5-0.5B, Llama 3.2-1B, and similarly-sized distilled models — small enough to run comfortably within this tier’s memory and bandwidth budget. Rockchip’s own benchmarks even show a distilled 1.5B reasoning model (DeepSeek-R1-Distill-Qwen-1.5B) running on RK3588’s 6 TOPS NPU at roughly 15 tokens/sec decode — a useful reminder that these tier boundaries are directional, not hard cutoffs; the right test is always your specific model on your specific hardware.
This tier fits products where language generation is one feature among several — not the product’s core value proposition. Retail signage generating short promotional copy, an industrial gateway summarizing sensor status into a short alert, a device doing basic intent classification on voice input. The onboard NPU is essentially free capability layered onto a chip you likely already needed for general compute.
Tier 2 — 1.5B to 3B Models: Dedicated GenAI Accelerator
Once you need genuinely capable on-device language interaction — multi-turn conversation, more nuanced instruction-following — a general SoC’s onboard NPU typically isn’t enough, and this is where a dedicated generative AI accelerator earns its place on the BOM.
Primary recommendation: Hailo-10H — Hailo’s second-generation architecture, purpose-built for generative AI with a direct DDR interface (unlike the CNN-optimized Hailo-8), delivering up to 40 TOPS at INT4. Hailo has demonstrated ~2B-parameter language and multimodal models running with sub-1-second time-to-first-token and throughput above 10 tokens/second — a genuinely usable interactive experience at a few watts.
Also worth considering: RK1820 — Rockchip’s dedicated LLM/VLM accelerator, purpose-built for this exact parameter range (up to 3B), with 2.5GB of on-chip DRAM and 20 TOPS INT8. It uses 3D-stacked memory to push far past typical LPDDR bandwidth, and pairs directly with an RK3588 or RK3576 host over PCIe — a natural fit if you’re already standardizing on Rockchip SoMs elsewhere in your platform.
Typical models: Qwen2.5-1.5B/3B and Qwen3 family models are the most commonly deployed at this tier. Rockchip’s own published benchmarks across the RK1820/RK1828 lineup show 59–180 tokens/sec depending on model size and task, and Hailo has internally demonstrated ~2B-parameter language and multimodal models on Hailo-10H at similar interactive speeds.
This is the tier where it matters to be precise about which accelerator you’re choosing: not every Hailo product is built for this. Hailo-8 remains an excellent, power-efficient choice for CNN-based vision — object detection, classification — but its architecture wasn’t designed around the memory-bandwidth demands of Transformer-based generation, and it’s not the right tool for this tier. Hailo-10H and RK1820 are purpose-built for it instead.
Tier 3 — 7B-Class Models: High-Bandwidth LLM Accelerators
For products that genuinely need larger-model reasoning quality — more complex multi-step instructions, broader world knowledge, longer context windows — you’re into accelerator designs built specifically around memory bandwidth at scale.
Primary recommendation: NXP Ara-240 — offers a large 16GB onboard memory capacity, giving meaningful headroom for bigger models or longer context windows without the same aggressive quantization trade-offs smaller-memory accelerators require. The larger memory pool is particularly useful if your product needs to retain longer conversation history or process longer documents.
Also worth considering: RK1828 — the 7B-class counterpart to the RK1820 above, with 5GB of on-chip DRAM and the same 3D-stacked memory architecture, delivering up to 20 TOPS INT8. Rockchip’s own reference benchmarks show 7B-class models running at usable decode speeds, and — like the RK1820 — it drops onto an existing RK3588/RK3576 platform over PCIe without a carrier board redesign, making it the lower-friction upgrade path if you’re already on Rockchip.
Typical models: Qwen2.5-7B and Llama-3-8B-class models are the most common fit at this tier. The same RK1820/RK1828 benchmark set referenced above (59–180 tokens/sec) spans this range too, with actual throughput depending heavily on the specific model and quantization level chosen.
This tier carries real cost and power implications, and it’s worth being honest about that trade-off before committing to it: a 7B-class deployment is meaningfully more expensive, per unit, than tiers 1 and 2. It’s the right call when the product’s core value genuinely depends on larger-model reasoning quality — not by default, and not because a bigger number felt safer during spec’ing.
A Practical Way to Decide
Work backward from the task, not forward from the model catalog:
- Write down the exact task the model needs to do — not “conversational AI,” but the specific input, the specific output, the specific failure mode you can’t tolerate.
- Test the smallest model that could plausibly do it, quantized, on your actual target hardware — not a cloud API standing in for what the edge model will eventually need to do.
- Only move up a tier when the smaller model demonstrably fails the task on your own evaluation data — not on a general benchmark that may not reflect your use case.
- Re-check memory bandwidth, not just parameter count, against the platform you’re evaluating — a model that “should” fit on paper can still perform poorly if the accelerator’s real sustained bandwidth doesn’t support it at your target token rate.
Most teams skip step 3 and jump straight to whatever model size feels safely capable, which is exactly how a product ends up shipping 7B-class hardware for a job a 1B model would have handled. The disciplined version of this process costs a few extra days of evaluation; the undisciplined version costs the BOM difference multiplied across your entire production run.
FAQ
Do SLMs sacrifice a lot of accuracy compared to larger cloud-hosted models?
For narrow, well-defined tasks — the kind most edge products actually need — the gap is often small enough not to matter. SLMs lose ground mainly on broad world knowledge and complex multi-step reasoning, which is exactly why matching model size to a specific, well-scoped task (rather than expecting general-purpose chatbot performance) is the right framing at the edge.
Can I run a 7B parameter model directly on an RK3588’s onboard NPU, without a dedicated accelerator?
Not at a usable interactive speed. RK3588’s 6 TOPS NPU and standard LPDDR memory bandwidth are sized for sub-1B to roughly 1.5B-parameter models; getting acceptable 7B-class performance requires a dedicated high-bandwidth accelerator like RK1828, which uses 3D-stacked memory specifically to solve this bottleneck.
How do I know which tier my product actually needs?
Start with the smallest tier and test your actual task against it on real hardware. If a sub-1B model handles your specific use case — most narrow, well-defined tasks do — there’s no reason to move up. Only escalate a tier when you have evaluation data showing the smaller model genuinely fails the task, not because a larger model number feels safer.
Do all three tiers use the same software toolchain?
No — each accelerator has its own compiler and runtime (RKNN-Toolkit2 for Rockchip SoCs and RK182X accelerators, Hailo’s Dataflow Compiler for Hailo-10H, NXP’s eIQ toolkit for Ara-240). Software/operator support for your specific model is worth validating early, since it affects real-world performance as much as the underlying silicon does.
Getting This Right Before You Commit to Hardware
Model size and hardware tier are a package deal — choosing one without validating it against the other is how projects end up over-provisioned or, worse, under-provisioned late in development. If you’re scoping a generative AI feature and aren’t yet sure which tier your task actually needs, Geniatech’s engineering team can help benchmark your specific model against real hardware — from onboard-NPU SoMs through Hailo-10H and RK1828-class accelerators — before you lock in a production spec.