Every generative AI product roadmap eventually runs into the same question: how big does the model actually need to be? The instinct is often to reach for the largest model that will fit — more parameters feels safer, like buying more headroom than you think you’ll need. At the edge, that instinct is expensive in a way it isn’t in the cloud. Every extra billion parameters costs real memory bandwidth, real power, real thermal budget, and real BOM dollars — on hardware that has to run fanless, on a few watts, inside an enclosure you can’t redesign every six months.
The good news: most edge AI products don’t need a large model at all. 2026’s shift toward small, purpose-built language models — sometimes called SLMs or “micro LLMs” — means the right question usually isn’t “which frontier model can we fit,” it’s “how small a model can we get away with for this specific task.” Getting that answer right determines your entire hardware spec.
Model Size Is a Design Decision, Not a Default
A model’s parameter count should be driven by the task, not by what’s trending. A few practical anchors:
- Sub-1B models handle narrow, well-defined tasks well: keyword spotting, short-form text generation from a template, simple classification-adjacent language tasks, basic summarization of structured data.
- 1.5B–3B models are the current sweet spot for genuinely useful on-device assistants — capable of multi-turn interaction, reasonable reasoning on everyday tasks, and instruction-following, without needing datacenter-class memory bandwidth.
- 7B-class models are where you start getting closer to general-purpose conversational quality and more complex reasoning — but at a real cost in memory, power, and hardware complexity that most fixed-function edge products don’t actually need.
The costly mistake isn’t picking a model that’s too small — that failure mode is obvious immediately, in testing, and gets caught early. The costly mistake is over-provisioning: shipping 7B-class hardware into a product that only ever needed a 1B model, and eating that BOM cost across an entire production run.
Why the Constraint Is Memory Bandwidth, Not Just Model Size
As covered in more detail in Why TOPS Is Not Enough in Edge AI, the number that actually determines whether a model runs usably at the edge isn’t parameter count or peak TOPS — it’s sustained memory bandwidth. Every token generated requires reading the model’s full weight set from memory; a larger model means more bytes to move per token, and if the memory bus can’t keep up, a fast compute core just sits idle waiting for data.
This is why model size and hardware tier are so tightly coupled. A platform with excellent peak compute but modest memory bandwidth will comfortably run a 0.5B model and then fall off a cliff on a 3B one — not because the compute ran out, but because the memory subsystem did.
Matching Model Size to Hardware Tier
Here’s how that plays out across real edge AI hardware, using Geniatech’s own SoM and accelerator lineup as a concrete reference:
Tier 1 — Sub-1B Models: General SoC with Onboard NPU
For lightweight, well-optimized tasks, a general-purpose ARM SoC’s onboard NPU is usually sufficient — no dedicated accelerator required:
- RK3588 (Rockchip) — 6 TOPS onboard NPU
- i.MX95 (NXP) — 2 TOPS onboard NPU (eIQ Neutron), expandable via M.2 if needs grow
- QCS6490 (Qualcomm) — 12 TOPS AI Engine
- Renesas RZ/V2N — DRP-AI3, up to 15 TOPS
This tier fits products where language generation is one feature among several — not the product’s core value proposition. Retail signage generating short promotional copy, an industrial gateway summarizing sensor status into a short alert, a device doing basic intent classification on voice input. The onboard NPU is essentially free capability layered onto a chip you likely already needed for general compute.
Tier 2 — 1.5B to 3B Models: Dedicated GenAI Accelerator
Once you need genuinely capable on-device language interaction — multi-turn conversation, more nuanced instruction-following — a general SoC’s onboard NPU typically isn’t enough, and this is where a dedicated generative AI accelerator earns its place on the BOM:
- Hailo-10H — Hailo’s second-generation architecture, purpose-built for generative AI with a direct DDR interface (unlike the CNN-optimized Hailo-8), delivering up to 40 TOPS at INT4. Hailo has demonstrated ~2B-parameter language and multimodal models running with sub-1-second time-to-first-token and throughput above 10 tokens/second — a genuinely usable interactive experience at a few watts.
This is the tier where it matters to be precise about which accelerator you’re choosing: not every Hailo product is built for this. Hailo-8 remains an excellent, power-efficient choice for CNN-based vision — object detection, classification — but its architecture wasn’t designed around the memory-bandwidth demands of Transformer-based generation, and it’s not the right tool for this tier. Hailo-10H is a different, newer architecture built specifically to solve that problem.
Tier 3 — 7B-Class Models: High-Bandwidth LLM Accelerators
For products that genuinely need larger-model reasoning quality — more complex multi-step instructions, broader world knowledge, longer context windows — you’re into accelerator designs built specifically around memory bandwidth at scale:
- RK1828 — uses 3D-stacked DRAM specifically to solve the memory-bandwidth bottleneck that limits smaller accelerators; capable of running 7B-class LLMs locally with INT4 quantization.
- NXP Ara-240 — offers a large 16GB onboard memory capacity, giving headroom for bigger models or longer context windows without the same aggressive quantization trade-offs.
This tier carries real cost and power implications, and it’s worth being honest about that trade-off before committing to it: a 7B-class deployment is meaningfully more expensive, per unit, than tiers 1 and 2. It’s the right call when the product’s core value genuinely depends on larger-model reasoning quality — not by default, and not because a bigger number felt safer during spec’ing.
A Practical Way to Decide
Work backward from the task, not forward from the model catalog:
- Write down the exact task the model needs to do — not “conversational AI,” but the specific input, the specific output, the specific failure mode you can’t tolerate.
- Test the smallest model that could plausibly do it, quantized, on your actual target hardware — not a cloud API standing in for what the edge model will eventually need to do.
- Only move up a tier when the smaller model demonstrably fails the task on your own evaluation data — not on a general benchmark that may not reflect your use case.
- Re-check memory bandwidth, not just parameter count, against the platform you’re evaluating — a model that “should” fit on paper can still perform poorly if the accelerator’s real sustained bandwidth doesn’t support it at your target token rate.
Most teams skip step 3 and jump straight to whatever model size feels safely capable, which is exactly how a product ends up shipping 7B-class hardware for a job a 1B model would have handled. The disciplined version of this process costs a few extra days of evaluation; the undisciplined version costs the BOM difference multiplied across your entire production run.
Getting This Right Before You Commit to Hardware
Model size and hardware tier are a package deal — choosing one without validating it against the other is how projects end up over-provisioned or, worse, under-provisioned late in development. If you’re scoping a generative AI feature and aren’t yet sure which tier your task actually needs, Geniatech’s engineering team can help benchmark your specific model against real hardware — from onboard-NPU SoMs through Hailo-10H and RK1828-class accelerators — before you lock in a production spec.