RK3588’s built-in 6 TOPS NPU is enough for most CNN-based vision workloads — object detection, classification, multi-stream analytics at moderate camera counts. You need an add-in M.2 accelerator when the workload exceeds that headroom: more camera streams than the onboard NPU can process in real time, larger or transformer-based models, or any on-device LLM/VLM inference, since RK3588’s NPU architecture wasn’t built for generative AI at usable speeds.
This is one of the most common — and most avoidable — mistakes in edge AI hardware selection. Teams either over-spec by adding an M.2 accelerator “just in case” when the onboard NPU would have handled the workload fine, or under-spec by assuming 6 TOPS covers everything, only to find frame rates collapsing once real camera counts and real models hit the board. Both mistakes cost money and time. This guide lays out how to tell which situation you’re actually in.
Quick answer
- Stick with the integrated NPU if your workload is CNN-based vision (YOLO-class detection, classification, segmentation) at a modest number of camera streams, and your models are already optimized for RKNN. RK3588’s 6 TOPS NPU handles this class of workload at sub-10W with no added BOM cost.
- Add an M.2 accelerator if you need more raw throughput than 6 TOPS INT8 provides (more streams, higher resolution, heavier models), or if the product needs to run an LLM, VLM, or other generative AI model — RK3588’s onboard NPU isn’t architected for that workload regardless of how the TOPS number is read.
What the integrated NPU actually handles well
RK3588’s NPU is rated at 6 TOPS at INT8 precision, split across three NPU cores, and supports INT4/INT8/INT16/FP16/BF16 data types through Rockchip’s RKNN toolchain. In real deployments, that translates to solid performance on CNN-based models: independent benchmarks report 50+ FPS on YOLOv5s and 200+ FPS on ResNet18 at INT8. One documented industrial deployment ran 16 channels of license-plate and vehicle detection at 18-20 FPS per channel, with the entire system drawing around 7.4W — a workload that would typically call for a discrete GPU or Jetson-class module on other platforms.
The pattern holds across a wide range of “detect and act” vision tasks: the onboard NPU is genuinely sufficient when the model is CNN-based, quantized for INT8, and the camera count stays in a moderate range. This is also the cheapest option by a wide margin — no added M.2 module, no added power draw, no added board space.
Where the integrated NPU runs out of headroom
Three situations reliably exceed what 6 TOPS can deliver:
More camera streams than the NPU can process in real time. The NPU’s 6 TOPS is a shared resource across every model running concurrently. Push past the stream count your models and resolution allow, and frame rates drop system-wide rather than gracefully degrading on just the newest stream.
Transformer or ViT-based models. RK3588’s NPU architecture is tuned for CNN operations; transformer-based vision models need more careful optimization to map well onto it, and often don’t reach the throughput CNN benchmarks would suggest. If your model roadmap includes anything transformer-based, budget extra evaluation time even before deciding whether to add an accelerator.
Any on-device LLM, VLM, or generative AI workload. This is the clearest line. RK3588’s NPU can technically run small LLMs through Rockchip’s RKLLM toolkit — community benchmarks report roughly 10-15 tokens/sec on a 1.1B parameter model — but it only supports W8A8 quantization for LLM inference, not the more memory-efficient W4A16 that newer chips like RK3576 offer, and the practical ceiling on model size and speed is low. For anything beyond a small on-device utility model, this isn’t the tool for the job.
What adding an M.2 accelerator actually buys you
The economics here work as a spectrum rather than a binary choice. RK3588’s own NPU delivers 6 TOPS with zero added cost. Adding a Hailo-8 M.2 module brings the platform to roughly 32 TOPS combined, aimed at scaling up vision throughput — more streams, higher resolution, heavier CNN models — without changing the host SoC. Adding a Hailo-10H M.2 module instead adds 40 TOPS INT4 with a direct DDR interface, which is the piece RK3588’s onboard NPU lacks for running LLMs and VLMs at usable speed.
This is the practical way to think about it: the RK3588 SoC is the CPU/GPU/multimedia host and baseline vision NPU in every configuration. The M.2 slot is where you scale AI capability up to match the workload, without redesigning the board or re-qualifying a new host processor. That’s a meaningfully lower-risk path than jumping to a different SoC platform once a project’s AI requirements grow past the original spec.
Cost, power, and board complexity trade-offs
Adding an M.2 accelerator isn’t free, and the trade-offs are worth being explicit about before defaulting to “add one just in case”:
- BOM cost increases with any additional module — a real consideration at volume, even when the per-unit accelerator cost looks small in isolation.
- Power budget increases, typically by 2.5-10W depending on the accelerator and workload, which can push a fanless design toward needing a heatsink or reconsidering enclosure thermals.
- Board space and M.2 slot availability need to be designed in from the start; retrofitting an M.2 slot onto a board that didn’t plan for one isn’t an option late in a design cycle.
- Software integration adds a second toolchain to validate — RKNN for the onboard NPU, plus HailoRT (or another accelerator’s SDK) for the add-in module — which means more validation work, not just more silicon.
None of these are reasons to avoid an accelerator when the workload genuinely needs one. They’re reasons not to add one by default when the onboard NPU would have done the job.
Decision guide by workload
| Workload | Recommendation | Why |
|---|---|---|
| Single or dual-camera object detection | Integrated NPU only | 6 TOPS comfortably handles CNN detection at this scale |
| Multi-camera NVR (8-16 channels, CNN models) | Integrated NPU, evaluate first | Documented deployments hit this range on 6 TOPS; benchmark your specific model before adding hardware |
| High channel-count or high-resolution multi-stream analytics | Add Hailo-8 (M.2) | Scales vision throughput without changing the host SoC |
| On-device LLM, VLM, or chatbot feature | Add Hailo-10H (M.2) | Onboard NPU lacks the memory architecture for generative AI at usable speed |
| Transformer/ViT-based vision model | Benchmark before deciding | Optimization-dependent; may run acceptably on-chip or may need an accelerator |
| Product roadmap likely to add generative AI later | Design in an M.2 slot now | Cheaper to reserve the slot upfront than to redesign the board later |
FAQ
Is RK3588’s built-in NPU enough for edge AI, or do I need an accelerator?
For CNN-based vision workloads — detection, classification, moderate multi-camera analytics — RK3588’s 6 TOPS NPU is generally enough on its own. You need an add-in M.2 accelerator once you exceed that in stream count, model complexity, or if the product needs to run an LLM or VLM, which the onboard NPU isn’t architected to handle well.
Can RK3588 run large language models without an accelerator?
Only small models, and with real limitations. RK3588’s NPU supports LLM inference through Rockchip’s RKLLM toolkit, but only with W8A8 quantization rather than the more memory-efficient W4A16 available on newer chips, and community benchmarks put throughput around 10-15 tokens/sec on a roughly 1B-parameter model. For anything beyond a small on-device utility model, an M.2 accelerator like Hailo-10H is the more practical path.
How many camera streams can RK3588’s integrated NPU handle?
It depends heavily on resolution and model choice, but documented industrial deployments have run 16 channels of CNN-based detection (license plate/vehicle recognition) at 18-20 FPS per channel on the onboard 6 TOPS NPU alone. Benchmark your specific model and resolution before assuming you need additional hardware.
Does adding an M.2 AI accelerator to RK3588 replace the onboard NPU, or work alongside it?
It works alongside it. RK3588’s onboard NPU continues handling whatever it was already doing — light inference, or nothing at all if the design routes all AI workload to the M.2 module — while the add-in accelerator handles the heavier workload it was chosen for. Nothing about adding a module disables or replaces the SoC’s own NPU.
Should I design in an M.2 slot even if I don’t need an accelerator at launch?
It’s often worth it if your product roadmap includes AI features that might grow — generative AI, more cameras, heavier models. Reserving the M.2 slot and board space at the design stage is far cheaper than a board respin later, even if the accelerator itself isn’t populated on day one.
Getting to hardware
Geniatech’s RK3588 platforms — including SOM3588 and APC3588-AI — ship with an open M.2 slot that supports Geniatech’s own accelerator lineup: the AIM-B-H8 and Hailo-8 M.2 variant for scaling vision throughput, or the AIM-M-H10 for on-device LLM and VLM workloads. For a broader comparison of accelerator options across vision and generative AI workloads, see our M.2 AI accelerator comparison.