RKNN Toolkit, HailoRT, and ONNX Runtime aren’t three competing options for the same job — they solve different parts of the edge AI deployment pipeline. RKNN Toolkit is Rockchip’s proprietary compiler and runtime for its own NPUs (RK3588, RK3576, and others). HailoRT is Hailo’s runtime, paired with its Dataflow Compiler, for Hailo’s own accelerator silicon. ONNX Runtime is a hardware-agnostic runtime that executes standard ONNX models across CPUs, GPUs, and any hardware with a matching execution provider — but it doesn’t natively run on Rockchip or Hailo NPUs the way RKNN and HailoRT do.
Most confusion around these three comes from treating ONNX as if it competes with the other two. It doesn’t, exactly — ONNX is mainly the interchange format that feeds into both RKNN Toolkit and Hailo’s Dataflow Compiler, and ONNX Runtime is the engine that executes ONNX models where no vendor-specific NPU compiler is involved. Understanding that relationship is the first step to picking the right toolchain for your deployment.
Quick answer
- Use RKNN Toolkit if you’re deploying to a Rockchip SoC’s onboard NPU (RK3588, RK3576, RV1103, etc.). There’s no alternative runtime that gets you onto that NPU — RKNN is the only path.
- Use HailoRT if you’re deploying to Hailo accelerator hardware (Hailo-8, Hailo-10H) via M.2 or B2B modules. Same situation: HailoRT is the runtime, and Hailo’s Dataflow Compiler is the required compilation step.
- Use ONNX Runtime for the parts of your pipeline that aren’t running on a vendor NPU — host-side pre/post-processing, CPU or GPU inference, or any hardware where ONNX Runtime has a supported execution provider (e.g., Qualcomm QNN, Intel OpenVINO, NVIDIA TensorRT).
- In most real edge AI products using Rockchip or Hailo hardware, you’ll touch all three at different stages of the pipeline — not choose just one.
What each tool actually is
| RKNN Toolkit | HailoRT | ONNX Runtime | |
|---|---|---|---|
| Type | Vendor NPU compiler + runtime | Vendor NPU runtime (paired with Dataflow Compiler) | Hardware-agnostic inference runtime |
| Target hardware | Rockchip NPUs only (RK3588, RK3576, RV-series) | Hailo accelerators only (Hailo-8, Hailo-10H) | CPU, GPU, and any hardware with an execution provider |
| Model input formats | PyTorch, TensorFlow, Caffe, ONNX | TensorFlow, ONNX | ONNX (native format) |
| Output format | Proprietary .rknn | Proprietary HEF | Runs ONNX graphs directly |
| Conversion host | x86_64 Linux (RKNN-Toolkit2) | x86_64 Linux (Dataflow Compiler; GPU speeds up compilation) | No separate compilation step required |
| OS support on target | Linux, Android | Linux, Windows, Android | Linux, Windows, macOS, Android, iOS |
| GenAI/LLM support | Separate RKLLM toolkit, INT8-only (W8A8) quantization on RK3588 | Model Zoo GenAI, growing LLM/VLM support via GenAI stack | onnxruntime-genai extension, broad but CPU/GPU-oriented |
| Portability | Locked to Rockchip silicon | Locked to Hailo silicon | Portable across supported hardware, but often not the fastest option on any single piece of it |
RKNN Toolkit: the only path onto a Rockchip NPU
If your product uses an RK3588, RK3576, or other Rockchip SoC and you want the onboard NPU doing inference, RKNN Toolkit isn’t a choice among alternatives — it’s the required path. The typical workflow: train a model in PyTorch or TensorFlow, export it to ONNX with fixed input shapes, then run RKNN-Toolkit2 on an x86_64 Linux machine to convert and quantize it into the proprietary .rknn format. That file deploys to the target board, where the RKNPU runtime and driver stack execute it.
Two practical constraints worth planning around: conversion has to happen on an x86_64 Linux system — you can’t convert models directly on the Rockchip board itself — and RKNN’s LLM support (through the separate RKLLM toolkit) is currently limited to INT8-level quantization on RK3588, without the more memory-efficient W4A16 quantization available on some other platforms. For CNN-based vision workloads, though, RKNN has a mature, well-documented pipeline with official integrations in common tools like Ultralytics YOLO exports.
HailoRT: the runtime side of Hailo’s compiler pipeline
HailoRT plays a structurally similar role for Hailo hardware. Models come from TensorFlow or ONNX and go through Hailo’s Dataflow Compiler on an x86_64 Linux build machine — a GPU on that machine speeds up compilation meaningfully, though CPU-only compilation is possible. The compiler outputs a HEF (Hailo Executable Format) file, which HailoRT then loads and runs on the target device, whether that’s a Hailo AI Vision Processor running standalone or a host processor (x86 or ARM) with a Hailo accelerator attached over M.2 or PCIe.
HailoRT is open-source and available via GitHub, supports C/C++ and Python APIs, and runs across Linux, Windows, and Android hosts. Hailo’s software suite has also expanded to include a GenAI Model Zoo alongside its established vision model zoo, reflecting the same generational shift covered in our Hailo-8 vs Hailo-10H comparison — Hailo-8 workloads run through the same vision-focused pipeline that’s been maturing for years, while Hailo-10H workloads increasingly lean on the newer GenAI-specific tooling.
ONNX Runtime: the portable layer, not the fast lane onto vendor NPUs
ONNX Runtime’s value proposition is different from the other two: it’s built around a standardized model format and a pluggable execution provider architecture that lets the same ONNX model run across CPUs, GPUs, and any hardware backend with a supported EP — NVIDIA TensorRT, Intel OpenVINO, and Qualcomm’s QNN execution provider are common examples. That portability is genuinely useful when a team is developing across multiple platforms or wants a single export pipeline before device-specific optimization.
What ONNX Runtime doesn’t do is run directly on Rockchip or Hailo NPU silicon. Neither vendor currently exposes an ONNX Runtime execution provider for their NPU hardware — instead, both require the ONNX model to go through their own compiler (RKNN Toolkit or Hailo’s Dataflow Compiler) to produce a vendor-specific compiled artifact. In practice, ONNX Runtime tends to show up in edge AI products for the parts of the pipeline that run on the host CPU — pre/post-processing, models that don’t need NPU acceleration, or fallback execution when the target hardware doesn’t have a fast path available.
How these fit together in a real deployment
A common pattern on an RK3588-plus-Hailo-accelerator design looks like this: a vision model is trained once, exported to ONNX as the universal intermediate format, then branches into two separate compilation paths — RKNN-Toolkit2 for anything running on the onboard NPU, and Hailo’s Dataflow Compiler for anything offloaded to an M.2 Hailo accelerator. Meanwhile, ONNX Runtime might handle a smaller CPU-side model, or serve as the fallback path during development before either NPU-specific conversion is finalized.
This is also why “portability” isn’t really a fair axis to compare RKNN and HailoRT against ONNX Runtime — RKNN and HailoRT were never meant to be portable across vendors; they exist specifically because generic runtimes like ONNX Runtime don’t reach peak performance (or, currently, run at all) on Rockchip and Hailo NPU silicon.
Decision guide by scenario
| Scenario | Toolchain | Why |
|---|---|---|
| Deploying to RK3588/RK3576 onboard NPU | RKNN Toolkit | Required — no alternative runtime accesses the Rockchip NPU |
| Deploying to a Hailo-8 or Hailo-10H accelerator | HailoRT + Dataflow Compiler | Required — HailoRT is the only runtime for Hailo silicon |
| Model needs to run on host CPU alongside NPU inference | ONNX Runtime | Hardware-agnostic, avoids a second vendor toolchain for non-NPU workloads |
| Multi-platform product targeting several different SoCs | ONNX as shared export format, then vendor-specific compilers per target | Keeps one training/export pipeline while still hitting each NPU’s native toolchain |
| Prototyping before hardware is finalized | ONNX Runtime | Runs the model immediately on CPU/GPU without committing to a vendor compile step |
| On-device LLM/VLM deployment | RKLLM (Rockchip) or HailoRT + GenAI Model Zoo (Hailo), depending on chosen hardware | Match the GenAI toolchain to whichever accelerator has the memory architecture the model needs |
FAQ
Is ONNX Runtime a replacement for RKNN Toolkit or HailoRT?
No. ONNX Runtime is a hardware-agnostic runtime for CPUs, GPUs, and hardware with a supported execution provider, but neither Rockchip nor Hailo currently exposes an ONNX Runtime execution provider for their NPUs. To run a model on an RK3588 or Hailo accelerator, the model still needs to go through RKNN Toolkit or Hailo’s Dataflow Compiler respectively.
Can I convert an ONNX model directly to run on a Rockchip NPU?
Yes — ONNX is actually the recommended input format for RKNN-Toolkit2, which converts and quantizes the ONNX model into Rockchip’s proprietary .rknn format. The conversion has to happen on an x86_64 Linux machine, and the resulting .rknn file is what actually runs on the target board’s NPU.
Do RKNN and HailoRT support the same model types?
Mostly overlapping but not identical. Both support standard CNN-based vision models (detection, classification, segmentation) well, with mature export paths from tools like Ultralytics YOLO. LLM and generative model support differs: Rockchip’s RKLLM toolkit currently supports INT8-level quantization only on RK3588, while Hailo’s GenAI Model Zoo and Hailo-10H’s architecture are built specifically around larger generative models.
Which toolchain has the easier learning curve for a team new to edge AI?
ONNX Runtime is generally the gentlest starting point since it runs standard ONNX models on CPU or GPU without a separate hardware-specific compilation step, which is useful for early prototyping. Once you commit to Rockchip or Hailo hardware for production, though, learning RKNN Toolkit or HailoRT respectively becomes unavoidable — the onboard NPU or accelerator won’t run inference without it.
Do I need to learn all three toolchains for one product?
Often, yes, at least partially. Any product combining a Rockchip SoC’s onboard NPU with a Hailo M.2 accelerator will typically need both RKNN Toolkit and HailoRT, with ONNX Runtime sometimes handling host-side or fallback inference. Budget development time for each toolchain separately rather than assuming one covers the full pipeline.
Deploy Edge AI Applications with Geniatech Platforms
Geniatech’s edge AI platforms are built around exactly this kind of mixed pipeline. SOM3588 and APC3588-AI run RK3588’s onboard NPU via RKNN, with an open M.2 slot for the AIM-M-H10 (Hailo-10H) accelerators running through HailoRT.