Same quantized model, two fates—one rejected, the other ran in 1.39 ms
- qualcomm
- AI
- edge AI
- benchmark
- quantization
I passed one quantized NanoDet model to two Qualcomm devices. One refused even to load it. The other ran all 324 layers on the NPU in 1.39 ms. The model was identical, and so was the quantization. The difference was the float tail left untouched by the quantizer, and whether the device allowed it.
Model
NanoDet was quantized to w8a8. For calibration, 23 real-world images from COCO val2017 were used (one of the planned 24 image downloads failed, so I won’t call this a clean set of numbers). We sent the same graph to two devices: Dragonwing RB3 Gen 2 (QCS6490) and Galaxy S25.
Results for each device
| Device | Path | Result |
|---|---|---|
| RB3 Gen 2 | TFLite, float | 90.0 ms — 352 layers: 169 on the CPU, 123 on the GPU, and 60 on the NPU. CPU time alone accounted for 85% of the total. |
| RB3 Gen 2 | TFLite, w8a8 | 6.47 ms (median 7.61 ms) — all 439 layers on the NPU. About 14× faster than float on the same path |
| RB3 Gen 2 | Precompiled QNN, w8a8 | Rejected — the graph contains float tail tensors that cannot be quantized (distribution integration after the GFL head and box-coordinate decoding operations). This device requires every I/O to be quantized. |
| Galaxy S25 | Precompiled QNN, w8a8 | 1.39 ms (median 1.40 ms) — all 324 layers on the NPU |
| Galaxy S25 | Precompiled QNN, float | 1.47 ms — all layers on the NPU here as well |
(The number of layers differs by precision because the quantized graph contains additional quantized operations. When comparing across precisions, use latency rather than the number of layers.)
Why the Same Graph Has Two Fates
In the earlier experiment, YOLOv11-N quantized cleanly on RB3—all 341 layers on the NPU, taking 5.03 ms. That was because its export produced raw convolution outputs. NanoDet's export includes the decode operations within the graph. The GFL head's distribution integration and box-coordinate decoding run in float after the convolution. The quantizer could not convert that tail, and RB3's QNN path does not accept graphs whose I/O is not fully quantized. The same quantized graph and the same device family—but the tail of one model is clean, while the other's is not.
S25 runs the same graph in 1.39 ms. Because its QNN path accepts it. And S25's float path takes 1.47 ms—on the flagship, quantization buys almost nothing. Because the float path already used the NPU for every layer. On RB3, quantization is the difference between 90 ms and 6.47 ms. Because the float path was almost entirely on the CPU.
There are two approaches under consideration for fixing RB3's rejection: compiling the quantized model via TFLite (convolutions on the NPU, the tail on the CPU), or splitting the export (sending the backbone and head to QNN as pure convolutions, with decoding on the host). Both arrive at the same conclusion: the decode tail belongs on the host, not inside the graph.
The Takeaway
Quantization is not a property of the model alone, but of the I/O contract between the graph and the device. Even a “quantized” model will not run on a device if it has float operations at its tail that the device does not accept. When reporting quantization benchmarks, state the execution path and device. “w8a8” alone does not even tell you whether it actually ran.
This is the story behind the earlier 27× result. There, the runtime path determined speed. Here, the runtime path determines whether the model runs at all. In both cases, the decisive evidence was the layer-by-layer profile, not the headline number.
The model itself is already published as an AI Hub external recipe— https://huggingface.co/anguoyang/nanodet —with w8a8 listed as a supported precision, and RB3's QNN rejection recorded in the recipe's empirical report, so the next person encounters a signpost rather than a wall.
