How feasible would it be to implement sbc npus such as rockchip with this? Some of them are already in mainline
It is feasible, in that on ARM, your compute options are going to be the vendor-native paths (like the NPU/VFP), GPU Vulkan lanes (Mali, Imagination, or Adreno), or generic GPU/CPU OpenCL.
Real-world mileage is going to vary massively based on the specific vendor’s silicon routing and how they handle unified memory. As, you probably know, these tiny boards share one single memory bus across the CPU, GPU, and NPU, bandwidth starvation becomes your main bottleneck the second you try sharding models.
That said, the CPU fallback path will absolutely work out of the box to guarantee a baseline.
It is feasible, in that on ARM, your compute options are going to be the vendor-native paths (like the NPU/VFP), GPU Vulkan lanes (Mali, Imagination, or Adreno), or generic GPU/CPU OpenCL.
Real-world mileage is going to vary massively based on the specific vendor’s silicon routing and how they handle unified memory. As, you probably know, these tiny boards share one single memory bus across the CPU, GPU, and NPU, bandwidth starvation becomes your main bottleneck the second you try sharding models.
That said, the CPU fallback path will absolutely work out of the box to guarantee a baseline.