Literature Review: LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load | Bnechmarking LLMs on Phones [R]

LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load

I am starting to benchmark LLMs on edge devices, particularly phones thus been reading a lot on the what has been done and what is currently being done and wanted to share you my journey of reading such papers and my takes on them.

Then chose Qwen-2.5-1.5B 4bit and single user-single prompt of 2048 token prompt through 20 rounds of testing done, there was also nom limit of max tokens to be generated - one of the limitations.

Also, they have used different inference engines for each hardware - MLC LLM, vLLM, hailo-ollama and MLX which is their one of the limitations.

- RPi5-Hailo gives one of the most consistent and stable performances in terms of thermals, power and throughput (CV of .04%) with no throttling. However, latency of about 72 seconds for 564 tokens, its quite slow for chatting purposes.

One of the reasons mentioned was that even through the Hailo Hat offered PCIe Gen3 x4 connector, the RPi5 has 1x gen 2 connector (400 MB/s vs 1GB/s) but more likely reason is how haillo-ollama orchestrate the CPU-NPU comms using their dataflow pipeline. Also, not every layer is executed on NPU but rather on CPU too automatically.

- iPhone had the best tok/sec (prefil + decode), decode time for the smartphone category, it showed instabilization for the initial and final few iters (from ~42 tok/sec to merely 23-24 tok/sec) due to its thermal activity mainly.

- For S24 Ultra, it's a bit interesting story. So, it used MLC-LLM famous for it its GPU computability with mobiles but also quite unstable; for this phone the authors had to refill chink (128) cus the whole prompt cause a huge spike in resources used -> DVFS kicked in -> thermals went up and the resource allocator had to probably pin down the GPU freq to the minimum thus causing a frequency floor.

So, with chunked prefill, the decade time was 56 seconds for 646 tokens with a similar throughput of 10.8 tok/sec (CV 4.2 %) but then thermals was quite stabilized with an avg of 64 +/- 1.9 C for GPU and CPU.

- For the laptop, the avg system power was 34 W well blew its TGP, but still its well the best one so far.

- They should have used CV to compare the four devices mainly and not to do it within themselves

- The power metric collected is varying like we are comparing the overall Watts used - not per component.

- No frequency study, DVFS nor used with some other apps or tasks running in the background.