[LLVMdev] Benchmarks for small embedded platforms

[Jim Grosbach]

On Jul 31, 2012, at 4:43 AM, James Pallister <james.pallister at embecosm.com> wrote:

> Hi Everyone,
> 
> As you may recall (I posted a while back) I'm currently doing a project
> measuring the energy consumption for different compiler flags. I've now
> gathered a set of benchmarks which I think are appropriate, for these
> (very) low end embedded platforms.
> 
> The benchmarks that have been chosen:
> - CRC32, SHA
> - Prime sifting
> - Integer / floating point matrix multiplication
> - Cubic root solver
> - 2D convolution, FFT, fast discrete cosine transform
> - Dijkstra, Patricia
> - Rjindael, Blowfish, Triple DES
> - Arithmetic coding (?)
> 
> More info at:
> http://jpallister.com/wiki/index.php?title=Benchmarks#Final_Benchmark_Choice
> 
> These have been chosen because:
> - most have code available under the GPL

Perhaps you mean any Open Source license? It seems rather pointless to intentionally exclude BSD, MIT, Apache, etc, licenses as well as public domain code.

> - they are small enough to fit on the target platforms (the ARM
> Cortex-M0 we have only has 64 KiB flash and 8 KiB RAM)

That's definitely going to restrict things a bit. Why that platform? It's Thumb1 only, and most compilers these days focus their optimization work on Thumb2. That's certainly very true for LLVM.

> - they are wide ranging enough to exercise most points of all the
> target platforms.

Chips of that class often spend most of their time doing non-compute intensive tasks. I/O over CAN, USB, plain serial, etc., with intermittent minor algorithmic work along the way. More compute heavy tasks like you outline above are typically performed on a more capable processor elsewhere in the system.

You have to choose your benchmark workload according to the platform that you're going to be testing to get meaningful results. Your testing methodology should mirror how engineers select the devices to be used in their application. Both your code choices and your target platform choices are reasonable, but they aren't typically used together in a real system, so your results are less likely to be representative of what engineers will see in their applications.

For example, your choices above have a number of floating point intensive codes. The M0 does not have hardware floating point, so the results for those benchmarks on that processor will not be very interesting. On the other hand, I could easily see a processor like this encrypting its output data streams, so those tests will be very interesting.


Best regards,
  Jim

> Currently I haven't found a benchmark for arithmetic coding, as a lot of
> it is covered by patents. Does anyone know of any (preferably GPL) code
> or benchmarks for this type of thing?
> 
> Also do you think there is anything I've missed, or have any other
> recommendations for benchmarks for small embedded platforms?
> 
> Thanks,
> James
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev