[LLVMdev] Benchmarks for small embedded platforms

[James Pallister]

> Perhaps you mean any Open Source license? It seems rather pointless to intentionally exclude BSD, MIT, Apache, etc, licenses as well as public domain code.
Thanks for catching this. I do mean any code under an Open Source
license is fine, just the particular benchmarks chosen happened to be
available under the GPL.

> That's definitely going to restrict things a bit. Why that platform? It's Thumb1 only, and most compilers these days focus their optimization work on Thumb2. That's certainly very true for LLVM.

That particular platform was chosen because it was readily available and
the development board is particularly easy to instrument with the power
monitoring hardware.

> For example, your choices above have a number of floating point intensive codes. The M0 does not have hardware floating point, so the results for those benchmarks on that processor will not be very interesting. On the other hand, I could easily see a processor like this encrypting its output data streams, so those tests will be very interesting.
I agree, some of the benchmarks may not be appropriate for every
platform. It is nice to have at least some benchmarks which can be
common between platforms so that comparisons can be drawn. A subset of
the benchmarks which makes more sense for the M0 could be used though.

> Chips of that class often spend most of their time doing non-compute intensive tasks. I/O over CAN, USB, plain serial, etc., with intermittent minor algorithmic work along the way. More compute heavy tasks like you outline above are typically performed on a more capable processor elsewhere in the system.
This is interesting, and something that I haven't considered. It may be
worthwhile me finding a type of benchmark that can cover these things
for the smaller processors.


Thanks,
James


On 31/07/12 18:17, Jim Grosbach wrote:
> On Jul 31, 2012, at 4:43 AM, James Pallister <james.pallister at embecosm.com> wrote:
>
>> Hi Everyone,
>>
>> As you may recall (I posted a while back) I'm currently doing a project
>> measuring the energy consumption for different compiler flags. I've now
>> gathered a set of benchmarks which I think are appropriate, for these
>> (very) low end embedded platforms.
>>
>> The benchmarks that have been chosen:
>> - CRC32, SHA
>> - Prime sifting
>> - Integer / floating point matrix multiplication
>> - Cubic root solver
>> - 2D convolution, FFT, fast discrete cosine transform
>> - Dijkstra, Patricia
>> - Rjindael, Blowfish, Triple DES
>> - Arithmetic coding (?)
>>
>> More info at:
>> http://jpallister.com/wiki/index.php?title=Benchmarks#Final_Benchmark_Choice
>>
>> These have been chosen because:
>> - most have code available under the GPL
> Perhaps you mean any Open Source license? It seems rather pointless to intentionally exclude BSD, MIT, Apache, etc, licenses as well as public domain code.
>
>> - they are small enough to fit on the target platforms (the ARM
>> Cortex-M0 we have only has 64 KiB flash and 8 KiB RAM)
> That's definitely going to restrict things a bit. Why that platform? It's Thumb1 only, and most compilers these days focus their optimization work on Thumb2. That's certainly very true for LLVM.
>
>> - they are wide ranging enough to exercise most points of all the
>> target platforms.
> Chips of that class often spend most of their time doing non-compute intensive tasks. I/O over CAN, USB, plain serial, etc., with intermittent minor algorithmic work along the way. More compute heavy tasks like you outline above are typically performed on a more capable processor elsewhere in the system.
>
> You have to choose your benchmark workload according to the platform that you're going to be testing to get meaningful results. Your testing methodology should mirror how engineers select the devices to be used in their application. Both your code choices and your target platform choices are reasonable, but they aren't typically used together in a real system, so your results are less likely to be representative of what engineers will see in their applications.
>
> For example, your choices above have a number of floating point intensive codes. The M0 does not have hardware floating point, so the results for those benchmarks on that processor will not be very interesting. On the other hand, I could easily see a processor like this encrypting its output data streams, so those tests will be very interesting.
>
>
> Best regards,
>   Jim
>
>> Currently I haven't found a benchmark for arithmetic coding, as a lot of
>> it is covered by patents. Does anyone know of any (preferably GPL) code
>> or benchmarks for this type of thing?
>>
>> Also do you think there is anything I've missed, or have any other
>> recommendations for benchmarks for small embedded platforms?
>>
>> Thanks,
>> James
>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev