On 4 July 2013 17:34, Siarhei Siamashka <siarhei.siamashka@gmail.com> wrote:

For getting reproducible benchmark results, you just need to ensure

that thermal throttling never kicks in. If the kernel is compiled
with cpufreq stats enabled, you can compare these stats before/after
your benchmark to ensure that it spent all the time running at the
same designated clock frequency.

I did that on my Chromebook, put it on power mode and I get pretty consistent build and benchmark times. It's an art to make sure the benchmark run-time is enough to give you statistically relevant results while not being too much to deal with overheating or scheduling issues, but that, as you say, can be "fixed" by running on lower frequencies, I don't mind about that.

What I really mind is to lower the frequency of our buildbots, the ones that should be building and testing under 20 minutes (like octo-core i7s) but take 3 hours to do so (on a dual Panda). While the comparison is in no way fair, reducing the freq. will only make it worse. Coming from a server farm culture, where noise, power and air-conditioning are always topped up and never too expensive, it's hard not to giggle when hearing that you should lower the frequency to get "expected results".

Yes, ARM devices were designed with the phone market in mind, but today they're a lot more than that, and if they're to get into the server space, they have to be consistent, even when cranked up all the way to 11.

Anyway, I recommend you to start the tests for the hardware
robustness with:

wget https://raw.github.com/ssvb/cpuburn/master/cpuburn-a9.S
arm-linux-gnueabihf-gcc -o cpuburn-a9 cpuburn-a9.S

I'll do that and report on my findings.

Thanks for the overall, it was very educational. ;)

cheers,

--renato