Alex, I'm not sure if this is a good way of addressing your request. What are the benchmarks/measurements you're trying to run? Which boards would you like to be tested? What is the acceptance criteria for the benchmarks (no worse than last run or sth else)? LKP isn't as good as it looks at first. There are quite a few assumptions made by Intel that aren't true for ARM boards. So IMHO the best way forward is to define your use case and than try to find tools, not the other way around.

BTW, we also have workload automation (https://github.com/ARM-software/workload-automation/) for benchmarking and possibly power measurement.

milosz

On 17 September 2015 at 10:24, Alex Shi alex.shi@linaro.org wrote:

Good news!

Did the LKP regularly run in linaro? Could I give your a kernel branch and then you give the testing results?

Regards Alex

On 09/17/2015 05:02 PM, Chase Qi wrote:

...

On 09/17/2015 04:44 PM, Alex Shi wrote:

...

Hi All,

Is there some performance or power auto-testing system running in Linaro? Like intel's lkp porting to LAVA?

Riku/Chase, Do you have a plan to give a update or show you work in coming connection?

Alex, there is a session on Wed Sep 23rd to talk about LKP in LAVA and result comparison. If you don't want to wait ;), here is the slide decks https://docs.google.com/a/linaro.org/presentation/d/1Fj5AMu3z6DQMb4QokSBGXgB...

Thanks, Chase

...

If there is a system can measure the performance/power change on any SoC for each of patch, that would be perfect!

Regards Alex

On 09/17/2015 08:17 PM, Amit Kucheria wrote:

(Adding Lisa)

Alex,

Lisa has been working to get several benchmarks integrated into workload automation. Very soon we will have a PMWG-dedicated lab in Cambridge to do some perf and power measurements (on 2-3 wired up boards in the beginning)

Sounds great!

When the system could work for us? Could I submit a branch for testing? I will be excited to see the system running. :)

Chase/Milosz, LKP capabilities in LAVA sound great and I look forward to using it. However, is improving of the general reporting and trend charting on the agenda?

Regards, Amit

On Thu, Sep 17, 2015 at 3:52 PM, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:

...

Alex, I'm not sure if this is a good way of addressing your request. What are the benchmarks/measurements you're trying to run? Which boards would you like to be tested? What is the acceptance criteria for the benchmarks (no worse than last run or sth else)? LKP isn't as good as it looks at first. There are quite a few assumptions made by Intel that aren't true for ARM boards. So IMHO the best way forward is to define your use case and than try to find tools, not the other way around.

BTW, we also have workload automation (https://github.com/ARM-software/workload-automation/) for benchmarking and possibly power measurement.

milosz

On 17 September 2015 at 10:24, Alex Shi alex.shi@linaro.org wrote:

...

Good news!

Did the LKP regularly run in linaro? Could I give your a kernel branch and then you give the testing results?

Regards Alex

On 09/17/2015 05:02 PM, Chase Qi wrote:

...

On 09/17/2015 04:44 PM, Alex Shi wrote:

...

Hi All,

Is there some performance or power auto-testing system running in Linaro? Like intel's lkp porting to LAVA?

Riku/Chase, Do you have a plan to give a update or show you work in coming connection?

Alex, there is a session on Wed Sep 23rd to talk about LKP in LAVA and result comparison. If you don't want to wait ;), here is the slide decks https://docs.google.com/a/linaro.org/presentation/d/1Fj5AMu3z6DQMb4QokSBGXgB...

Thanks, Chase

...

If there is a system can measure the performance/power change on any SoC for each of patch, that would be perfect!

Regards Alex

On 09/17/2015 09:34 PM, Amit Kucheria wrote:

On Thu, Sep 17, 2015 at 6:51 PM, Alex Shi alex.shi@linaro.org wrote:

...

On 09/17/2015 08:17 PM, Amit Kucheria wrote:

...

(Adding Lisa)

Alex,

Lisa has been working to get several benchmarks integrated into workload automation. Very soon we will have a PMWG-dedicated lab in Cambridge to do some perf and power measurements (on 2-3 wired up boards in the beginning)

Sounds great!

When the system could work for us? Could I submit a branch for testing? I will be excited to see the system running. :)

I'd expect it to start after Connect. A server has been ordered to run workload automation and setup SSH access. I can request for you to have access along with Lisa.

Great news! Thanks for help! and hope the system working soon!

Regards, Amit

On 17 September 2015 at 14:15, Alex Shi alex.shi@linaro.org wrote:

Hi Milosz,

I have no specific requirements for details. But I guess the rt-tests' benchmarks are first thing we can start. alexs@alexs-des:/mmkernels/lsk$ apt-file list rt-tests rt-tests: /usr/bin/cyclictest rt-tests: /usr/bin/hackbench rt-tests: /usr/bin/pi_stress rt-tests: /usr/bin/pip_stress rt-tests: /usr/bin/pmqtest rt-tests: /usr/bin/ptsematest rt-tests: /usr/bin/rt-migrate-test rt-tests: /usr/bin/sendme rt-tests: /usr/bin/signaltest rt-tests: /usr/bin/sigwaittest rt-tests: /usr/bin/svsematest rt-tests: /usr/sbin/hwlatdetect

and the typical boards are appreciated, like tc2, juno, and others.

We had these running for LSK 3.14 RT some time ago so you can use the same gear to run the tests already now. There is no need to wait for anything.

For performance/power measure, the criteria may not the result value. It should be the change percentage, like the performance increase/decrease percentage. or the power(W)/energy(J) changed percentage. Guess Amit will have better suggestion on power testing.

I amn't a keen of LKP. I'd like any auto testing framework if it can give me the per/power changes of two branches. :)

I'm really not sure what are you expecting here? That can already be done with a little bit of scripting using LAVA. As Amit mentioned above, we don't have any reporting tools in place at this time.

milosz

Regards Alex

On 09/17/2015 06:22 PM, Milosz Wasilewski wrote:

...

Alex, I'm not sure if this is a good way of addressing your request. What are the benchmarks/measurements you're trying to run? Which boards would you like to be tested? What is the acceptance criteria for the benchmarks (no worse than last run or sth else)? LKP isn't as good as it looks at first. There are quite a few assumptions made by Intel that aren't true for ARM boards. So IMHO the best way forward is to define your use case and than try to find tools, not the other way around.

BTW, we also have workload automation (https://github.com/ARM-software/workload-automation/) for benchmarking and possibly power measurement.

milosz

On 17 September 2015 at 10:24, Alex Shi alex.shi@linaro.org wrote:

...

Good news!

Did the LKP regularly run in linaro? Could I give your a kernel branch and then you give the testing results?

Regards Alex

On 09/17/2015 05:02 PM, Chase Qi wrote:

...

On 09/17/2015 04:44 PM, Alex Shi wrote:

...

Hi All,

Is there some performance or power auto-testing system running in Linaro? Like intel's lkp porting to LAVA?

Riku/Chase, Do you have a plan to give a update or show you work in coming connection?

Alex, there is a session on Wed Sep 23rd to talk about LKP in LAVA and result comparison. If you don't want to wait ;), here is the slide decks https://docs.google.com/a/linaro.org/presentation/d/1Fj5AMu3z6DQMb4QokSBGXgB...

Thanks, Chase

...

If there is a system can measure the performance/power change on any SoC for each of patch, that would be perfect!

Regards Alex

On 17 September 2015 at 15:45, Alex Shi alex.shi@linaro.org wrote:

On 09/17/2015 09:35 PM, Milosz Wasilewski wrote:

...

...

...

For performance/power measure, the criteria may not the result value. It should be the change percentage, like the performance increase/decrease percentage. or the power(W)/energy(J) changed percentage. Guess Amit will have better suggestion on power testing.

I amn't a keen of LKP. I'd like any auto testing framework if it can give me the per/power changes of two branches. :)

I'm really not sure what are you expecting here? That can already be done with a little bit of scripting using LAVA. As Amit mentioned above, we don't have any reporting tools in place at this time.

Hi Milosz,

I don't know if I am not clear with the following part. Do you mean lava is already can figure out this:

like, we can start from the performance measurement. We always has result for the performance testing, like the task running time, if one kernel patch cause 10 more percent running time. we say the performance decreases 10%. And if we set the performance alarm criteria as 5%. then, we know it mean the patch is bad. The other performance data or power data could be deal with similar idea.

LAVA is not doing any postprocessing/analysis of the data it generates. It's purely the tool to automatically run tests. So it will tell you what the test results are with the build (*) you submit. The decision whether the results are 'good' or 'bad' is always the responsibility of requester. From what you wrote above I assume you're interested in single measurement (like hackbench with fixed parameters). That is already available. If you submit the test jobs in a way they can be filtered in LAVA, creating a simple report with threshold is also possible. I'm not sure about threshold alerts as I never used them.

* it isn't enough to point to the tree/branch to have the testing done.

milosz