On 27 Sep 2021, at 19:02, Andrew MacLeod amacleod@redhat.com wrote:
On 9/27/21 11:39 AM, Maxim Kuvyrkov via Gcc wrote:
On 27 Sep 2021, at 16:52, Aldy Hernandez aldyh@redhat.com wrote:
[CCing Jeff and list for broader audience]
On 9/27/21 2:53 PM, Maxim Kuvyrkov wrote:
Hi Aldy, Your patch seems to slow down 471.omnetpp by 8% at -O3. Could you please take a look if this is something that could be easily fixed?
First of all, thanks for chasing this down. It's incredibly useful to have these types of bug reports.
Thanks, Aldy, this is music to my ears :-).
We have built this automated benchmarking CI that bisects code-speed and code-size regressions down to a single commit. It is still work-in-progress, and I’m forwarding these reports to patch authors, whose patches caused regressions. If GCC community finds these useful, we can also setup posting to one of GCC’s mailing lists.
I second that this sort of thing is incredibly useful. I don't suppose its easy to do the reverse?... let patch authors know when they've caused a significant improvement? :-) That would be much less common I suspect, so perhaps not worth it :-)
We do this occasionally, when identifying a regression in a patch revert commit :-). Seriously, though, it’s an easy enough code-change to the metric, but we are maxing out our benchmarking capacity with current configuration matrix.
Its certainly very useful when we are making a wholesale change to a pass which we think is beneficial, but aren't sure.
And a followup question... Sometimes we have no good way of determining the widespread run-time effects of a change. You seem to be running SPEC/other things continuously then?
We continuously run SPEC CPU2006 on {arm,aarch64}-{-Os/-O2/-O3}-{no LTO/LTO} matrix for GNU and LLVM toolchains.
In the GNU toolchain we track master branches and latest-release branches of Binutils, GCC and Glibc — and detect code-speed and code-size regressions across all toolchain components.
Does it run like once a day/some-time-period, and if you note a regression, narrow it down?
Configurations that track master branches have 3-day intervals. Configurations that track release branches — 6 days. If a regression is detected it is narrowed down to component first — binutils, gcc or glibc — and then the commit range of the component is bisected down to a specific commit. All. Done. Automatically.
I will make a presentation on this CI at the next GNU Tools Cauldron.
Regardless, I think it could be very useful to be able to see the results of anything you do run at whatever frequency it happens.
Thanks!
-- Maxim Kuvyrkov https://www.linaro.org