After llvm commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d Author: David Blaikie dblaikie@gmail.com
Add missing header
the following benchmarks slowed down by more than 2%: - 433.milc slowed down by 4% from 12427 to 12916 perf samples
Below reproducer instructions can be used to re-build both "first_bad" and "last_good" cross-toolchains used in this bisection. Naturally, the scripts will fail when triggerring benchmarking jobs if you don't have access to Linaro TCWG CI.
For your convenience, we have uploaded tarballs with pre-processed source and assembly files at: - First_bad save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... - Last_good save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... - Baseline save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Configuration: - Benchmark: SPEC CPU2006 - Toolchain: Clang + Glibc + LLVM Linker - Version: all components were built from their tip of trunk - Target: aarch64-linux-gnu - Compiler flags: -O2 -flto - Hardware: NVidia TX1 4x Cortex-A57
This benchmarking CI is work-in-progress, and we welcome feedback and suggestions at linaro-toolchain@lists.linaro.org . In our improvement plans is to add support for SPEC CPU2017 benchmarks and provide "perf report/annotate" data behind these reports.
THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT.
This commit has regressed these CI configurations: - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2_LTO
First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Reproduce builds: <cut> mkdir investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d cd investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d
# Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts
# Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail chmod +x artifacts/test.sh
# Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh
# Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /llvm/ ./ ./bisect/baseline/
cd llvm
# Reproduce first_bad build git checkout --detach bd4c6a476fd037fb07a1c484f75d93ee40713d3d ../artifacts/test.sh
# Reproduce last_good build git checkout --detach 7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7 ../artifacts/test.sh
cd .. </cut>
Full commit (up to 1000 lines): <cut> commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d Author: David Blaikie dblaikie@gmail.com Date: Mon Nov 29 16:29:25 2021 -0800
Add missing header --- llvm/lib/Demangle/DLangDemangle.cpp | 1 + 1 file changed, 1 insertion(+)
diff --git a/llvm/lib/Demangle/DLangDemangle.cpp b/llvm/lib/Demangle/DLangDemangle.cpp index faf91b239490..f380aa90035e 100644 --- a/llvm/lib/Demangle/DLangDemangle.cpp +++ b/llvm/lib/Demangle/DLangDemangle.cpp @@ -17,6 +17,7 @@ #include "llvm/Demangle/StringView.h" #include "llvm/Demangle/Utility.h"
+#include <cctype> #include <cstring> #include <limits>
</cut>
Seems... unlikely this change had a performance impact.
Also is this email meant to be sent to public contributors like myself, or only intended for some Linaro folks?
On Sun, Dec 5, 2021 at 6:18 AM ci_notify@linaro.org wrote:
After llvm commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d Author: David Blaikie dblaikie@gmail.com
Add missing headerthe following benchmarks slowed down by more than 2%:
- 433.milc slowed down by 4% from 12427 to 12916 perf samples
Below reproducer instructions can be used to re-build both "first_bad" and "last_good" cross-toolchains used in this bisection. Naturally, the scripts will fail when triggerring benchmarking jobs if you don't have access to Linaro TCWG CI.
For your convenience, we have uploaded tarballs with pre-processed source and assembly files at:
- First_bad save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
- Last_good save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
- Baseline save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Configuration:
- Benchmark: SPEC CPU2006
- Toolchain: Clang + Glibc + LLVM Linker
- Version: all components were built from their tip of trunk
- Target: aarch64-linux-gnu
- Compiler flags: -O2 -flto
- Hardware: NVidia TX1 4x Cortex-A57
This benchmarking CI is work-in-progress, and we welcome feedback and suggestions at linaro-toolchain@lists.linaro.org . In our improvement plans is to add support for SPEC CPU2017 benchmarks and provide "perf report/annotate" data behind these reports.
THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT.
This commit has regressed these CI configurations:
- tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2_LTO
First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Reproduce builds:
<cut> mkdir investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d cd investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d
# Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts
# Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail chmod +x artifacts/test.sh
# Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh
# Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /llvm/ ./ ./bisect/baseline/
cd llvm
# Reproduce first_bad build git checkout --detach bd4c6a476fd037fb07a1c484f75d93ee40713d3d ../artifacts/test.sh
# Reproduce last_good build git checkout --detach 7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7 ../artifacts/test.sh
cd ..
</cut>
Full commit (up to 1000 lines):
<cut> commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d Author: David Blaikie <dblaikie@gmail.com> Date: Mon Nov 29 16:29:25 2021 -0800
Add missing header
llvm/lib/Demangle/DLangDemangle.cpp | 1 + 1 file changed, 1 insertion(+)
diff --git a/llvm/lib/Demangle/DLangDemangle.cpp b/llvm/lib/Demangle/DLangDemangle.cpp index faf91b239490..f380aa90035e 100644 --- a/llvm/lib/Demangle/DLangDemangle.cpp +++ b/llvm/lib/Demangle/DLangDemangle.cpp @@ -17,6 +17,7 @@ #include "llvm/Demangle/StringView.h" #include "llvm/Demangle/Utility.h"
+#include <cctype> #include <cstring> #include <limits>
</cut>
Hi David,
This is a false positive, sorry for the noise. Our CI bisects performance regressions down to a single commit, and notifies patch authors about significant regressions.
Since this is benchmarking CI, some noise is expected. Apparently, 433.milc started to show bi-modal performance, and bisection [mistakenly] converged on this patch. We work to reduce benchmarking noise and otherwise improve benchmarking CI.
Regards!
-- Maxim Kuvyrkov https://www.linaro.org
On 7 Dec 2021, at 06:12, David Blaikie dblaikie@gmail.com wrote:
Seems... unlikely this change had a performance impact.
Also is this email meant to be sent to public contributors like myself, or only intended for some Linaro folks?
On Sun, Dec 5, 2021 at 6:18 AM ci_notify@linaro.org wrote:
After llvm commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d Author: David Blaikie dblaikie@gmail.com
Add missing header
the following benchmarks slowed down by more than 2%:
- 433.milc slowed down by 4% from 12427 to 12916 perf samples
Below reproducer instructions can be used to re-build both "first_bad" and "last_good" cross-toolchains used in this bisection. Naturally, the scripts will fail when triggerring benchmarking jobs if you don't have access to Linaro TCWG CI.
For your convenience, we have uploaded tarballs with pre-processed source and assembly files at:
- First_bad save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
- Last_good save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
- Baseline save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Configuration:
- Benchmark: SPEC CPU2006
- Toolchain: Clang + Glibc + LLVM Linker
- Version: all components were built from their tip of trunk
- Target: aarch64-linux-gnu
- Compiler flags: -O2 -flto
- Hardware: NVidia TX1 4x Cortex-A57
This benchmarking CI is work-in-progress, and we welcome feedback and suggestions at linaro-toolchain@lists.linaro.org . In our improvement plans is to add support for SPEC CPU2017 benchmarks and provide "perf report/annotate" data behind these reports.
THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT.
This commit has regressed these CI configurations:
- tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2_LTO
First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Reproduce builds:
<cut> mkdir investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d cd investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d
# Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts
# Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail chmod +x artifacts/test.sh
# Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh
# Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /llvm/ ./ ./bisect/baseline/
cd llvm
# Reproduce first_bad build git checkout --detach bd4c6a476fd037fb07a1c484f75d93ee40713d3d ../artifacts/test.sh
# Reproduce last_good build git checkout --detach 7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7 ../artifacts/test.sh
cd ..
</cut>
Full commit (up to 1000 lines):
<cut> commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d Author: David Blaikie <dblaikie@gmail.com> Date: Mon Nov 29 16:29:25 2021 -0800
Add missing header
llvm/lib/Demangle/DLangDemangle.cpp | 1 + 1 file changed, 1 insertion(+)
diff --git a/llvm/lib/Demangle/DLangDemangle.cpp b/llvm/lib/Demangle/DLangDemangle.cpp index faf91b239490..f380aa90035e 100644 --- a/llvm/lib/Demangle/DLangDemangle.cpp +++ b/llvm/lib/Demangle/DLangDemangle.cpp @@ -17,6 +17,7 @@ #include "llvm/Demangle/StringView.h" #include "llvm/Demangle/Utility.h"
+#include <cctype> #include <cstring> #include <limits>
</cut>
linaro-toolchain mailing list linaro-toolchain@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-toolchain
OK , thanks - appreciate all the hard work of making performance regression analysis & continued improvement to reduce false positives!
On Tue, Dec 7, 2021 at 4:03 AM Maxim Kuvyrkov maxim.kuvyrkov@linaro.org wrote:
Hi David,
This is a false positive, sorry for the noise. Our CI bisects performance regressions down to a single commit, and notifies patch authors about significant regressions.
Since this is benchmarking CI, some noise is expected. Apparently, 433.milc started to show bi-modal performance, and bisection [mistakenly] converged on this patch. We work to reduce benchmarking noise and otherwise improve benchmarking CI.
Regards!
-- Maxim Kuvyrkov https://www.linaro.org
On 7 Dec 2021, at 06:12, David Blaikie dblaikie@gmail.com wrote:
Seems... unlikely this change had a performance impact.
Also is this email meant to be sent to public contributors like myself,
or
only intended for some Linaro folks?
On Sun, Dec 5, 2021 at 6:18 AM ci_notify@linaro.org wrote:
After llvm commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d Author: David Blaikie dblaikie@gmail.com
Add missing header
the following benchmarks slowed down by more than 2%:
- 433.milc slowed down by 4% from 12427 to 12916 perf samples
Below reproducer instructions can be used to re-build both "first_bad"
and
"last_good" cross-toolchains used in this bisection. Naturally, the scripts will fail when triggerring benchmarking jobs if you don't have access to Linaro TCWG CI.
For your convenience, we have uploaded tarballs with pre-processed
source
and assembly files at:
- First_bad save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
- Last_good save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
- Baseline save-temps:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Configuration:
- Benchmark: SPEC CPU2006
- Toolchain: Clang + Glibc + LLVM Linker
- Version: all components were built from their tip of trunk
- Target: aarch64-linux-gnu
- Compiler flags: -O2 -flto
- Hardware: NVidia TX1 4x Cortex-A57
This benchmarking CI is work-in-progress, and we welcome feedback and suggestions at linaro-toolchain@lists.linaro.org . In our improvement plans is to add support for SPEC CPU2017 benchmarks and provide "perf report/annotate" data behind these reports.
THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT.
This commit has regressed these CI configurations:
- tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2_LTO
First_bad build:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Last_good build:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Baseline build:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Even more details:
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
Reproduce builds:
<cut> mkdir investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d cd investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d
# Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts
# Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
--fail curl -o artifacts/manifests/build-parameters.sh
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
--fail curl -o artifacts/test.sh
https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...
--fail chmod +x artifacts/test.sh
# Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh
# Save baseline build state (which is then restored in
artifacts/test.sh)
mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude
/artifacts/
--exclude /llvm/ ./ ./bisect/baseline/
cd llvm
# Reproduce first_bad build git checkout --detach bd4c6a476fd037fb07a1c484f75d93ee40713d3d ../artifacts/test.sh
# Reproduce last_good build git checkout --detach 7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7 ../artifacts/test.sh
cd ..
</cut>
Full commit (up to 1000 lines):
<cut> commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d Author: David Blaikie <dblaikie@gmail.com> Date: Mon Nov 29 16:29:25 2021 -0800
Add missing header
llvm/lib/Demangle/DLangDemangle.cpp | 1 + 1 file changed, 1 insertion(+)
diff --git a/llvm/lib/Demangle/DLangDemangle.cpp b/llvm/lib/Demangle/DLangDemangle.cpp index faf91b239490..f380aa90035e 100644 --- a/llvm/lib/Demangle/DLangDemangle.cpp +++ b/llvm/lib/Demangle/DLangDemangle.cpp @@ -17,6 +17,7 @@ #include "llvm/Demangle/StringView.h" #include "llvm/Demangle/Utility.h"
+#include <cctype> #include <cstring> #include <limits>
</cut>
linaro-toolchain mailing list linaro-toolchain@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-toolchain
linaro-toolchain@lists.linaro.org
-
ci_notify＠linaro.org -
David Blaikie -
Maxim Kuvyrkov