Successfully identified regression in *llvm* in CI configuration tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2. So far, this commit has regressed CI configurations: - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2

Culprit: <cut> commit d39d3a327b1303012370e47d991459ffbfce45ef Author: Peyton, Jonathan L jonathan.l.peyton@intel.com Date: Fri Aug 20 16:06:13 2021 -0500

[OpenMP][test] fix omp_get_wtime.c test to be more accommodating

The omp_get_wtime.c test fails intermittently if the recorded times are off by too much which can happen when many tests are run in parallel.

Instead of failing if one timing is a little off, take average of 100 timings minus the 10 worst.

Differential Revision: https://reviews.llvm.org/D108488 </cut>

Results regressed to (for first_bad == d39d3a327b1303012370e47d991459ffbfce45ef) # reset_artifacts: -10 # build_abe binutils: -9 # build_abe stage1 -- --set gcc_override_configure=--disable-libsanitizer: -8 # build_abe linux: -7 # build_abe glibc: -6 # build_abe stage2 -- --set gcc_override_configure=--disable-libsanitizer: -5 # build_llvm true: -3 # true: 0 # benchmark -- -O2 artifacts/build-d39d3a327b1303012370e47d991459ffbfce45ef/results_id: 1 # 447.dealII,dealII_base.default regressed by 105

from (for last_good == f77174d4b8cfba3c0a53c78e53edbbaf57e37fc5) # reset_artifacts: -10 # build_abe binutils: -9 # build_abe stage1 -- --set gcc_override_configure=--disable-libsanitizer: -8 # build_abe linux: -7 # build_abe glibc: -6 # build_abe stage2 -- --set gcc_override_configure=--disable-libsanitizer: -5 # build_llvm true: -3 # true: 0 # benchmark -- -O2 artifacts/build-f77174d4b8cfba3c0a53c78e53edbbaf57e37fc5/results_id: 1

Artifacts of last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Results ID of last_good: tx1_64/tcwg_bmk_llvm_tx1/bisect-llvm-master-aarch64-spec2k6-O2/4734 Artifacts of first_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Results ID of first_bad: tx1_64/tcwg_bmk_llvm_tx1/bisect-llvm-master-aarch64-spec2k6-O2/4757 Build top page/logs: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...

Configuration details:

Reproduce builds: <cut> mkdir investigate-llvm-d39d3a327b1303012370e47d991459ffbfce45ef cd investigate-llvm-d39d3a327b1303012370e47d991459ffbfce45ef

git clone https://git.linaro.org/toolchain/jenkins-scripts

mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... --fail chmod +x artifacts/test.sh

# Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh

# Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /llvm/ ./ ./bisect/baseline/

cd llvm

# Reproduce first_bad build git checkout --detach d39d3a327b1303012370e47d991459ffbfce45ef ../artifacts/test.sh

# Reproduce last_good build git checkout --detach f77174d4b8cfba3c0a53c78e53edbbaf57e37fc5 ../artifacts/test.sh

cd .. </cut>

History of pending regressions and results: https://git.linaro.org/toolchain/ci/base-artifacts.git/log/?h=linaro-local/c...

Artifacts: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a... Build log: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-a...

Full commit (up to 1000 lines): <cut> commit d39d3a327b1303012370e47d991459ffbfce45ef Author: Peyton, Jonathan L jonathan.l.peyton@intel.com Date: Fri Aug 20 16:06:13 2021 -0500

[OpenMP][test] fix omp_get_wtime.c test to be more accommodating

The omp_get_wtime.c test fails intermittently if the recorded times are off by too much which can happen when many tests are run in parallel.

Instead of failing if one timing is a little off, take average of 100 timings minus the 10 worst.

Differential Revision: https://reviews.llvm.org/D108488 --- openmp/runtime/test/api/omp_get_wtime.c | 75 ++++++++++++++++++++++++++------- 1 file changed, 59 insertions(+), 16 deletions(-)

diff --git a/openmp/runtime/test/api/omp_get_wtime.c b/openmp/runtime/test/api/omp_get_wtime.c index e2bb211e0ce4..a862e07fc5a2 100644 --- a/openmp/runtime/test/api/omp_get_wtime.c +++ b/openmp/runtime/test/api/omp_get_wtime.c @@ -4,30 +4,73 @@ #include "omp_testsuite.h" #include "omp_my_sleep.h"

-int test_omp_get_wtime() -{ +#define NTIMES 100 + +// This is the error % threshold. Be generous with the error threshold since +// this test may be run in parallel with many other tests it may throw off the +// sleep timing. +#define THRESHOLD 33.0 + +double test_omp_get_wtime(double desired_wait_time) { double start; double end; - double measured_time; - double wait_time = 0.2; start = 0; end = 0; start = omp_get_wtime(); - my_sleep (wait_time); + my_sleep(desired_wait_time); end = omp_get_wtime(); - measured_time = end-start; - return ((measured_time > 0.97 * wait_time) && (measured_time < 1.03 * wait_time)) ; + return end - start; }

-int main() -{ - int i; - int num_failed=0; +int compare_times(const void *lhs, const void *rhs) { + const double *a = (const double *)lhs; + const double *b = (const double *)rhs; + return *a - *b; +} + +int main() { + int i, final_count; + double percent_off; + double *begin, *end, *ptr; + double wait_time = 0.01; + double average = 0.0; + double n = 0.0; + double *times = (double *)malloc(sizeof(double) * NTIMES); + + // Get each timing + for (i = 0; i < NTIMES; i++) { + times[i] = test_omp_get_wtime(wait_time); + } + + // Remove approx the "worst" tenth of the timings + qsort(times, NTIMES, sizeof(double), compare_times); + begin = times; + end = times + NTIMES; + for (i = 0; i < NTIMES / 10; ++i) { + if (i % 2 == 0) + begin++; + else + end--; + } + + // Get the average of the remaining timings + for (ptr = begin, final_count = 0; ptr != end; ++ptr, ++final_count) + average += times[i]; + average /= (double)final_count; + free(times); + + // Calculate the percent off of desired wait time + percent_off = (average - wait_time) / wait_time * 100.0; + // Should always be positive, but just in case + if (percent_off < 0) + percent_off = -percent_off;

- for(i = 0; i < REPETITIONS; i++) { - if(!test_omp_get_wtime()) { - num_failed++; - } + if (percent_off > (double)THRESHOLD) { + fprintf(stderr, "error: average of %d runs (%lf) is of by %lf%%\n", NTIMES, + average, percent_off); + return EXIT_FAILURE; } - return num_failed; + printf("pass: average of %d runs (%lf) is only off by %lf%%\n", NTIMES, + average, percent_off); + return EXIT_SUCCESS; } </cut>