Fwd: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - failed test (failure) (main)
---------- Forwarded message --------- From: David Blaikie dblaikie@gmail.com Date: Thu, Jul 25, 2024 at 6:43 PM Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - failed test (failure) (main) To: llvm.buildmaster@lab.llvm.org Cc: llvm.buildmaster@lab.llvm.org
This bot seems to be sending fail-mail when it's already red. My understanding was that fail mail only went out on green->red transitions. Is that understanding incorrect? Should this be reconfigured to match that understanding/not send fail mail when it's already red?
On Thu, Jul 25, 2024 at 4:10 PM llvm.buildmaster@lab.llvm.org wrote:
The Buildbot has detected a failed build on builder clang-aarch64-sve-vls-2stage while building clang.
Full details are available at: https://lab.llvm.org/buildbot/#/builders/4/builds/943
Worker for this Build: linaro-g3-02 Blamelist: David Blaikie dblaikie@gmail.com
BUILD FAILED: failed test (failure)
Step 18 (test-suite) failure: test (failure) ... [ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_1_f90 cd /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression && /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size --format=sysv /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size make[2]: Leaving directory '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' [ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_4_f90 make[1]: Leaving directory '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' make: *** [Makefile:136: all] Error 2 2024-07-25 23:09:27 INFO: Testing... 2024-07-25 23:09:27 INFO: Execute: /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit -v -j 32 /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build -o /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json -- Testing: 9770 tests, 32 workers -- NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test (1 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test (2 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test (3 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test (4 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test (5 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test (6 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test (7 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test (8 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test (9 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031' is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test (10 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED ******************** Executable '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39' is missing
Sincerely, LLVM Buildbot
I don't see any special configuration for this builder that would make it do this. ( https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1... )
It's unlikely that this build finished before previous jobs because it only takes an hour and we have several builds in the hours before it. I thought it may also be the first build on that worker that failed, but this is also not the case (and if it were, this would be a buildbot bug since it's supposed to look at the builder as a whole, not specific workers).
It is a single commit blame list, and I know we started posting results to GitHub in these cases. Presumably those have to also be green -> red transitions, but perhaps something slipped through here and because there was no PR to associate this change with, it sent an email instead?
Thanks, David Spickett.
On Fri, 26 Jul 2024 at 03:03, David Blaikie dblaikie@gmail.com wrote:
---------- Forwarded message --------- From: David Blaikie dblaikie@gmail.com Date: Thu, Jul 25, 2024 at 6:43 PM Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - failed test (failure) (main) To: llvm.buildmaster@lab.llvm.org Cc: llvm.buildmaster@lab.llvm.org
This bot seems to be sending fail-mail when it's already red. My understanding was that fail mail only went out on green->red transitions. Is that understanding incorrect? Should this be reconfigured to match that understanding/not send fail mail when it's already red?
On Thu, Jul 25, 2024 at 4:10 PM llvm.buildmaster@lab.llvm.org wrote:
The Buildbot has detected a failed build on builder clang-aarch64-sve-vls-2stage while building clang.
Full details are available at: https://lab.llvm.org/buildbot/#/builders/4/builds/943
Worker for this Build: linaro-g3-02 Blamelist: David Blaikie dblaikie@gmail.com
BUILD FAILED: failed test (failure)
Step 18 (test-suite) failure: test (failure) ... [ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_1_f90 cd
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression
&&
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size
--format=sysv
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size
make[2]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
[ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_4_f90 make[1]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
make: *** [Makefile:136: all] Error 2 2024-07-25 23:09:27 INFO: Testing... 2024-07-25 23:09:27 INFO: Execute:
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit
-v -j 32
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build
-o
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json
-- Testing: 9770 tests, 32 workers -- NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test (1
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test (2
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test (3 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test (4 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test (5 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test (6
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test (7 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test (8 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test (9
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test (10 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39'
is missing
Sincerely, LLVM Buildbot
linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.org
Any update on this?
On Fri, Jul 26, 2024 at 1:28 AM David Spickett david.spickett@linaro.org wrote:
I don't see any special configuration for this builder that would make it do this. ( https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1... )
It's unlikely that this build finished before previous jobs because it only takes an hour and we have several builds in the hours before it. I thought it may also be the first build on that worker that failed, but this is also not the case (and if it were, this would be a buildbot bug since it's supposed to look at the builder as a whole, not specific workers).
It is a single commit blame list, and I know we started posting results to GitHub in these cases. Presumably those have to also be green -> red transitions, but perhaps something slipped through here and because there was no PR to associate this change with, it sent an email instead?
Thanks, David Spickett.
On Fri, 26 Jul 2024 at 03:03, David Blaikie dblaikie@gmail.com wrote:
---------- Forwarded message --------- From: David Blaikie dblaikie@gmail.com Date: Thu, Jul 25, 2024 at 6:43 PM Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - failed test (failure) (main) To: llvm.buildmaster@lab.llvm.org Cc: llvm.buildmaster@lab.llvm.org
This bot seems to be sending fail-mail when it's already red. My understanding was that fail mail only went out on green->red transitions. Is that understanding incorrect? Should this be reconfigured to match that understanding/not send fail mail when it's already red?
On Thu, Jul 25, 2024 at 4:10 PM llvm.buildmaster@lab.llvm.org wrote:
The Buildbot has detected a failed build on builder clang-aarch64-sve-vls-2stage while building clang.
Full details are available at: https://lab.llvm.org/buildbot/#/builders/4/builds/943
Worker for this Build: linaro-g3-02 Blamelist: David Blaikie dblaikie@gmail.com
BUILD FAILED: failed test (failure)
Step 18 (test-suite) failure: test (failure) ... [ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_1_f90 cd
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression
&&
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size
--format=sysv
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size
make[2]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
[ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_4_f90 make[1]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
make: *** [Makefile:136: all] Error 2 2024-07-25 23:09:27 INFO: Testing... 2024-07-25 23:09:27 INFO: Execute:
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit
-v -j 32
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build
-o
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json
-- Testing: 9770 tests, 32 workers -- NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test (1
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test (2
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test (3
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test (4 of 9770) ******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test (5
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test (6
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test (7
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test (8
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test (9
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test (10
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39'
is missing
Sincerely, LLVM Buildbot
linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.org
I've looked at the build status reporter, which I think is https://github.com/llvm/llvm-zorg/blob/ec94a29bfb0c247f0123c4efaf080a9b215d4.... I don't see anything wrong with that, and nothing that it removed from Buildbot's version of that function looks like it would cause this problem.
If the previous result was somehow still the last green build, this could happen. However I don't see how that could be the case as there is a finished build on the same worker prior to this one. So even if every other worker was still busy, there's at least one previously failed result. My guess from the Buildbot code is that prev_build["result"] will be None if the build has not finished, so even in this case it would return False and not send a message (though I suspect in progress builds aren't part of these checks or perhaps it waits for the previous one to finish, not sure).
I do see that the build prior to yours finished later, and some of the builds immediately after the green build have some unusual timings where earlier builds finish after later builds. This is expected with the 3 workers we have, but perhaps could have affected when the results get written to the database? Perhaps every build between the last green and yours was notified and either no one said anything, or only builds with 1 commit were notified and didn't say anything.
The other 1 commit build is https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the author if they received anything. Though even if they did I'm not sure what we could conclude from that.
On Wed, 7 Aug 2024 at 02:12, David Blaikie dblaikie@gmail.com wrote:
Any update on this?
On Fri, Jul 26, 2024 at 1:28 AM David Spickett david.spickett@linaro.org wrote:
I don't see any special configuration for this builder that would make it do this. ( https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1... )
It's unlikely that this build finished before previous jobs because it only takes an hour and we have several builds in the hours before it. I thought it may also be the first build on that worker that failed, but this is also not the case (and if it were, this would be a buildbot bug since it's supposed to look at the builder as a whole, not specific workers).
It is a single commit blame list, and I know we started posting results to GitHub in these cases. Presumably those have to also be green -> red transitions, but perhaps something slipped through here and because there was no PR to associate this change with, it sent an email instead?
Thanks, David Spickett.
On Fri, 26 Jul 2024 at 03:03, David Blaikie dblaikie@gmail.com wrote:
---------- Forwarded message --------- From: David Blaikie dblaikie@gmail.com Date: Thu, Jul 25, 2024 at 6:43 PM Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - failed test (failure) (main) To: llvm.buildmaster@lab.llvm.org Cc: llvm.buildmaster@lab.llvm.org
This bot seems to be sending fail-mail when it's already red. My understanding was that fail mail only went out on green->red transitions. Is that understanding incorrect? Should this be reconfigured to match that understanding/not send fail mail when it's already red?
On Thu, Jul 25, 2024 at 4:10 PM llvm.buildmaster@lab.llvm.org wrote:
The Buildbot has detected a failed build on builder clang-aarch64-sve-vls-2stage while building clang.
Full details are available at: https://lab.llvm.org/buildbot/#/builders/4/builds/943
Worker for this Build: linaro-g3-02 Blamelist: David Blaikie dblaikie@gmail.com
BUILD FAILED: failed test (failure)
Step 18 (test-suite) failure: test (failure) ... [ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_1_f90 cd
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression
&&
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size
--format=sysv
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size
make[2]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
[ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_4_f90 make[1]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
make: *** [Makefile:136: all] Error 2 2024-07-25 23:09:27 INFO: Testing... 2024-07-25 23:09:27 INFO: Execute:
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit
-v -j 32
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build
-o
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json
-- Testing: 9770 tests, 32 workers -- NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test
(1 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test
(2 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test (3
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test (4
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test (5
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test
(6 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test (7
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test (8
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test
(9 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test (10
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39'
is missing
Sincerely, LLVM Buildbot
linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.org
The other 1 commit build is
https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the author if they received anything. Though even if they did I'm not sure what we could conclude from that.
They did not receive any notifications. And I'm out of ideas at this point, maybe someone else can spot something. There is https://github.com/llvm/llvm-zorg/issues/219 which is along the same lines but no solid evidence that it's the same cause.
David thanks for reporting this, even if we don't manage to figure it out it's still better to know about these than not.
On Thu, 8 Aug 2024 at 16:36, David Spickett david.spickett@linaro.org wrote:
I've looked at the build status reporter, which I think is https://github.com/llvm/llvm-zorg/blob/ec94a29bfb0c247f0123c4efaf080a9b215d4.... I don't see anything wrong with that, and nothing that it removed from Buildbot's version of that function looks like it would cause this problem.
If the previous result was somehow still the last green build, this could happen. However I don't see how that could be the case as there is a finished build on the same worker prior to this one. So even if every other worker was still busy, there's at least one previously failed result. My guess from the Buildbot code is that prev_build["result"] will be None if the build has not finished, so even in this case it would return False and not send a message (though I suspect in progress builds aren't part of these checks or perhaps it waits for the previous one to finish, not sure).
I do see that the build prior to yours finished later, and some of the builds immediately after the green build have some unusual timings where earlier builds finish after later builds. This is expected with the 3 workers we have, but perhaps could have affected when the results get written to the database? Perhaps every build between the last green and yours was notified and either no one said anything, or only builds with 1 commit were notified and didn't say anything.
The other 1 commit build is https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the author if they received anything. Though even if they did I'm not sure what we could conclude from that.
On Wed, 7 Aug 2024 at 02:12, David Blaikie dblaikie@gmail.com wrote:
Any update on this?
On Fri, Jul 26, 2024 at 1:28 AM David Spickett david.spickett@linaro.org wrote:
I don't see any special configuration for this builder that would make it do this. ( https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1... )
It's unlikely that this build finished before previous jobs because it only takes an hour and we have several builds in the hours before it. I thought it may also be the first build on that worker that failed, but this is also not the case (and if it were, this would be a buildbot bug since it's supposed to look at the builder as a whole, not specific workers).
It is a single commit blame list, and I know we started posting results to GitHub in these cases. Presumably those have to also be green -> red transitions, but perhaps something slipped through here and because there was no PR to associate this change with, it sent an email instead?
Thanks, David Spickett.
On Fri, 26 Jul 2024 at 03:03, David Blaikie dblaikie@gmail.com wrote:
---------- Forwarded message --------- From: David Blaikie dblaikie@gmail.com Date: Thu, Jul 25, 2024 at 6:43 PM Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - failed test (failure) (main) To: llvm.buildmaster@lab.llvm.org Cc: llvm.buildmaster@lab.llvm.org
This bot seems to be sending fail-mail when it's already red. My understanding was that fail mail only went out on green->red transitions. Is that understanding incorrect? Should this be reconfigured to match that understanding/not send fail mail when it's already red?
On Thu, Jul 25, 2024 at 4:10 PM llvm.buildmaster@lab.llvm.org wrote:
The Buildbot has detected a failed build on builder clang-aarch64-sve-vls-2stage while building clang.
Full details are available at: https://lab.llvm.org/buildbot/#/builders/4/builds/943
Worker for this Build: linaro-g3-02 Blamelist: David Blaikie dblaikie@gmail.com
BUILD FAILED: failed test (failure)
Step 18 (test-suite) failure: test (failure) ... [ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_1_f90 cd
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression
&&
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size
--format=sysv
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size
make[2]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
[ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_4_f90 make[1]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
make: *** [Makefile:136: all] Error 2 2024-07-25 23:09:27 INFO: Testing... 2024-07-25 23:09:27 INFO: Execute:
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit
-v -j 32
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build
-o
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json
-- Testing: 9770 tests, 32 workers -- NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test
(1 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test
(2 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test
(3 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test (4
of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test
(5 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test
(6 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test
(7 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test
(8 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test
(9 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test
(10 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39'
is missing
Sincerely, LLVM Buildbot
linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.org
Fair - any chance we can put some logging/more verbose output into the buildbot config so we might know more about how this happens next time?
On Fri, Aug 9, 2024 at 1:31 AM David Spickett david.spickett@linaro.org wrote:
The other 1 commit build is
https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the author if they received anything. Though even if they did I'm not sure what we could conclude from that.
They did not receive any notifications. And I'm out of ideas at this point, maybe someone else can spot something. There is https://github.com/llvm/llvm-zorg/issues/219 which is along the same lines but no solid evidence that it's the same cause.
David thanks for reporting this, even if we don't manage to figure it out it's still better to know about these than not.
On Thu, 8 Aug 2024 at 16:36, David Spickett david.spickett@linaro.org wrote:
I've looked at the build status reporter, which I think is https://github.com/llvm/llvm-zorg/blob/ec94a29bfb0c247f0123c4efaf080a9b215d4.... I don't see anything wrong with that, and nothing that it removed from Buildbot's version of that function looks like it would cause this problem.
If the previous result was somehow still the last green build, this could happen. However I don't see how that could be the case as there is a finished build on the same worker prior to this one. So even if every other worker was still busy, there's at least one previously failed result. My guess from the Buildbot code is that prev_build["result"] will be None if the build has not finished, so even in this case it would return False and not send a message (though I suspect in progress builds aren't part of these checks or perhaps it waits for the previous one to finish, not sure).
I do see that the build prior to yours finished later, and some of the builds immediately after the green build have some unusual timings where earlier builds finish after later builds. This is expected with the 3 workers we have, but perhaps could have affected when the results get written to the database? Perhaps every build between the last green and yours was notified and either no one said anything, or only builds with 1 commit were notified and didn't say anything.
The other 1 commit build is https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the author if they received anything. Though even if they did I'm not sure what we could conclude from that.
On Wed, 7 Aug 2024 at 02:12, David Blaikie dblaikie@gmail.com wrote:
Any update on this?
On Fri, Jul 26, 2024 at 1:28 AM David Spickett < david.spickett@linaro.org> wrote:
I don't see any special configuration for this builder that would make it do this. ( https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1... )
It's unlikely that this build finished before previous jobs because it only takes an hour and we have several builds in the hours before it. I thought it may also be the first build on that worker that failed, but this is also not the case (and if it were, this would be a buildbot bug since it's supposed to look at the builder as a whole, not specific workers).
It is a single commit blame list, and I know we started posting results to GitHub in these cases. Presumably those have to also be green -> red transitions, but perhaps something slipped through here and because there was no PR to associate this change with, it sent an email instead?
Thanks, David Spickett.
On Fri, 26 Jul 2024 at 03:03, David Blaikie dblaikie@gmail.com wrote:
---------- Forwarded message --------- From: David Blaikie dblaikie@gmail.com Date: Thu, Jul 25, 2024 at 6:43 PM Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - failed test (failure) (main) To: llvm.buildmaster@lab.llvm.org Cc: llvm.buildmaster@lab.llvm.org
This bot seems to be sending fail-mail when it's already red. My understanding was that fail mail only went out on green->red transitions. Is that understanding incorrect? Should this be reconfigured to match that understanding/not send fail mail when it's already red?
On Thu, Jul 25, 2024 at 4:10 PM llvm.buildmaster@lab.llvm.org wrote:
The Buildbot has detected a failed build on builder clang-aarch64-sve-vls-2stage while building clang.
Full details are available at: https://lab.llvm.org/buildbot/#/builders/4/builds/943
Worker for this Build: linaro-g3-02 Blamelist: David Blaikie dblaikie@gmail.com
BUILD FAILED: failed test (failure)
Step 18 (test-suite) failure: test (failure) ... [ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_1_f90 cd
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression
&&
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size
--format=sysv
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90
>
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size
make[2]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
[ 24%] Built target gfortran-regression-execute-regression__internal_readwrite_4_f90 make[1]: Leaving directory
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build'
make: *** [Makefile:136: all] Error 2 2024-07-25 23:09:27 INFO: Testing... 2024-07-25 23:09:27 INFO: Execute:
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit
-v -j 32
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build
-o
/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json
-- Testing: 9770 tests, 32 workers -- NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test
(1 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test
(2 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test
(3 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test
(4 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test
(5 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test
(6 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test
(7 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test
(8 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test
(9 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031'
is missing
NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test
(10 of
******************** TEST 'test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED
Executable
'/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39'
is missing
Sincerely, LLVM Buildbot
linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.org
There are some `log.msg(` calls in the code already, those would show up on the master's side. I don't know whether they're being saved or not. Galina do you know about that?
On Fri, 9 Aug 2024 at 18:13, David Blaikie dblaikie@gmail.com wrote:
Fair - any chance we can put some logging/more verbose output into the buildbot config so we might know more about how this happens next time?
On Fri, Aug 9, 2024 at 1:31 AM David Spickett david.spickett@linaro.org wrote:
The other 1 commit build is
https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the author if they received anything. Though even if they did I'm not sure what we could conclude from that.
They did not receive any notifications. And I'm out of ideas at this point, maybe someone else can spot something. There is https://github.com/llvm/llvm-zorg/issues/219 which is along the same lines but no solid evidence that it's the same cause.
David thanks for reporting this, even if we don't manage to figure it out it's still better to know about these than not.
On Thu, 8 Aug 2024 at 16:36, David Spickett david.spickett@linaro.org wrote:
I've looked at the build status reporter, which I think is https://github.com/llvm/llvm-zorg/blob/ec94a29bfb0c247f0123c4efaf080a9b215d4.... I don't see anything wrong with that, and nothing that it removed from Buildbot's version of that function looks like it would cause this problem.
If the previous result was somehow still the last green build, this could happen. However I don't see how that could be the case as there is a finished build on the same worker prior to this one. So even if every other worker was still busy, there's at least one previously failed result. My guess from the Buildbot code is that prev_build["result"] will be None if the build has not finished, so even in this case it would return False and not send a message (though I suspect in progress builds aren't part of these checks or perhaps it waits for the previous one to finish, not sure).
I do see that the build prior to yours finished later, and some of the builds immediately after the green build have some unusual timings where earlier builds finish after later builds. This is expected with the 3 workers we have, but perhaps could have affected when the results get written to the database? Perhaps every build between the last green and yours was notified and either no one said anything, or only builds with 1 commit were notified and didn't say anything.
The other 1 commit build is https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the author if they received anything. Though even if they did I'm not sure what we could conclude from that.
On Wed, 7 Aug 2024 at 02:12, David Blaikie dblaikie@gmail.com wrote:
Any update on this?
On Fri, Jul 26, 2024 at 1:28 AM David Spickett < david.spickett@linaro.org> wrote:
I don't see any special configuration for this builder that would make it do this. ( https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1... )
It's unlikely that this build finished before previous jobs because it only takes an hour and we have several builds in the hours before it. I thought it may also be the first build on that worker that failed, but this is also not the case (and if it were, this would be a buildbot bug since it's supposed to look at the builder as a whole, not specific workers).
It is a single commit blame list, and I know we started posting results to GitHub in these cases. Presumably those have to also be green -> red transitions, but perhaps something slipped through here and because there was no PR to associate this change with, it sent an email instead?
Thanks, David Spickett.
On Fri, 26 Jul 2024 at 03:03, David Blaikie dblaikie@gmail.com wrote:
---------- Forwarded message --------- From: David Blaikie dblaikie@gmail.com Date: Thu, Jul 25, 2024 at 6:43 PM Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage
failed test (failure) (main) To: llvm.buildmaster@lab.llvm.org Cc: llvm.buildmaster@lab.llvm.org
This bot seems to be sending fail-mail when it's already red. My understanding was that fail mail only went out on green->red transitions. Is that understanding incorrect? Should this be reconfigured to match that understanding/not send fail mail when it's already red?
On Thu, Jul 25, 2024 at 4:10 PM llvm.buildmaster@lab.llvm.org wrote:
> The Buildbot has detected a failed build on builder > clang-aarch64-sve-vls-2stage while building clang. > > Full details are available at: > https://lab.llvm.org/buildbot/#/builders/4/builds/943 > > Worker for this Build: linaro-g3-02 > Blamelist: > David Blaikie dblaikie@gmail.com > > BUILD FAILED: failed test (failure) > > Step 18 (test-suite) failure: test (failure) > ... > [ 24%] Built target > gfortran-regression-execute-regression__internal_readwrite_1_f90 > cd > /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression > && > /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size > --format=sysv > /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90 > > > /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size > make[2]: Leaving directory > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' > [ 24%] Built target > gfortran-regression-execute-regression__internal_readwrite_4_f90 > make[1]: Leaving directory > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' > make: *** [Makefile:136: all] Error 2 > 2024-07-25 23:09:27 INFO: Testing... > 2024-07-25 23:09:27 INFO: Execute: > /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit > -v -j 32 > /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build > -o > /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json > -- Testing: 9770 tests, 32 workers -- > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test (1 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test (2 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test (3 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test (4 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test (5 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test (6 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test (7 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test (8 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test (9 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031' > is missing > ******************** > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test (10 of > 9770) > ******************** TEST 'test-suite :: > Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED
> Executable > '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39' > is missing > ******************** > > Sincerely, > LLVM Buildbot > > _______________________________________________ linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.org
linaro-toolchain@lists.linaro.org
-
David Blaikie -
David Spickett