Thanks for looking into this, Maxim!
On Mon, Feb 8, 2021 at 1:15 AM Maxim Kuvyrkov maxim.kuvyrkov@linaro.org wrote:
On Feb 7, 2021, at 6:10 AM, Galina Kistanova gkistanova@gmail.com
wrote:
Hello Linaro Toolchain Working Group,
Hi Galina,
Thanks for looking out for buildbot problems. We are doing our best to keep ARM and AArch64 builders and workers green and happy, but things do slip through the cracks occasionally.
libcxx-libcxxabi-libunwind-armv8-linux has been red for 5 days. clang-cmake-aarch64-global-isel has been red for 10 days. clang-cmake-aarch64-quick has been red for 10 days. clang-cmake-armv7-global-isel has been red for 10 days. clang-cmake-armv7-lnt has been red for 10 days. clang-cmake-aarch64-lld has been red for 11 days. clang-cmake-armv8-lld has been red for 11 days. clang-cmake-thumbv7-full-sh has been red for 11 days. clang-cmake-armv7-full has been red for 11 days.
Is anybody looking at this?
Yes. The underlying problem was that we removed python2 packages from our buildbot docker images, and that caused all builders that use on virtualenv (which uses python2 by default) to break. We've fixed that late last week, but some of the builders accumulated new failures that we are going through now.
I also noticed that some of your workers are less reliable than others.
For
example, inaro-tk1-09 worker was last time producing a green build 2
months
ago - http://lab.llvm.org:8011/#/workers/2?numbuilds=500
I'm removing the linaro-tk1-09 worker from the production buildbot for
now.
Please feel free to connect it to the staging and make the builder
reliably
green. Then it could be returned back to the production.
This wasn't on our radar. We'll investigate -- thanks!
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have been
red
for the least 4 months. I'm removing them from the production buildbot.
You
can return them back to production after they are reliably green in the staging.
These ones had several compounding problems over the last 4 months. We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org