On Feb 7, 2021, at 6:10 AM, Galina Kistanova gkistanova@gmail.com wrote:
Hello Linaro Toolchain Working Group,
Hi Galina,
Thanks for looking out for buildbot problems. We are doing our best to keep ARM and AArch64 builders and workers green and happy, but things do slip through the cracks occasionally.
libcxx-libcxxabi-libunwind-armv8-linux has been red for 5 days. clang-cmake-aarch64-global-isel has been red for 10 days. clang-cmake-aarch64-quick has been red for 10 days. clang-cmake-armv7-global-isel has been red for 10 days. clang-cmake-armv7-lnt has been red for 10 days. clang-cmake-aarch64-lld has been red for 11 days. clang-cmake-armv8-lld has been red for 11 days. clang-cmake-thumbv7-full-sh has been red for 11 days. clang-cmake-armv7-full has been red for 11 days.
Is anybody looking at this?
Yes. The underlying problem was that we removed python2 packages from our buildbot docker images, and that caused all builders that use on virtualenv (which uses python2 by default) to break. We've fixed that late last week, but some of the builders accumulated new failures that we are going through now.
I also noticed that some of your workers are less reliable than others. For example, inaro-tk1-09 worker was last time producing a green build 2 months ago - http://lab.llvm.org:8011/#/workers/2?numbuilds=500
I'm removing the linaro-tk1-09 worker from the production buildbot for now. Please feel free to connect it to the staging and make the builder reliably green. Then it could be returned back to the production.
This wasn't on our radar. We'll investigate -- thanks!
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have been red for the least 4 months. I'm removing them from the production buildbot. You can return them back to production after they are reliably green in the staging.
These ones had several compounding problems over the last 4 months. We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org