Hello Linaro Toolchain Working Group,
libcxx-libcxxabi-libunwind-armv8-linux has been red for 5 days. clang-cmake-aarch64-global-isel has been red for 10 days. clang-cmake-aarch64-quick has been red for 10 days. clang-cmake-armv7-global-isel has been red for 10 days. clang-cmake-armv7-lnt has been red for 10 days. clang-cmake-aarch64-lld has been red for 11 days. clang-cmake-armv8-lld has been red for 11 days. clang-cmake-thumbv7-full-sh has been red for 11 days. clang-cmake-armv7-full has been red for 11 days.
Is anybody looking at this?
I also noticed that some of your workers are less reliable than others. For example, inaro-tk1-09 worker was last time producing a green build 2 months ago - http://lab.llvm.org:8011/#/workers/2?numbuilds=500
I'm removing the linaro-tk1-09 worker from the production buildbot for now. Please feel free to connect it to the staging and make the builder reliably green. Then it could be returned back to the production.
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have been red for the least 4 months. I'm removing them from the production buildbot. You can return them back to production after they are reliably green in the staging.
Please feel free to ask if you have questions.
Thanks
Galina
On Feb 7, 2021, at 6:10 AM, Galina Kistanova gkistanova@gmail.com wrote:
Hello Linaro Toolchain Working Group,
Hi Galina,
Thanks for looking out for buildbot problems. We are doing our best to keep ARM and AArch64 builders and workers green and happy, but things do slip through the cracks occasionally.
libcxx-libcxxabi-libunwind-armv8-linux has been red for 5 days. clang-cmake-aarch64-global-isel has been red for 10 days. clang-cmake-aarch64-quick has been red for 10 days. clang-cmake-armv7-global-isel has been red for 10 days. clang-cmake-armv7-lnt has been red for 10 days. clang-cmake-aarch64-lld has been red for 11 days. clang-cmake-armv8-lld has been red for 11 days. clang-cmake-thumbv7-full-sh has been red for 11 days. clang-cmake-armv7-full has been red for 11 days.
Is anybody looking at this?
Yes. The underlying problem was that we removed python2 packages from our buildbot docker images, and that caused all builders that use on virtualenv (which uses python2 by default) to break. We've fixed that late last week, but some of the builders accumulated new failures that we are going through now.
I also noticed that some of your workers are less reliable than others. For example, inaro-tk1-09 worker was last time producing a green build 2 months ago - http://lab.llvm.org:8011/#/workers/2?numbuilds=500
I'm removing the linaro-tk1-09 worker from the production buildbot for now. Please feel free to connect it to the staging and make the builder reliably green. Then it could be returned back to the production.
This wasn't on our radar. We'll investigate -- thanks!
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have been red for the least 4 months. I'm removing them from the production buildbot. You can return them back to production after they are reliably green in the staging.
These ones had several compounding problems over the last 4 months. We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org
Thanks for looking into this, Maxim!
On Mon, Feb 8, 2021 at 1:15 AM Maxim Kuvyrkov maxim.kuvyrkov@linaro.org wrote:
On Feb 7, 2021, at 6:10 AM, Galina Kistanova gkistanova@gmail.com
wrote:
Hello Linaro Toolchain Working Group,
Hi Galina,
Thanks for looking out for buildbot problems. We are doing our best to keep ARM and AArch64 builders and workers green and happy, but things do slip through the cracks occasionally.
libcxx-libcxxabi-libunwind-armv8-linux has been red for 5 days. clang-cmake-aarch64-global-isel has been red for 10 days. clang-cmake-aarch64-quick has been red for 10 days. clang-cmake-armv7-global-isel has been red for 10 days. clang-cmake-armv7-lnt has been red for 10 days. clang-cmake-aarch64-lld has been red for 11 days. clang-cmake-armv8-lld has been red for 11 days. clang-cmake-thumbv7-full-sh has been red for 11 days. clang-cmake-armv7-full has been red for 11 days.
Is anybody looking at this?
Yes. The underlying problem was that we removed python2 packages from our buildbot docker images, and that caused all builders that use on virtualenv (which uses python2 by default) to break. We've fixed that late last week, but some of the builders accumulated new failures that we are going through now.
I also noticed that some of your workers are less reliable than others.
For
example, inaro-tk1-09 worker was last time producing a green build 2
months
ago - http://lab.llvm.org:8011/#/workers/2?numbuilds=500
I'm removing the linaro-tk1-09 worker from the production buildbot for
now.
Please feel free to connect it to the staging and make the builder
reliably
green. Then it could be returned back to the production.
This wasn't on our radar. We'll investigate -- thanks!
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have been
red
for the least 4 months. I'm removing them from the production buildbot.
You
can return them back to production after they are reliably green in the staging.
These ones had several compounding problems over the last 4 months. We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org
On Feb 8, 2021, at 7:26 PM, Galina Kistanova gkistanova@gmail.com wrote:
Thanks for looking into this, Maxim!
On Mon, Feb 8, 2021 at 1:15 AM Maxim Kuvyrkov maxim.kuvyrkov@linaro.org wrote:
On Feb 7, 2021, at 6:10 AM, Galina Kistanova gkistanova@gmail.com wrote:
Hello Linaro Toolchain Working Group,
...
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have been red for the least 4 months. I'm removing them from the production buildbot. You can return them back to production after they are reliably green in the staging.
These ones had several compounding problems over the last 4 months. We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Hi Galina,
linaro-armv8-windows-msvc-01 and -02 are in good shape now: - http://lab.llvm.org:8014/#/workers/89 - http://lab.llvm.org:8014/#/workers/32
Would you please re-enable them for the production buildbot?
We are continuing to investigate and fix problems in our linux bots.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org
Hi Maxim,
Please feel free to move linaro-armv8-windows-msvc-01 and -02 back to the production buildbot.
Thanks
Galina
On Mon, Feb 15, 2021 at 8:16 AM Maxim Kuvyrkov maxim.kuvyrkov@linaro.org wrote:
On Feb 8, 2021, at 7:26 PM, Galina Kistanova gkistanova@gmail.com
wrote:
Thanks for looking into this, Maxim!
On Mon, Feb 8, 2021 at 1:15 AM Maxim Kuvyrkov maxim.kuvyrkov@linaro.org
wrote:
On Feb 7, 2021, at 6:10 AM, Galina Kistanova gkistanova@gmail.com
wrote:
Hello Linaro Toolchain Working Group,
...
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have
been red
for the least 4 months. I'm removing them from the production
buildbot. You
can return them back to production after they are reliably green in the staging.
These ones had several compounding problems over the last 4 months.
We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Hi Galina,
linaro-armv8-windows-msvc-01 and -02 are in good shape now:
Would you please re-enable them for the production buildbot?
We are continuing to investigate and fix problems in our linux bots.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org
By the way, I still see 'linaro-tk1-09' trying to connect from 51.148.40.55 to the production buildbot.
Could you stop that worker or point it to the staging, please? It is going to get blacklisted, and, since this is a NAT address, multiple of your workers which share the same IP address will be affected.
Thanks
Galina
On Tue, Feb 16, 2021 at 3:47 PM Galina Kistanova gkistanova@gmail.com wrote:
Hi Maxim,
Please feel free to move linaro-armv8-windows-msvc-01 and -02 back to the production buildbot.
Thanks
Galina
On Mon, Feb 15, 2021 at 8:16 AM Maxim Kuvyrkov maxim.kuvyrkov@linaro.org wrote:
On Feb 8, 2021, at 7:26 PM, Galina Kistanova gkistanova@gmail.com
wrote:
Thanks for looking into this, Maxim!
On Mon, Feb 8, 2021 at 1:15 AM Maxim Kuvyrkov <
maxim.kuvyrkov@linaro.org> wrote:
On Feb 7, 2021, at 6:10 AM, Galina Kistanova gkistanova@gmail.com
wrote:
Hello Linaro Toolchain Working Group,
...
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have
been red
for the least 4 months. I'm removing them from the production
buildbot. You
can return them back to production after they are reliably green in
the
staging.
These ones had several compounding problems over the last 4 months.
We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Hi Galina,
linaro-armv8-windows-msvc-01 and -02 are in good shape now:
Would you please re-enable them for the production buildbot?
We are continuing to investigate and fix problems in our linux bots.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org
Hi Galina,
I have stopped worker linaro-tk1-09. I hope this issue will be fixed now.
Thanks!
On Wed, 17 Feb 2021 at 11:10, Galina Kistanova gkistanova@gmail.com wrote:
By the way, I still see 'linaro-tk1-09' trying to connect from 51.148.40.55 to the production buildbot.
Could you stop that worker or point it to the staging, please? It is going to get blacklisted, and, since this is a NAT address, multiple of your workers which share the same IP address will be affected.
Thanks
Galina
On Tue, Feb 16, 2021 at 3:47 PM Galina Kistanova gkistanova@gmail.com wrote:
Hi Maxim,
Please feel free to move linaro-armv8-windows-msvc-01 and -02 back to the production buildbot.
Thanks
Galina
On Mon, Feb 15, 2021 at 8:16 AM Maxim Kuvyrkov <
maxim.kuvyrkov@linaro.org>
wrote:
On Feb 8, 2021, at 7:26 PM, Galina Kistanova gkistanova@gmail.com
wrote:
Thanks for looking into this, Maxim!
On Mon, Feb 8, 2021 at 1:15 AM Maxim Kuvyrkov <
maxim.kuvyrkov@linaro.org> wrote:
On Feb 7, 2021, at 6:10 AM, Galina Kistanova gkistanova@gmail.com
wrote:
Hello Linaro Toolchain Working Group,
...
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have
been red
for the least 4 months. I'm removing them from the production
buildbot. You
can return them back to production after they are reliably green in
the
staging.
These ones had several compounding problems over the last 4 months.
We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Hi Galina,
linaro-armv8-windows-msvc-01 and -02 are in good shape now:
Would you please re-enable them for the production buildbot?
We are continuing to investigate and fix problems in our linux bots.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org
linaro-toolchain mailing list linaro-toolchain@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-toolchain
Thanks, Omair!
On Wed, Feb 17, 2021 at 2:06 AM Omair Javaid omair.javaid@linaro.org wrote:
Hi Galina,
I have stopped worker linaro-tk1-09. I hope this issue will be fixed now.
Thanks!
On Wed, 17 Feb 2021 at 11:10, Galina Kistanova gkistanova@gmail.com wrote:
By the way, I still see 'linaro-tk1-09' trying to connect from 51.148.40.55 to the production buildbot.
Could you stop that worker or point it to the staging, please? It is going to get blacklisted, and, since this is a NAT address, multiple of your workers which share the same IP address will be affected.
Thanks
Galina
On Tue, Feb 16, 2021 at 3:47 PM Galina Kistanova gkistanova@gmail.com wrote:
Hi Maxim,
Please feel free to move linaro-armv8-windows-msvc-01 and -02 back to
the
production buildbot.
Thanks
Galina
On Mon, Feb 15, 2021 at 8:16 AM Maxim Kuvyrkov <
maxim.kuvyrkov@linaro.org>
wrote:
On Feb 8, 2021, at 7:26 PM, Galina Kistanova gkistanova@gmail.com
wrote:
Thanks for looking into this, Maxim!
On Mon, Feb 8, 2021 at 1:15 AM Maxim Kuvyrkov <
maxim.kuvyrkov@linaro.org> wrote:
On Feb 7, 2021, at 6:10 AM, Galina Kistanova <gkistanova@gmail.com
wrote:
Hello Linaro Toolchain Working Group,
...
linaro-armv8-windows-msvc-01 and linaro-armv8-windows-msvc-02 have
been red
for the least 4 months. I'm removing them from the production
buildbot. You
can return them back to production after they are reliably green in
the
staging.
These ones had several compounding problems over the last 4 months.
We've fixed all but the final problem, which is being discussed in https://reviews.llvm.org/rGb2851aea80e5 . I hope to get windows builders green by end of this week.
Hi Galina,
linaro-armv8-windows-msvc-01 and -02 are in good shape now:
Would you please re-enable them for the production buildbot?
We are continuing to investigate and fix problems in our linux bots.
Regards,
-- Maxim Kuvyrkov https://www.linaro.org
linaro-toolchain mailing list linaro-toolchain@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-toolchain
-- Omair Javaid www.linaro.org
linaro-toolchain@lists.linaro.org
-
Galina Kistanova -
Maxim Kuvyrkov -
Omair Javaid