Hi,
Has anyone noticed that https://lab.llvm.org/buildbot/#/builders/198 (clang-aarch64-sve-vla-2stagehttps://lab.llvm.org/buildbot/#/builders/198) has been timing out for the past few days? "Duration" is often less than 1hr, so that's odd. And all Flang buildbots are green, so it's unlikely caused by changes to that sub-project (https://lab.llvm.org/buildbot/#/builders/198/builds/1804). Would anyone be able to take a look?
Best regards,
Andrzej
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Thanks, Andrzej!
Hi Antoine, would you please take a look at this?
Kind regards,
-- Maxim Kuvyrkov https://www.linaro.org
Yeah sure, I am looking at this.
Here the timeout happens after 1200 seconds without any new output on stdout/stderr. I reproduce the issue on tcwg-g3-01 with -j34 (the default on this machine): the build is stuck near the end when building flang objects. The same build succeeds using -j16 on this machine. So I wonder if this could simply be due to insufficient memory.
I will relaunch a few builds to check if something significant changed with last commits.
KO * 0245dc94ebee [test][sanitizers] Disable new test on Android ?? * 795f93007e10 [AArch64] Add test for #62620. ?? * 4fe91e083a25 [llvm-profdata] ProfileReader cleanup - preparation for MD5 refactoring - 3 ?? * be5747e51693 [flang] Fixed global name creation for literal constants. OK * 26a7f4230613 [mlir][tosa] Add accumulator type attribute to TOSA dialect
Antoine
On Mon, 15 May 2023 at 22:37, Maxim Kuvyrkov maxim.kuvyrkov@linaro.org wrote:
Thanks for looking into this!
Based on the logs and your experiment, looks like it's Flang's fault. 64GB used to be sufficient, but perhaps that's no longer the case (sadly, Flang has insane requirements). I will ask internally.
If this is indeed Flang's fault, I'd be tempted to simply disable it on this bot. We know that that sub-project is tested quite extensively through dedicated Flang bots (8 in total). What do you think? That would have the additional benefit of making this bot much _much_ faster :)
-Andrzej
On 16/05/2023 07:23, Antoine Moynault wrote: Yeah sure, I am looking at this.
Here the timeout happens after 1200 seconds without any new output on stdout/stderr. I reproduce the issue on tcwg-g3-01 with -j34 (the default on this machine): the build is stuck near the end when building flang objects. The same build succeeds using -j16 on this machine. So I wonder if this could simply be due to insufficient memory.
I will relaunch a few builds to check if something significant changed with last commits. KO * 0245dc94ebee [test][sanitizers] Disable new test on Android ?? * 795f93007e10 [AArch64] Add test for #62620. ?? * 4fe91e083a25 [llvm-profdata] ProfileReader cleanup - preparation for MD5 refactoring - 3 ?? * be5747e51693 [flang] Fixed global name creation for literal constants. OK * 26a7f4230613 [mlir][tosa] Add accumulator type attribute to TOSA dialect
Antoine
On Mon, 15 May 2023 at 22:37, Maxim Kuvyrkov <maxim.kuvyrkov@linaro.orgmailto:maxim.kuvyrkov@linaro.org> wrote: Thanks, Andrzej!
Hi Antoine, would you please take a look at this?
Kind regards,
-- Maxim Kuvyrkov https://www.linaro.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
+ @Kiran Chandramohanmailto:Kiran.Chandramohan@arm.com
Kiran suggested that this might be this recent change in MLIR that’s increased build times: https://github.com/llvm/llvm-project/issues/62687
-Andrzej
Thanks for looking into this!
Based on the logs and your experiment, looks like it's Flang's fault. 64GB used to be sufficient, but perhaps that's no longer the case (sadly, Flang has insane requirements). I will ask internally.
If this is indeed Flang's fault, I'd be tempted to simply disable it on this bot. We know that that sub-project is tested quite extensively through dedicated Flang bots. What do you think?
-Andrzej On 16/05/2023 07:23, Antoine Moynault wrote: Yeah sure, I am looking at this.
Here the timeout happens after 1200 seconds without any new output on stdout/stderr. I reproduce the issue on tcwg-g3-01 with -j34 (the default on this machine): the build is stuck near the end when building flang objects. The same build succeeds using -j16 on this machine. So I wonder if this could simply be due to insufficient memory.
I will relaunch a few builds to check if something significant changed with last commits. KO * 0245dc94ebee [test][sanitizers] Disable new test on Android ?? * 795f93007e10 [AArch64] Add test for #62620. ?? * 4fe91e083a25 [llvm-profdata] ProfileReader cleanup - preparation for MD5 refactoring - 3 ?? * be5747e51693 [flang] Fixed global name creation for literal constants. OK * 26a7f4230613 [mlir][tosa] Add accumulator type attribute to TOSA dialect
Antoine
On Mon, 15 May 2023 at 22:37, Maxim Kuvyrkov <maxim.kuvyrkov@linaro.orgmailto:maxim.kuvyrkov@linaro.org> wrote: Thanks, Andrzej!
Hi Antoine, would you please take a look at this?
Kind regards,
-- Maxim Kuvyrkov https://www.linaro.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Yeah thanks, it could have explained this indeed. But I guess bots would be green again if it was linked to this mlir issue 62687, as the fix for it was merged several hours ago and part of these jobs:
https://lab.llvm.org/buildbot/#/builders/198/builds/1879 https://lab.llvm.org/buildbot/#/builders/176/builds/2357
Actually Maxim probably found the issue: The swap was not enabled on these g3 machines, which is a problem as the build of flang (-j34) now needs a bit more than the 58GB of RAM allocated for the buildbot container. I also understand that a custom ninja binary is used on these containers, behaving differently when processes are waiting for something (memory/cpu). Enabling the swap on these linaro-g3 machines will probably fix the bot failures.
And yes indeed, building flang looks to require a lot of memory: ~60Go RAM for a full build (j34) vs <20Go RAM for a build without flang !
Antoine
On Tue, 16 May 2023 at 10:56, Andrzej Warzynski Andrzej.Warzynski@arm.com wrote:
Great detective work, thank you for fixing this :)
-Andrzej
On 16/05/2023 16:55, Antoine Moynault wrote:
Yeah thanks, it could have explained this indeed. But I guess bots would be green again if it was linked to this mlir issue 62687, as the fix for it was merged several hours ago and part of these jobs: https://lab.llvm.org/buildbot/#/builders/198/builds/1879 https://lab.llvm.org/buildbot/#/builders/176/builds/2357
Actually Maxim probably found the issue: The swap was not enabled on these g3 machines, which is a problem as the build of flang (-j34) now needs a bit more than the 58GB of RAM allocated for the buildbot container. I also understand that a custom ninja binary is used on these containers, behaving differently when processes are waiting for something (memory/cpu). Enabling the swap on these linaro-g3 machines will probably fix the bot failures.
And yes indeed, building flang looks to require a lot of memory: ~60Go RAM for a full build (j34) vs <20Go RAM for a build without flang !
Antoine
On Tue, 16 May 2023 at 10:56, Andrzej Warzynski <Andrzej.Warzynski@arm.commailto:Andrzej.Warzynski@arm.com> wrote:
+ @Kiran Chandramohanmailto:Kiran.Chandramohan@arm.com
Kiran suggested that this might be this recent change in MLIR that’s increased build times: https://github.com/llvm/llvm-project/issues/62687
-Andrzej
Thanks for looking into this!
Based on the logs and your experiment, looks like it's Flang's fault. 64GB used to be sufficient, but perhaps that's no longer the case (sadly, Flang has insane requirements). I will ask internally.
If this is indeed Flang's fault, I'd be tempted to simply disable it on this bot. We know that that sub-project is tested quite extensively through dedicated Flang bots. What do you think?
-Andrzej On 16/05/2023 07:23, Antoine Moynault wrote: Yeah sure, I am looking at this.
Here the timeout happens after 1200 seconds without any new output on stdout/stderr. I reproduce the issue on tcwg-g3-01 with -j34 (the default on this machine): the build is stuck near the end when building flang objects. The same build succeeds using -j16 on this machine. So I wonder if this could simply be due to insufficient memory.
I will relaunch a few builds to check if something significant changed with last commits. KO * 0245dc94ebee [test][sanitizers] Disable new test on Android ?? * 795f93007e10 [AArch64] Add test for #62620. ?? * 4fe91e083a25 [llvm-profdata] ProfileReader cleanup - preparation for MD5 refactoring - 3 ?? * be5747e51693 [flang] Fixed global name creation for literal constants. OK * 26a7f4230613 [mlir][tosa] Add accumulator type attribute to TOSA dialect
Antoine
On Mon, 15 May 2023 at 22:37, Maxim Kuvyrkov <maxim.kuvyrkov@linaro.orgmailto:maxim.kuvyrkov@linaro.org> wrote: Thanks, Andrzej!
Hi Antoine, would you please take a look at this?
Kind regards,
-- Maxim Kuvyrkov https://www.linaro.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
linaro-toolchain@lists.linaro.org
-
Andrzej Warzynski -
Andrzej Warzynski
-
Antoine Moynault -
Maxim Kuvyrkov