linaro-toolchain

linaro-toolchain@lists.linaro.org

3 participants
5603 discussions

[TCWG CI] 464.h264ref slowed down by 7% after llvm: [JumpThreading] Ignore free instructions

by ci_notify＠linaro.org

After llvm commit 1e3c6fc7cb9d2ee6a5328881f95d6643afeadbff Author: Nikita Popov <nikita.ppv(a)gmail.com> [JumpThreading] Ignore free instructions the following benchmarks slowed down by more than 2%: - 464.h264ref slowed down by 7% from 10715 to 11434 perf samples Below reproducer instructions can be used to re-build both "first_bad" and "last_good" cross-toolchains used in this bisection. Naturally, the scripts will fail when triggerring benchmarking jobs if you don't have access to Linaro TCWG CI. For your convenience, we have uploaded tarballs with pre-processed source and assembly files at: - First_bad save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… - Last_good save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… - Baseline save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… Configuration: - Benchmark: SPEC CPU2006 - Toolchain: Clang + Glibc + LLVM Linker - Version: all components were built from their tip of trunk - Target: aarch64-linux-gnu - Compiler flags: -O3 -flto - Hardware: NVidia TX1 4x Cortex-A57 This benchmarking CI is work-in-progress, and we welcome feedback and suggestions at linaro-toolchain(a)lists.linaro.org . In our improvement plans is to add support for SPEC CPU2017 benchmarks and provide "perf report/annotate" data behind these reports. THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. This commit has regressed these CI configurations: - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O3_LTO First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… Reproduce builds: <cut> mkdir investigate-llvm-1e3c6fc7cb9d2ee6a5328881f95d6643afeadbff cd investigate-llvm-1e3c6fc7cb9d2ee6a5328881f95d6643afeadbff # Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts # Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-… --fail chmod +x artifacts/test.sh # Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh # Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /llvm/ ./ ./bisect/baseline/ cd llvm # Reproduce first_bad build git checkout --detach 1e3c6fc7cb9d2ee6a5328881f95d6643afeadbff ../artifacts/test.sh # Reproduce last_good build git checkout --detach 1a6e1ee42a6af255d45e3fd2fe87021dd31f79bb ../artifacts/test.sh cd .. </cut> Full commit (up to 1000 lines): <cut> commit 1e3c6fc7cb9d2ee6a5328881f95d6643afeadbff Author: Nikita Popov <nikita.ppv(a)gmail.com> Date: Wed Sep 22 21:34:24 2021 +0200 [JumpThreading] Ignore free instructions This is basically D108837 but for jump threading. Free instructions should be ignored for the threading decision. JumpThreading already skips some free instructions (like pointer bitcasts), but does not skip various free intrinsics -- in fact, it currently gives them a fairly large cost of 2. Differential Revision: https://reviews.llvm.org/D110290 --- .../include/llvm/Transforms/Scalar/JumpThreading.h | 8 +-- llvm/lib/Transforms/Scalar/JumpThreading.cpp | 61 ++++++++++------------ .../Transforms/JumpThreading/free_instructions.ll | 24 +++++---- .../inlining-alignment-assumptions.ll | 12 ++--- 4 files changed, 52 insertions(+), 53 deletions(-) diff --git a/llvm/include/llvm/Transforms/Scalar/JumpThreading.h b/llvm/include/llvm/Transforms/Scalar/JumpThreading.h index 816ea1071e52..0ac7d7c62b7a 100644 --- a/llvm/include/llvm/Transforms/Scalar/JumpThreading.h +++ b/llvm/include/llvm/Transforms/Scalar/JumpThreading.h @@ -44,6 +44,7 @@ class PHINode; class SelectInst; class SwitchInst; class TargetLibraryInfo; +class TargetTransformInfo; class Value; /// A private "module" namespace for types and utilities used by @@ -78,6 +79,7 @@ enum ConstantPreference { WantInteger, WantBlockAddress }; /// revectored to the false side of the second if. class JumpThreadingPass : public PassInfoMixin<JumpThreadingPass> { TargetLibraryInfo *TLI; + TargetTransformInfo *TTI; LazyValueInfo *LVI; AAResults *AA; DomTreeUpdater *DTU; @@ -99,9 +101,9 @@ public: JumpThreadingPass(bool InsertFreezeWhenUnfoldingSelect = false, int T = -1); // Glue for old PM. - bool runImpl(Function &F, TargetLibraryInfo *TLI, LazyValueInfo *LVI, - AAResults *AA, DomTreeUpdater *DTU, bool HasProfileData, - std::unique_ptr<BlockFrequencyInfo> BFI, + bool runImpl(Function &F, TargetLibraryInfo *TLI, TargetTransformInfo *TTI, + LazyValueInfo *LVI, AAResults *AA, DomTreeUpdater *DTU, + bool HasProfileData, std::unique_ptr<BlockFrequencyInfo> BFI, std::unique_ptr<BranchProbabilityInfo> BPI); PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM); diff --git a/llvm/lib/Transforms/Scalar/JumpThreading.cpp b/llvm/lib/Transforms/Scalar/JumpThreading.cpp index 688902ecb9ff..fe9a7211967c 100644 --- a/llvm/lib/Transforms/Scalar/JumpThreading.cpp +++ b/llvm/lib/Transforms/Scalar/JumpThreading.cpp @@ -331,7 +331,7 @@ bool JumpThreading::runOnFunction(Function &F) { BFI.reset(new BlockFrequencyInfo(F, *BPI, LI)); } - bool Changed = Impl.runImpl(F, TLI, LVI, AA, &DTU, F.hasProfileData(), + bool Changed = Impl.runImpl(F, TLI, TTI, LVI, AA, &DTU, F.hasProfileData(), std::move(BFI), std::move(BPI)); if (PrintLVIAfterJumpThreading) { dbgs() << "LVI for function '" << F.getName() << "':\n"; @@ -360,7 +360,7 @@ PreservedAnalyses JumpThreadingPass::run(Function &F, BFI.reset(new BlockFrequencyInfo(F, *BPI, LI)); } - bool Changed = runImpl(F, &TLI, &LVI, &AA, &DTU, F.hasProfileData(), + bool Changed = runImpl(F, &TLI, &TTI, &LVI, &AA, &DTU, F.hasProfileData(), std::move(BFI), std::move(BPI)); if (PrintLVIAfterJumpThreading) { @@ -377,12 +377,14 @@ PreservedAnalyses JumpThreadingPass::run(Function &F, } bool JumpThreadingPass::runImpl(Function &F, TargetLibraryInfo *TLI_, - LazyValueInfo *LVI_, AliasAnalysis *AA_, - DomTreeUpdater *DTU_, bool HasProfileData_, + TargetTransformInfo *TTI_, LazyValueInfo *LVI_, + AliasAnalysis *AA_, DomTreeUpdater *DTU_, + bool HasProfileData_, std::unique_ptr<BlockFrequencyInfo> BFI_, std::unique_ptr<BranchProbabilityInfo> BPI_) { LLVM_DEBUG(dbgs() << "Jump threading on function '" << F.getName() << "'\n"); TLI = TLI_; + TTI = TTI_; LVI = LVI_; AA = AA_; DTU = DTU_; @@ -514,7 +516,8 @@ static void replaceFoldableUses(Instruction *Cond, Value *ToVal) { /// Return the cost of duplicating a piece of this block from first non-phi /// and before StopAt instruction to thread across it. Stop scanning the block /// when exceeding the threshold. If duplication is impossible, returns ~0U. -static unsigned getJumpThreadDuplicationCost(BasicBlock *BB, +static unsigned getJumpThreadDuplicationCost(const TargetTransformInfo *TTI, + BasicBlock *BB, Instruction *StopAt, unsigned Threshold) { assert(StopAt->getParent() == BB && "Not an instruction from proper BB?"); @@ -550,26 +553,21 @@ static unsigned getJumpThreadDuplicationCost(BasicBlock *BB, if (Size > Threshold) return Size; - // Debugger intrinsics don't incur code size. - if (isa<DbgInfoIntrinsic>(I)) continue; - - // Pseudo-probes don't incur code size. - if (isa<PseudoProbeInst>(I)) - continue; - - // If this is a pointer->pointer bitcast, it is free. - if (isa<BitCastInst>(I) && I->getType()->isPointerTy()) - continue; - - // Freeze instruction is free, too. - if (isa<FreezeInst>(I)) - continue; - // Bail out if this instruction gives back a token type, it is not possible // to duplicate it if it is used outside this BB. if (I->getType()->isTokenTy() && I->isUsedOutsideOfBlock(BB)) return ~0U; + // Blocks with NoDuplicate are modelled as having infinite cost, so they + // are never duplicated. + if (const CallInst *CI = dyn_cast<CallInst>(I)) + if (CI->cannotDuplicate() || CI->isConvergent()) + return ~0U; + + if (TTI->getUserCost(&*I, TargetTransformInfo::TCK_SizeAndLatency) + == TargetTransformInfo::TCC_Free) + continue; + // All other instructions count for at least one unit. ++Size; @@ -578,11 +576,7 @@ static unsigned getJumpThreadDuplicationCost(BasicBlock *BB, // as having cost of 2 total, and if they are a vector intrinsic, we model // them as having cost 1. if (const CallInst *CI = dyn_cast<CallInst>(I)) { - if (CI->cannotDuplicate() || CI->isConvergent()) - // Blocks with NoDuplicate are modelled as having infinite cost, so they - // are never duplicated. - return ~0U; - else if (!isa<IntrinsicInst>(CI)) + if (!isa<IntrinsicInst>(CI)) Size += 3; else if (!CI->getType()->isVectorTy()) Size += 1; @@ -2234,10 +2228,10 @@ bool JumpThreadingPass::maybethreadThroughTwoBasicBlocks(BasicBlock *BB, } // Compute the cost of duplicating BB and PredBB. - unsigned BBCost = - getJumpThreadDuplicationCost(BB, BB->getTerminator(), BBDupThreshold); + unsigned BBCost = getJumpThreadDuplicationCost( + TTI, BB, BB->getTerminator(), BBDupThreshold); unsigned PredBBCost = getJumpThreadDuplicationCost( - PredBB, PredBB->getTerminator(), BBDupThreshold); + TTI, PredBB, PredBB->getTerminator(), BBDupThreshold); // Give up if costs are too high. We need to check BBCost and PredBBCost // individually before checking their sum because getJumpThreadDuplicationCost @@ -2345,8 +2339,8 @@ bool JumpThreadingPass::tryThreadEdge( return false; } - unsigned JumpThreadCost = - getJumpThreadDuplicationCost(BB, BB->getTerminator(), BBDupThreshold); + unsigned JumpThreadCost = getJumpThreadDuplicationCost( + TTI, BB, BB->getTerminator(), BBDupThreshold); if (JumpThreadCost > BBDupThreshold) { LLVM_DEBUG(dbgs() << " Not threading BB '" << BB->getName() << "' - Cost is too high: " << JumpThreadCost << "\n"); @@ -2614,8 +2608,8 @@ bool JumpThreadingPass::duplicateCondBranchOnPHIIntoPred( return false; } - unsigned DuplicationCost = - getJumpThreadDuplicationCost(BB, BB->getTerminator(), BBDupThreshold); + unsigned DuplicationCost = getJumpThreadDuplicationCost( + TTI, BB, BB->getTerminator(), BBDupThreshold); if (DuplicationCost > BBDupThreshold) { LLVM_DEBUG(dbgs() << " Not duplicating BB '" << BB->getName() << "' - Cost is too high: " << DuplicationCost << "\n"); @@ -3031,7 +3025,8 @@ bool JumpThreadingPass::threadGuard(BasicBlock *BB, IntrinsicInst *Guard, ValueToValueMapTy UnguardedMapping, GuardedMapping; Instruction *AfterGuard = Guard->getNextNode(); - unsigned Cost = getJumpThreadDuplicationCost(BB, AfterGuard, BBDupThreshold); + unsigned Cost = + getJumpThreadDuplicationCost(TTI, BB, AfterGuard, BBDupThreshold); if (Cost > BBDupThreshold) return false; // Duplicate all instructions before the guard and the guard itself to the diff --git a/llvm/test/Transforms/JumpThreading/free_instructions.ll b/llvm/test/Transforms/JumpThreading/free_instructions.ll index f768ec996779..76392af77d33 100644 --- a/llvm/test/Transforms/JumpThreading/free_instructions.ll +++ b/llvm/test/Transforms/JumpThreading/free_instructions.ll @@ -5,26 +5,28 @@ ; the jump threading threshold, as everything else are free instructions. define i32 @free_instructions(i1 %c, i32* %p) { ; CHECK-LABEL: @free_instructions( -; CHECK-NEXT: br i1 [[C:%.*]], label [[IF:%.*]], label [[ELSE:%.*]] -; CHECK: if: +; CHECK-NEXT: br i1 [[C:%.*]], label [[IF2:%.*]], label [[ELSE2:%.*]] +; CHECK: if2: ; CHECK-NEXT: store i32 -1, i32* [[P:%.*]], align 4 -; CHECK-NEXT: br label [[JOIN:%.*]] -; CHECK: else: -; CHECK-NEXT: store i32 -2, i32* [[P]], align 4 -; CHECK-NEXT: br label [[JOIN]] -; CHECK: join: ; CHECK-NEXT: call void @llvm.experimental.noalias.scope.decl(metadata [[META0:![0-9]+]]) ; CHECK-NEXT: store i32 1, i32* [[P]], align 4, !noalias !0 ; CHECK-NEXT: call void @llvm.assume(i1 true) [ "align"(i32* [[P]], i64 32) ] ; CHECK-NEXT: store i32 2, i32* [[P]], align 4 +; CHECK-NEXT: [[P21:%.*]] = bitcast i32* [[P]] to i8* +; CHECK-NEXT: [[P32:%.*]] = call i8* @llvm.launder.invariant.group.p0i8(i8* [[P21]]) +; CHECK-NEXT: [[P43:%.*]] = bitcast i8* [[P32]] to i32* +; CHECK-NEXT: store i32 3, i32* [[P43]], align 4, !invariant.group !3 +; CHECK-NEXT: ret i32 0 +; CHECK: else2: +; CHECK-NEXT: store i32 -2, i32* [[P]], align 4 +; CHECK-NEXT: call void @llvm.experimental.noalias.scope.decl(metadata [[META4:![0-9]+]]) +; CHECK-NEXT: store i32 1, i32* [[P]], align 4, !noalias !4 +; CHECK-NEXT: call void @llvm.assume(i1 true) [ "align"(i32* [[P]], i64 32) ] +; CHECK-NEXT: store i32 2, i32* [[P]], align 4 ; CHECK-NEXT: [[P2:%.*]] = bitcast i32* [[P]] to i8* ; CHECK-NEXT: [[P3:%.*]] = call i8* @llvm.launder.invariant.group.p0i8(i8* [[P2]]) ; CHECK-NEXT: [[P4:%.*]] = bitcast i8* [[P3]] to i32* ; CHECK-NEXT: store i32 3, i32* [[P4]], align 4, !invariant.group !3 -; CHECK-NEXT: br i1 [[C]], label [[IF2:%.*]], label [[ELSE2:%.*]] -; CHECK: if2: -; CHECK-NEXT: ret i32 0 -; CHECK: else2: ; CHECK-NEXT: ret i32 1 ; br i1 %c, label %if, label %else diff --git a/llvm/test/Transforms/PhaseOrdering/inlining-alignment-assumptions.ll b/llvm/test/Transforms/PhaseOrdering/inlining-alignment-assumptions.ll index 57014e856a09..f764a59dd8a2 100644 --- a/llvm/test/Transforms/PhaseOrdering/inlining-alignment-assumptions.ll +++ b/llvm/test/Transforms/PhaseOrdering/inlining-alignment-assumptions.ll @@ -32,13 +32,10 @@ define void @caller1(i1 %c, i64* align 1 %ptr) { ; ASSUMPTIONS-OFF-NEXT: br label [[COMMON_RET]] ; ; ASSUMPTIONS-ON-LABEL: @caller1( -; ASSUMPTIONS-ON-NEXT: br i1 [[C:%.*]], label [[COMMON_RET:%.*]], label [[FALSE1:%.*]] -; ASSUMPTIONS-ON: false1: -; ASSUMPTIONS-ON-NEXT: store volatile i64 1, i64* [[PTR:%.*]], align 4 -; ASSUMPTIONS-ON-NEXT: br label [[COMMON_RET]] +; ASSUMPTIONS-ON-NEXT: br i1 [[C:%.*]], label [[COMMON_RET:%.*]], label [[FALSE2:%.*]] ; ASSUMPTIONS-ON: common.ret: -; ASSUMPTIONS-ON-NEXT: [[DOTSINK:%.*]] = phi i64 [ 3, [[FALSE1]] ], [ 2, [[TMP0:%.*]] ] -; ASSUMPTIONS-ON-NEXT: call void @llvm.assume(i1 true) [ "align"(i64* [[PTR]], i64 8) ] +; ASSUMPTIONS-ON-NEXT: [[DOTSINK:%.*]] = phi i64 [ 3, [[FALSE2]] ], [ 2, [[TMP0:%.*]] ] +; ASSUMPTIONS-ON-NEXT: call void @llvm.assume(i1 true) [ "align"(i64* [[PTR:%.*]], i64 8) ] ; ASSUMPTIONS-ON-NEXT: store volatile i64 0, i64* [[PTR]], align 8 ; ASSUMPTIONS-ON-NEXT: store volatile i64 -1, i64* [[PTR]], align 8 ; ASSUMPTIONS-ON-NEXT: store volatile i64 -1, i64* [[PTR]], align 8 @@ -47,6 +44,9 @@ define void @caller1(i1 %c, i64* align 1 %ptr) { ; ASSUMPTIONS-ON-NEXT: store volatile i64 -1, i64* [[PTR]], align 8 ; ASSUMPTIONS-ON-NEXT: store volatile i64 [[DOTSINK]], i64* [[PTR]], align 8 ; ASSUMPTIONS-ON-NEXT: ret void +; ASSUMPTIONS-ON: false2: +; ASSUMPTIONS-ON-NEXT: store volatile i64 1, i64* [[PTR]], align 4 +; ASSUMPTIONS-ON-NEXT: br label [[COMMON_RET]] ; br i1 %c, label %true1, label %false1 </cut>

3 years, 9 months

[ACTIVITY] report week ending 24 Sep

by Peter Maydell

Progress * UM-2 [QEMU upstream maintainership] + Still looking at the mess that is non-unique bus names. Worked through exactly which devices and machine types are affected for the i2c bus. + Sent a patchset which tries to make the "create a bus" function names a bit more regular across different bus types. * QEMU-406 [QEMU support for MVE (M-profile Vector Extension; Helium)] + Luis figured out why GDB was crashing when fed the MVE XML by QEMU's gdbstub; this was a combination of QEMU giving GDB some non-standard extra registers in its "vfp" XML feature and GDB not being robust enough against those unexpected extras. Sent out a patchset which cleans up QEMU's XML in this area and also implements the extra XML for MVE. (This will only go into QEMU once the GDB patches have landed and the XML format is nailed down.) -- PMM

3 years, 9 months

Re: [TCWG CI] 456.hmmer slowed down by 6% after llvm: Allow rematerialization of virtual reg uses

by Maxim Kuvyrkov

Hi Stanislav, FYI, your patch seems to be slowing down two of SPEC CPU2006 tests on 32-bit ARM at -O2 and -O3 optimization levels. -- Maxim Kuvyrkov https://www.linaro.org > On 15 Sep 2021, at 12:54, ci_notify(a)linaro.org wrote: > > After llvm commit 92c1fd19abb15bc68b1127a26137a69e033cdb39 > Author: Stanislav Mekhanoshin <Stanislav.Mekhanoshin(a)amd.com> > > Allow rematerialization of virtual reg uses > > the following benchmarks slowed down by more than 2%: > - 456.hmmer slowed down by 6% > - 482.sphinx3 slowed down by 3% > > Benchmark: > Toolchain: Clang + Glibc + LLVM Linker > Version: all components were built from their tip of trunk > Target: arm-linux-gnueabihf > Compiler flags: -O3 -marm > Hardware: NVidia TK1 4x Cortex-A15 > > This commit has regressed these CI configurations: > - tcwg_bmk_llvm_tk1/llvm-master-arm-spec2k6-O2 > - tcwg_bmk_llvm_tk1/llvm-master-arm-spec2k6-O3 > > First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tk1-llvm-master-… > Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tk1-llvm-master-… > Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tk1-llvm-master-… > Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tk1-llvm-master-… > > Reproduce builds: > <cut> > mkdir investigate-llvm-92c1fd19abb15bc68b1127a26137a69e033cdb39 > cd investigate-llvm-92c1fd19abb15bc68b1127a26137a69e033cdb39 > > # Fetch scripts > git clone https://git.linaro.org/toolchain/jenkins-scripts > > # Fetch manifests and test.sh script > mkdir -p artifacts/manifests > curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tk1-llvm-master-… --fail > curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tk1-llvm-master-… --fail > curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tk1-llvm-master-… --fail > chmod +x artifacts/test.sh > > # Reproduce the baseline build (build all pre-requisites) > ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh > > # Save baseline build state (which is then restored in artifacts/test.sh) > mkdir -p ./bisect > rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /llvm/ ./ ./bisect/baseline/ > > cd llvm > > # Reproduce first_bad build > git checkout --detach 92c1fd19abb15bc68b1127a26137a69e033cdb39 > ../artifacts/test.sh > > # Reproduce last_good build > git checkout --detach 1d02a8bcd393ea9c50f0212797059888efc78002 > ../artifacts/test.sh > > cd .. > </cut> > > Full commit (up to 1000 lines): > <cut> > commit 92c1fd19abb15bc68b1127a26137a69e033cdb39 > Author: Stanislav Mekhanoshin <Stanislav.Mekhanoshin(a)amd.com> > Date: Thu Aug 19 11:42:09 2021 -0700 > > Allow rematerialization of virtual reg uses > > Currently isReallyTriviallyReMaterializableGeneric() implementation > prevents rematerialization on any virtual register use on the grounds > that is not a trivial rematerialization and that we do not want to > extend liveranges. > > It appears that LRE logic does not attempt to extend a liverange of > a source register for rematerialization so that is not an issue. > That is checked in the LiveRangeEdit::allUsesAvailableAt(). > > The only non-trivial aspect of it is accounting for tied-defs which > normally represent a read-modify-write operation and not rematerializable. > > The test for a tied-def situation already exists in the > /CodeGen/AMDGPU/remat-vop.mir, > test_no_remat_v_cvt_f32_i32_sdwa_dst_unused_preserve. > > The change has affected ARM/Thumb, Mips, RISCV, and x86. For the targets > where I more or less understand the asm it seems to reduce spilling > (as expected) or be neutral. However, it needs a review by all targets' > specialists. > > Differential Revision: https://reviews.llvm.org/D106408 > --- > llvm/include/llvm/CodeGen/TargetInstrInfo.h | 12 +- > llvm/lib/CodeGen/TargetInstrInfo.cpp | 9 +- > llvm/test/CodeGen/AMDGPU/remat-sop.mir | 60 + > llvm/test/CodeGen/ARM/arm-shrink-wrapping-linux.ll | 28 +- > llvm/test/CodeGen/ARM/funnel-shift-rot.ll | 32 +- > llvm/test/CodeGen/ARM/funnel-shift.ll | 30 +- > .../test/CodeGen/ARM/illegal-bitfield-loadstore.ll | 30 +- > llvm/test/CodeGen/ARM/neon-copy.ll | 10 +- > llvm/test/CodeGen/Mips/llvm-ir/ashr.ll | 227 +- > llvm/test/CodeGen/Mips/llvm-ir/lshr.ll | 206 +- > llvm/test/CodeGen/Mips/llvm-ir/shl.ll | 95 +- > llvm/test/CodeGen/Mips/llvm-ir/sub.ll | 31 +- > llvm/test/CodeGen/Mips/tls.ll | 4 +- > llvm/test/CodeGen/RISCV/atomic-rmw.ll | 120 +- > llvm/test/CodeGen/RISCV/atomic-signext.ll | 24 +- > llvm/test/CodeGen/RISCV/bswap-ctlz-cttz-ctpop.ll | 96 +- > llvm/test/CodeGen/RISCV/rv32i-rv64i-half.ll | 12 +- > llvm/test/CodeGen/RISCV/rv32zbb-zbp.ll | 526 +-- > llvm/test/CodeGen/RISCV/rv32zbb.ll | 94 +- > llvm/test/CodeGen/RISCV/rv32zbp.ll | 282 +- > llvm/test/CodeGen/RISCV/rv32zbt.ll | 348 +- > .../CodeGen/RISCV/rvv/fixed-vectors-bitreverse.ll | 324 +- > llvm/test/CodeGen/RISCV/rvv/fixed-vectors-bswap.ll | 146 +- > llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ctlz.ll | 3540 ++++++++++---------- > llvm/test/CodeGen/RISCV/rvv/fixed-vectors-cttz.ll | 720 ++-- > llvm/test/CodeGen/RISCV/srem-vector-lkk.ll | 208 +- > llvm/test/CodeGen/RISCV/urem-vector-lkk.ll | 190 +- > llvm/test/CodeGen/Thumb/dyn-stackalloc.ll | 7 +- > .../tail-pred-disabled-in-loloops.ll | 14 +- > .../LowOverheadLoops/varying-outer-2d-reduction.ll | 64 +- > .../CodeGen/Thumb2/LowOverheadLoops/while-loops.ll | 67 +- > llvm/test/CodeGen/Thumb2/ldr-str-imm12.ll | 30 +- > llvm/test/CodeGen/Thumb2/mve-float16regloops.ll | 82 +- > llvm/test/CodeGen/Thumb2/mve-float32regloops.ll | 98 +- > llvm/test/CodeGen/Thumb2/mve-postinc-dct.ll | 529 ++- > llvm/test/CodeGen/X86/addcarry.ll | 20 +- > llvm/test/CodeGen/X86/callbr-asm-blockplacement.ll | 12 +- > llvm/test/CodeGen/X86/dag-update-nodetomatch.ll | 17 +- > llvm/test/CodeGen/X86/inalloca-invoke.ll | 2 +- > llvm/test/CodeGen/X86/licm-regpressure.ll | 28 +- > llvm/test/CodeGen/X86/ragreedy-hoist-spill.ll | 40 +- > llvm/test/CodeGen/X86/sdiv_fix.ll | 5 +- > 42 files changed, 4217 insertions(+), 4202 deletions(-) > > diff --git a/llvm/include/llvm/CodeGen/TargetInstrInfo.h b/llvm/include/llvm/CodeGen/TargetInstrInfo.h > index 2f853a2c6f9f..1c05afba730d 100644 > --- a/llvm/include/llvm/CodeGen/TargetInstrInfo.h > +++ b/llvm/include/llvm/CodeGen/TargetInstrInfo.h > @@ -117,10 +117,11 @@ public: > const MachineFunction &MF) const; > > /// Return true if the instruction is trivially rematerializable, meaning it > - /// has no side effects and requires no operands that aren't always available. > - /// This means the only allowed uses are constants and unallocatable physical > - /// registers so that the instructions result is independent of the place > - /// in the function. > + /// has no side effects. Uses of constants and unallocatable physical > + /// registers are always trivial to rematerialize so that the instructions > + /// result is independent of the place in the function. Uses of virtual > + /// registers are allowed but it is caller's responsility to ensure these > + /// operands are valid at the point the instruction is beeing moved. > bool isTriviallyReMaterializable(const MachineInstr &MI, > AAResults *AA = nullptr) const { > return MI.getOpcode() == TargetOpcode::IMPLICIT_DEF || > @@ -140,8 +141,7 @@ protected: > /// set, this hook lets the target specify whether the instruction is actually > /// trivially rematerializable, taking into consideration its operands. This > /// predicate must return false if the instruction has any side effects other > - /// than producing a value, or if it requres any address registers that are > - /// not always available. > + /// than producing a value. > /// Requirements must be check as stated in isTriviallyReMaterializable() . > virtual bool isReallyTriviallyReMaterializable(const MachineInstr &MI, > AAResults *AA) const { > diff --git a/llvm/lib/CodeGen/TargetInstrInfo.cpp b/llvm/lib/CodeGen/TargetInstrInfo.cpp > index 1eab8e7443a7..fe7d60e0b7e2 100644 > --- a/llvm/lib/CodeGen/TargetInstrInfo.cpp > +++ b/llvm/lib/CodeGen/TargetInstrInfo.cpp > @@ -921,7 +921,8 @@ bool TargetInstrInfo::isReallyTriviallyReMaterializableGeneric( > const MachineRegisterInfo &MRI = MF.getRegInfo(); > > // Remat clients assume operand 0 is the defined register. > - if (!MI.getNumOperands() || !MI.getOperand(0).isReg()) > + if (!MI.getNumOperands() || !MI.getOperand(0).isReg() || > + MI.getOperand(0).isTied()) > return false; > Register DefReg = MI.getOperand(0).getReg(); > > @@ -983,12 +984,6 @@ bool TargetInstrInfo::isReallyTriviallyReMaterializableGeneric( > // same virtual register, though. > if (MO.isDef() && Reg != DefReg) > return false; > - > - // Don't allow any virtual-register uses. Rematting an instruction with > - // virtual register uses would length the live ranges of the uses, which > - // is not necessarily a good idea, certainly not "trivial". > - if (MO.isUse()) > - return false; > } > > // Everything checked out. > diff --git a/llvm/test/CodeGen/AMDGPU/remat-sop.mir b/llvm/test/CodeGen/AMDGPU/remat-sop.mir > index ed799bfca028..c9915aaabfde 100644 > --- a/llvm/test/CodeGen/AMDGPU/remat-sop.mir > +++ b/llvm/test/CodeGen/AMDGPU/remat-sop.mir > @@ -51,6 +51,66 @@ body: | > S_NOP 0, implicit %2 > S_ENDPGM 0 > ... > +# The liverange of %0 covers a point of rematerialization, source value is > +# availabe. > +--- > +name: test_remat_s_mov_b32_vreg_src_long_lr > +tracksRegLiveness: true > +machineFunctionInfo: > + stackPtrOffsetReg: $sgpr32 > +body: | > + bb.0: > + ; GCN-LABEL: name: test_remat_s_mov_b32_vreg_src_long_lr > + ; GCN: renamable $sgpr0 = IMPLICIT_DEF > + ; GCN: renamable $sgpr1 = S_MOV_B32 renamable $sgpr0 > + ; GCN: S_NOP 0, implicit killed renamable $sgpr1 > + ; GCN: renamable $sgpr1 = S_MOV_B32 renamable $sgpr0 > + ; GCN: S_NOP 0, implicit killed renamable $sgpr1 > + ; GCN: renamable $sgpr1 = S_MOV_B32 renamable $sgpr0 > + ; GCN: S_NOP 0, implicit killed renamable $sgpr1 > + ; GCN: S_NOP 0, implicit killed renamable $sgpr0 > + ; GCN: S_ENDPGM 0 > + %0:sreg_32 = IMPLICIT_DEF > + %1:sreg_32 = S_MOV_B32 %0:sreg_32 > + %2:sreg_32 = S_MOV_B32 %0:sreg_32 > + %3:sreg_32 = S_MOV_B32 %0:sreg_32 > + S_NOP 0, implicit %1 > + S_NOP 0, implicit %2 > + S_NOP 0, implicit %3 > + S_NOP 0, implicit %0 > + S_ENDPGM 0 > +... > +# The liverange of %0 does not cover a point of rematerialization, source value is > +# unavailabe and we do not want to artificially extend the liverange. > +--- > +name: test_no_remat_s_mov_b32_vreg_src_short_lr > +tracksRegLiveness: true > +machineFunctionInfo: > + stackPtrOffsetReg: $sgpr32 > +body: | > + bb.0: > + ; GCN-LABEL: name: test_no_remat_s_mov_b32_vreg_src_short_lr > + ; GCN: renamable $sgpr0 = IMPLICIT_DEF > + ; GCN: renamable $sgpr1 = S_MOV_B32 renamable $sgpr0 > + ; GCN: SI_SPILL_S32_SAVE killed renamable $sgpr1, %stack.1, implicit $exec, implicit $sgpr32 :: (store (s32) into %stack.1, addrspace 5) > + ; GCN: renamable $sgpr1 = S_MOV_B32 renamable $sgpr0 > + ; GCN: SI_SPILL_S32_SAVE killed renamable $sgpr1, %stack.0, implicit $exec, implicit $sgpr32 :: (store (s32) into %stack.0, addrspace 5) > + ; GCN: renamable $sgpr0 = S_MOV_B32 killed renamable $sgpr0 > + ; GCN: renamable $sgpr1 = SI_SPILL_S32_RESTORE %stack.1, implicit $exec, implicit $sgpr32 :: (load (s32) from %stack.1, addrspace 5) > + ; GCN: S_NOP 0, implicit killed renamable $sgpr1 > + ; GCN: renamable $sgpr1 = SI_SPILL_S32_RESTORE %stack.0, implicit $exec, implicit $sgpr32 :: (load (s32) from %stack.0, addrspace 5) > + ; GCN: S_NOP 0, implicit killed renamable $sgpr1 > + ; GCN: S_NOP 0, implicit killed renamable $sgpr0 > + ; GCN: S_ENDPGM 0 > + %0:sreg_32 = IMPLICIT_DEF > + %1:sreg_32 = S_MOV_B32 %0:sreg_32 > + %2:sreg_32 = S_MOV_B32 %0:sreg_32 > + %3:sreg_32 = S_MOV_B32 %0:sreg_32 > + S_NOP 0, implicit %1 > + S_NOP 0, implicit %2 > + S_NOP 0, implicit %3 > + S_ENDPGM 0 > +... > --- > name: test_remat_s_mov_b64 > tracksRegLiveness: true > diff --git a/llvm/test/CodeGen/ARM/arm-shrink-wrapping-linux.ll b/llvm/test/CodeGen/ARM/arm-shrink-wrapping-linux.ll > index a4243276c70a..175a2069a441 100644 > --- a/llvm/test/CodeGen/ARM/arm-shrink-wrapping-linux.ll > +++ b/llvm/test/CodeGen/ARM/arm-shrink-wrapping-linux.ll > @@ -29,20 +29,20 @@ define fastcc i8* @wrongUseOfPostDominate(i8* readonly %s, i32 %off, i8* readnon > ; ENABLE-NEXT: pophs {r11, pc} > ; ENABLE-NEXT: .LBB0_3: @ %while.body.preheader > ; ENABLE-NEXT: movw r12, :lower16:skip > -; ENABLE-NEXT: sub r1, r1, #1 > +; ENABLE-NEXT: sub r3, r1, #1 > ; ENABLE-NEXT: movt r12, :upper16:skip > ; ENABLE-NEXT: .LBB0_4: @ %while.body > ; ENABLE-NEXT: @ =>This Inner Loop Header: Depth=1 > -; ENABLE-NEXT: ldrb r3, [r0] > -; ENABLE-NEXT: ldrb r3, [r12, r3] > -; ENABLE-NEXT: add r0, r0, r3 > -; ENABLE-NEXT: sub r3, r1, #1 > -; ENABLE-NEXT: cmp r3, r1 > +; ENABLE-NEXT: ldrb r1, [r0] > +; ENABLE-NEXT: ldrb r1, [r12, r1] > +; ENABLE-NEXT: add r0, r0, r1 > +; ENABLE-NEXT: sub r1, r3, #1 > +; ENABLE-NEXT: cmp r1, r3 > ; ENABLE-NEXT: bhs .LBB0_6 > ; ENABLE-NEXT: @ %bb.5: @ %while.body > ; ENABLE-NEXT: @ in Loop: Header=BB0_4 Depth=1 > ; ENABLE-NEXT: cmp r0, r2 > -; ENABLE-NEXT: mov r1, r3 > +; ENABLE-NEXT: mov r3, r1 > ; ENABLE-NEXT: blo .LBB0_4 > ; ENABLE-NEXT: .LBB0_6: @ %if.end29 > ; ENABLE-NEXT: pop {r11, pc} > @@ -119,20 +119,20 @@ define fastcc i8* @wrongUseOfPostDominate(i8* readonly %s, i32 %off, i8* readnon > ; DISABLE-NEXT: pophs {r11, pc} > ; DISABLE-NEXT: .LBB0_3: @ %while.body.preheader > ; DISABLE-NEXT: movw r12, :lower16:skip > -; DISABLE-NEXT: sub r1, r1, #1 > +; DISABLE-NEXT: sub r3, r1, #1 > ; DISABLE-NEXT: movt r12, :upper16:skip > ; DISABLE-NEXT: .LBB0_4: @ %while.body > ; DISABLE-NEXT: @ =>This Inner Loop Header: Depth=1 > -; DISABLE-NEXT: ldrb r3, [r0] > -; DISABLE-NEXT: ldrb r3, [r12, r3] > -; DISABLE-NEXT: add r0, r0, r3 > -; DISABLE-NEXT: sub r3, r1, #1 > -; DISABLE-NEXT: cmp r3, r1 > +; DISABLE-NEXT: ldrb r1, [r0] > +; DISABLE-NEXT: ldrb r1, [r12, r1] > +; DISABLE-NEXT: add r0, r0, r1 > +; DISABLE-NEXT: sub r1, r3, #1 > +; DISABLE-NEXT: cmp r1, r3 > ; DISABLE-NEXT: bhs .LBB0_6 > ; DISABLE-NEXT: @ %bb.5: @ %while.body > ; DISABLE-NEXT: @ in Loop: Header=BB0_4 Depth=1 > ; DISABLE-NEXT: cmp r0, r2 > -; DISABLE-NEXT: mov r1, r3 > +; DISABLE-NEXT: mov r3, r1 > ; DISABLE-NEXT: blo .LBB0_4 > ; DISABLE-NEXT: .LBB0_6: @ %if.end29 > ; DISABLE-NEXT: pop {r11, pc} > diff --git a/llvm/test/CodeGen/ARM/funnel-shift-rot.ll b/llvm/test/CodeGen/ARM/funnel-shift-rot.ll > index 55157875d355..ea15fcc5c824 100644 > --- a/llvm/test/CodeGen/ARM/funnel-shift-rot.ll > +++ b/llvm/test/CodeGen/ARM/funnel-shift-rot.ll > @@ -73,13 +73,13 @@ define i64 @rotl_i64(i64 %x, i64 %z) { > ; SCALAR-NEXT: push {r4, r5, r11, lr} > ; SCALAR-NEXT: rsb r3, r2, #0 > ; SCALAR-NEXT: and r4, r2, #63 > -; SCALAR-NEXT: and lr, r3, #63 > -; SCALAR-NEXT: rsb r3, lr, #32 > +; SCALAR-NEXT: and r12, r3, #63 > +; SCALAR-NEXT: rsb r3, r12, #32 > ; SCALAR-NEXT: lsl r2, r0, r4 > -; SCALAR-NEXT: lsr r12, r0, lr > -; SCALAR-NEXT: orr r3, r12, r1, lsl r3 > -; SCALAR-NEXT: subs r12, lr, #32 > -; SCALAR-NEXT: lsrpl r3, r1, r12 > +; SCALAR-NEXT: lsr lr, r0, r12 > +; SCALAR-NEXT: orr r3, lr, r1, lsl r3 > +; SCALAR-NEXT: subs lr, r12, #32 > +; SCALAR-NEXT: lsrpl r3, r1, lr > ; SCALAR-NEXT: subs r5, r4, #32 > ; SCALAR-NEXT: movwpl r2, #0 > ; SCALAR-NEXT: cmp r5, #0 > @@ -88,8 +88,8 @@ define i64 @rotl_i64(i64 %x, i64 %z) { > ; SCALAR-NEXT: lsr r3, r0, r3 > ; SCALAR-NEXT: orr r3, r3, r1, lsl r4 > ; SCALAR-NEXT: lslpl r3, r0, r5 > -; SCALAR-NEXT: lsr r0, r1, lr > -; SCALAR-NEXT: cmp r12, #0 > +; SCALAR-NEXT: lsr r0, r1, r12 > +; SCALAR-NEXT: cmp lr, #0 > ; SCALAR-NEXT: movwpl r0, #0 > ; SCALAR-NEXT: orr r1, r3, r0 > ; SCALAR-NEXT: mov r0, r2 > @@ -245,15 +245,15 @@ define i64 @rotr_i64(i64 %x, i64 %z) { > ; CHECK: @ %bb.0: > ; CHECK-NEXT: .save {r4, r5, r11, lr} > ; CHECK-NEXT: push {r4, r5, r11, lr} > -; CHECK-NEXT: and lr, r2, #63 > +; CHECK-NEXT: and r12, r2, #63 > ; CHECK-NEXT: rsb r2, r2, #0 > -; CHECK-NEXT: rsb r3, lr, #32 > +; CHECK-NEXT: rsb r3, r12, #32 > ; CHECK-NEXT: and r4, r2, #63 > -; CHECK-NEXT: lsr r12, r0, lr > -; CHECK-NEXT: orr r3, r12, r1, lsl r3 > -; CHECK-NEXT: subs r12, lr, #32 > +; CHECK-NEXT: lsr lr, r0, r12 > +; CHECK-NEXT: orr r3, lr, r1, lsl r3 > +; CHECK-NEXT: subs lr, r12, #32 > ; CHECK-NEXT: lsl r2, r0, r4 > -; CHECK-NEXT: lsrpl r3, r1, r12 > +; CHECK-NEXT: lsrpl r3, r1, lr > ; CHECK-NEXT: subs r5, r4, #32 > ; CHECK-NEXT: movwpl r2, #0 > ; CHECK-NEXT: cmp r5, #0 > @@ -262,8 +262,8 @@ define i64 @rotr_i64(i64 %x, i64 %z) { > ; CHECK-NEXT: lsr r3, r0, r3 > ; CHECK-NEXT: orr r3, r3, r1, lsl r4 > ; CHECK-NEXT: lslpl r3, r0, r5 > -; CHECK-NEXT: lsr r0, r1, lr > -; CHECK-NEXT: cmp r12, #0 > +; CHECK-NEXT: lsr r0, r1, r12 > +; CHECK-NEXT: cmp lr, #0 > ; CHECK-NEXT: movwpl r0, #0 > ; CHECK-NEXT: orr r1, r0, r3 > ; CHECK-NEXT: mov r0, r2 > diff --git a/llvm/test/CodeGen/ARM/funnel-shift.ll b/llvm/test/CodeGen/ARM/funnel-shift.ll > index 54c93b493c98..6372f9be2ca3 100644 > --- a/llvm/test/CodeGen/ARM/funnel-shift.ll > +++ b/llvm/test/CodeGen/ARM/funnel-shift.ll > @@ -224,31 +224,31 @@ define i37 @fshr_i37(i37 %x, i37 %y, i37 %z) { > ; CHECK-NEXT: mov r3, #0 > ; CHECK-NEXT: bl __aeabi_uldivmod > ; CHECK-NEXT: add r0, r2, #27 > -; CHECK-NEXT: lsl r6, r6, #27 > -; CHECK-NEXT: and r1, r0, #63 > ; CHECK-NEXT: lsl r2, r7, #27 > +; CHECK-NEXT: and r12, r0, #63 > +; CHECK-NEXT: lsl r6, r6, #27 > ; CHECK-NEXT: orr r7, r6, r7, lsr #5 > +; CHECK-NEXT: rsb r3, r12, #32 > +; CHECK-NEXT: lsr r2, r2, r12 > ; CHECK-NEXT: mov r6, #63 > -; CHECK-NEXT: rsb r3, r1, #32 > -; CHECK-NEXT: lsr r2, r2, r1 > -; CHECK-NEXT: subs r12, r1, #32 > -; CHECK-NEXT: bic r6, r6, r0 > ; CHECK-NEXT: orr r2, r2, r7, lsl r3 > +; CHECK-NEXT: subs r3, r12, #32 > +; CHECK-NEXT: bic r6, r6, r0 > ; CHECK-NEXT: lsl r5, r9, #1 > -; CHECK-NEXT: lsrpl r2, r7, r12 > +; CHECK-NEXT: lsrpl r2, r7, r3 > +; CHECK-NEXT: subs r1, r6, #32 > ; CHECK-NEXT: lsl r0, r5, r6 > -; CHECK-NEXT: subs r4, r6, #32 > -; CHECK-NEXT: lsl r3, r8, #1 > +; CHECK-NEXT: lsl r4, r8, #1 > ; CHECK-NEXT: movwpl r0, #0 > -; CHECK-NEXT: orr r3, r3, r9, lsr #31 > +; CHECK-NEXT: orr r4, r4, r9, lsr #31 > ; CHECK-NEXT: orr r0, r0, r2 > ; CHECK-NEXT: rsb r2, r6, #32 > -; CHECK-NEXT: cmp r4, #0 > -; CHECK-NEXT: lsr r1, r7, r1 > +; CHECK-NEXT: cmp r1, #0 > ; CHECK-NEXT: lsr r2, r5, r2 > -; CHECK-NEXT: orr r2, r2, r3, lsl r6 > -; CHECK-NEXT: lslpl r2, r5, r4 > -; CHECK-NEXT: cmp r12, #0 > +; CHECK-NEXT: orr r2, r2, r4, lsl r6 > +; CHECK-NEXT: lslpl r2, r5, r1 > +; CHECK-NEXT: lsr r1, r7, r12 > +; CHECK-NEXT: cmp r3, #0 > ; CHECK-NEXT: movwpl r1, #0 > ; CHECK-NEXT: orr r1, r2, r1 > ; CHECK-NEXT: pop {r4, r5, r6, r7, r8, r9, r11, pc} > diff --git a/llvm/test/CodeGen/ARM/illegal-bitfield-loadstore.ll b/llvm/test/CodeGen/ARM/illegal-bitfield-loadstore.ll > index 2922e0ed5423..0a0bb62b0a09 100644 > --- a/llvm/test/CodeGen/ARM/illegal-bitfield-loadstore.ll > +++ b/llvm/test/CodeGen/ARM/illegal-bitfield-loadstore.ll > @@ -91,17 +91,17 @@ define void @i56_or(i56* %a) { > ; BE-LABEL: i56_or: > ; BE: @ %bb.0: > ; BE-NEXT: mov r1, r0 > -; BE-NEXT: ldr r12, [r0] > ; BE-NEXT: ldrh r2, [r1, #4]! > ; BE-NEXT: ldrb r3, [r1, #2] > ; BE-NEXT: orr r2, r3, r2, lsl #8 > -; BE-NEXT: orr r2, r2, r12, lsl #24 > -; BE-NEXT: orr r2, r2, #384 > -; BE-NEXT: strb r2, [r1, #2] > -; BE-NEXT: lsr r3, r2, #8 > -; BE-NEXT: strh r3, [r1] > -; BE-NEXT: bic r1, r12, #255 > -; BE-NEXT: orr r1, r1, r2, lsr #24 > +; BE-NEXT: ldr r3, [r0] > +; BE-NEXT: orr r2, r2, r3, lsl #24 > +; BE-NEXT: orr r12, r2, #384 > +; BE-NEXT: strb r12, [r1, #2] > +; BE-NEXT: lsr r2, r12, #8 > +; BE-NEXT: strh r2, [r1] > +; BE-NEXT: bic r1, r3, #255 > +; BE-NEXT: orr r1, r1, r12, lsr #24 > ; BE-NEXT: str r1, [r0] > ; BE-NEXT: mov pc, lr > %aa = load i56, i56* %a > @@ -127,13 +127,13 @@ define void @i56_and_or(i56* %a) { > ; BE-NEXT: ldrb r3, [r1, #2] > ; BE-NEXT: strb r2, [r1, #2] > ; BE-NEXT: orr r2, r3, r12, lsl #8 > -; BE-NEXT: ldr r12, [r0] > -; BE-NEXT: orr r2, r2, r12, lsl #24 > -; BE-NEXT: orr r2, r2, #384 > -; BE-NEXT: lsr r3, r2, #8 > -; BE-NEXT: strh r3, [r1] > -; BE-NEXT: bic r1, r12, #255 > -; BE-NEXT: orr r1, r1, r2, lsr #24 > +; BE-NEXT: ldr r3, [r0] > +; BE-NEXT: orr r2, r2, r3, lsl #24 > +; BE-NEXT: orr r12, r2, #384 > +; BE-NEXT: lsr r2, r12, #8 > +; BE-NEXT: strh r2, [r1] > +; BE-NEXT: bic r1, r3, #255 > +; BE-NEXT: orr r1, r1, r12, lsr #24 > ; BE-NEXT: str r1, [r0] > ; BE-NEXT: mov pc, lr > > diff --git a/llvm/test/CodeGen/ARM/neon-copy.ll b/llvm/test/CodeGen/ARM/neon-copy.ll > index 09a991da2e59..46490efb6631 100644 > --- a/llvm/test/CodeGen/ARM/neon-copy.ll > +++ b/llvm/test/CodeGen/ARM/neon-copy.ll > @@ -1340,16 +1340,16 @@ define <4 x i16> @test_extracts_inserts_varidx_insert(<8 x i16> %x, i32 %idx) { > ; CHECK-NEXT: .pad #8 > ; CHECK-NEXT: sub sp, sp, #8 > ; CHECK-NEXT: vmov.u16 r1, d0[1] > -; CHECK-NEXT: and r0, r0, #3 > +; CHECK-NEXT: and r12, r0, #3 > ; CHECK-NEXT: vmov.u16 r2, d0[2] > -; CHECK-NEXT: mov r3, sp > -; CHECK-NEXT: vmov.u16 r12, d0[3] > -; CHECK-NEXT: orr r0, r3, r0, lsl #1 > +; CHECK-NEXT: mov r0, sp > +; CHECK-NEXT: vmov.u16 r3, d0[3] > +; CHECK-NEXT: orr r0, r0, r12, lsl #1 > ; CHECK-NEXT: vst1.16 {d0[0]}, [r0:16] > ; CHECK-NEXT: vldr d0, [sp] > ; CHECK-NEXT: vmov.16 d0[1], r1 > ; CHECK-NEXT: vmov.16 d0[2], r2 > -; CHECK-NEXT: vmov.16 d0[3], r12 > +; CHECK-NEXT: vmov.16 d0[3], r3 > ; CHECK-NEXT: add sp, sp, #8 > ; CHECK-NEXT: bx lr > %tmp = extractelement <8 x i16> %x, i32 0 > diff --git a/llvm/test/CodeGen/Mips/llvm-ir/ashr.ll b/llvm/test/CodeGen/Mips/llvm-ir/ashr.ll > index 8be7100d368b..a125446b27c3 100644 > --- a/llvm/test/CodeGen/Mips/llvm-ir/ashr.ll > +++ b/llvm/test/CodeGen/Mips/llvm-ir/ashr.ll > @@ -766,79 +766,85 @@ define signext i128 @ashr_i128(i128 signext %a, i128 signext %b) { > ; MMR3-NEXT: .cfi_offset 17, -4 > ; MMR3-NEXT: .cfi_offset 16, -8 > ; MMR3-NEXT: move $8, $7 > -; MMR3-NEXT: sw $6, 32($sp) # 4-byte Folded Spill > -; MMR3-NEXT: sw $5, 36($sp) # 4-byte Folded Spill > -; MMR3-NEXT: sw $4, 8($sp) # 4-byte Folded Spill > +; MMR3-NEXT: move $2, $6 > +; MMR3-NEXT: sw $5, 0($sp) # 4-byte Folded Spill > +; MMR3-NEXT: sw $4, 12($sp) # 4-byte Folded Spill > ; MMR3-NEXT: lw $16, 76($sp) > -; MMR3-NEXT: srlv $4, $7, $16 > -; MMR3-NEXT: not16 $3, $16 > -; MMR3-NEXT: sw $3, 24($sp) # 4-byte Folded Spill > -; MMR3-NEXT: sll16 $2, $6, 1 > -; MMR3-NEXT: sllv $3, $2, $3 > -; MMR3-NEXT: li16 $2, 64 > -; MMR3-NEXT: or16 $3, $4 > -; MMR3-NEXT: srlv $6, $6, $16 > -; MMR3-NEXT: sw $6, 12($sp) # 4-byte Folded Spill > -; MMR3-NEXT: subu16 $7, $2, $16 > +; MMR3-NEXT: srlv $3, $7, $16 > +; MMR3-NEXT: not16 $6, $16 > +; MMR3-NEXT: sw $6, 24($sp) # 4-byte Folded Spill > +; MMR3-NEXT: move $4, $2 > +; MMR3-NEXT: sw $2, 32($sp) # 4-byte Folded Spill > +; MMR3-NEXT: sll16 $2, $2, 1 > +; MMR3-NEXT: sllv $2, $2, $6 > +; MMR3-NEXT: li16 $6, 64 > +; MMR3-NEXT: or16 $2, $3 > +; MMR3-NEXT: srlv $4, $4, $16 > +; MMR3-NEXT: sw $4, 16($sp) # 4-byte Folded Spill > +; MMR3-NEXT: subu16 $7, $6, $16 > ; MMR3-NEXT: sllv $9, $5, $7 > -; MMR3-NEXT: andi16 $2, $7, 32 > -; MMR3-NEXT: sw $2, 28($sp) # 4-byte Folded Spill > -; MMR3-NEXT: andi16 $5, $16, 32 > -; MMR3-NEXT: sw $5, 16($sp) # 4-byte Folded Spill > -; MMR3-NEXT: move $4, $9 > +; MMR3-NEXT: andi16 $5, $7, 32 > +; MMR3-NEXT: sw $5, 28($sp) # 4-byte Folded Spill > +; MMR3-NEXT: andi16 $6, $16, 32 > +; MMR3-NEXT: sw $6, 36($sp) # 4-byte Folded Spill > +; MMR3-NEXT: move $3, $9 > ; MMR3-NEXT: li16 $17, 0 > -; MMR3-NEXT: movn $4, $17, $2 > -; MMR3-NEXT: movn $3, $6, $5 > -; MMR3-NEXT: addiu $2, $16, -64 > -; MMR3-NEXT: lw $5, 36($sp) # 4-byte Folded Reload > -; MMR3-NEXT: srlv $5, $5, $2 > -; MMR3-NEXT: sw $5, 20($sp) # 4-byte Folded Spill > -; MMR3-NEXT: lw $17, 8($sp) # 4-byte Folded Reload > -; MMR3-NEXT: sll16 $6, $17, 1 > -; MMR3-NEXT: sw $6, 4($sp) # 4-byte Folded Spill > -; MMR3-NEXT: not16 $5, $2 > -; MMR3-NEXT: sllv $5, $6, $5 > -; MMR3-NEXT: or16 $3, $4 > -; MMR3-NEXT: lw $4, 20($sp) # 4-byte Folded Reload > -; MMR3-NEXT: or16 $5, $4 > -; MMR3-NEXT: srav $1, $17, $2 > -; MMR3-NEXT: andi16 $2, $2, 32 > -; MMR3-NEXT: sw $2, 20($sp) # 4-byte Folded Spill > -; MMR3-NEXT: movn $5, $1, $2 > -; MMR3-NEXT: sllv $2, $17, $7 > -; MMR3-NEXT: not16 $4, $7 > -; MMR3-NEXT: lw $7, 36($sp) # 4-byte Folded Reload > -; MMR3-NEXT: srl16 $6, $7, 1 > -; MMR3-NEXT: srlv $6, $6, $4 > +; MMR3-NEXT: movn $3, $17, $5 > +; MMR3-NEXT: movn $2, $4, $6 > +; MMR3-NEXT: addiu $4, $16, -64 > +; MMR3-NEXT: lw $17, 0($sp) # 4-byte Folded Reload > +; MMR3-NEXT: srlv $4, $17, $4 > +; MMR3-NEXT: sw $4, 20($sp) # 4-byte Folded Spill > +; MMR3-NEXT: lw $6, 12($sp) # 4-byte Folded Reload > +; MMR3-NEXT: sll16 $4, $6, 1 > +; MMR3-NEXT: sw $4, 8($sp) # 4-byte Folded Spill > +; MMR3-NEXT: addiu $5, $16, -64 > +; MMR3-NEXT: not16 $5, $5 > +; MMR3-NEXT: sllv $5, $4, $5 > +; MMR3-NEXT: or16 $2, $3 > +; MMR3-NEXT: lw $3, 20($sp) # 4-byte Folded Reload > +; MMR3-NEXT: or16 $5, $3 > +; MMR3-NEXT: addiu $3, $16, -64 > +; MMR3-NEXT: srav $1, $6, $3 > +; MMR3-NEXT: andi16 $3, $3, 32 > +; MMR3-NEXT: sw $3, 20($sp) # 4-byte Folded Spill > +; MMR3-NEXT: movn $5, $1, $3 > +; MMR3-NEXT: sllv $3, $6, $7 > +; MMR3-NEXT: sw $3, 4($sp) # 4-byte Folded Spill > +; MMR3-NEXT: not16 $3, $7 > +; MMR3-NEXT: srl16 $4, $17, 1 > +; MMR3-NEXT: srlv $3, $4, $3 > ; MMR3-NEXT: sltiu $10, $16, 64 > -; MMR3-NEXT: movn $5, $3, $10 > -; MMR3-NEXT: or16 $6, $2 > -; MMR3-NEXT: srlv $2, $7, $16 > -; MMR3-NEXT: lw $3, 24($sp) # 4-byte Folded Reload > -; MMR3-NEXT: lw $4, 4($sp) # 4-byte Folded Reload > -; MMR3-NEXT: sllv $3, $4, $3 > +; MMR3-NEXT: movn $5, $2, $10 > +; MMR3-NEXT: lw $2, 4($sp) # 4-byte Folded Reload > ; MMR3-NEXT: or16 $3, $2 > -; MMR3-NEXT: srav $11, $17, $16 > -; MMR3-NEXT: lw $4, 16($sp) # 4-byte Folded Reload > -; MMR3-NEXT: movn $3, $11, $4 > -; MMR3-NEXT: sra $2, $17, 31 > +; MMR3-NEXT: srlv $2, $17, $16 > +; MMR3-NEXT: lw $4, 24($sp) # 4-byte Folded Reload > +; MMR3-NEXT: lw $7, 8($sp) # 4-byte Folded Reload > +; MMR3-NEXT: sllv $17, $7, $4 > +; MMR3-NEXT: or16 $17, $2 > +; MMR3-NEXT: srav $11, $6, $16 > +; MMR3-NEXT: lw $2, 36($sp) # 4-byte Folded Reload > +; MMR3-NEXT: movn $17, $11, $2 > +; MMR3-NEXT: sra $2, $6, 31 > ; MMR3-NEXT: movz $5, $8, $16 > -; MMR3-NEXT: move $8, $2 > -; MMR3-NEXT: movn $8, $3, $10 > -; MMR3-NEXT: lw $3, 28($sp) # 4-byte Folded Reload > -; MMR3-NEXT: movn $6, $9, $3 > -; MMR3-NEXT: li16 $3, 0 > -; MMR3-NEXT: lw $7, 12($sp) # 4-byte Folded Reload > -; MMR3-NEXT: movn $7, $3, $4 > -; MMR3-NEXT: or16 $7, $6 > +; MMR3-NEXT: move $4, $2 > +; MMR3-NEXT: movn $4, $17, $10 > +; MMR3-NEXT: lw $6, 28($sp) # 4-byte Folded Reload > +; MMR3-NEXT: movn $3, $9, $6 > +; MMR3-NEXT: lw $6, 36($sp) # 4-byte Folded Reload > +; MMR3-NEXT: li16 $17, 0 > +; MMR3-NEXT: lw $7, 16($sp) # 4-byte Folded Reload > +; MMR3-NEXT: movn $7, $17, $6 > +; MMR3-NEXT: or16 $7, $3 > ; MMR3-NEXT: lw $3, 20($sp) # 4-byte Folded Reload > ; MMR3-NEXT: movn $1, $2, $3 > ; MMR3-NEXT: movn $1, $7, $10 > ; MMR3-NEXT: lw $3, 32($sp) # 4-byte Folded Reload > ; MMR3-NEXT: movz $1, $3, $16 > -; MMR3-NEXT: movn $11, $2, $4 > +; MMR3-NEXT: movn $11, $2, $6 > ; MMR3-NEXT: movn $2, $11, $10 > -; MMR3-NEXT: move $3, $8 > +; MMR3-NEXT: move $3, $4 > ; MMR3-NEXT: move $4, $1 > ; MMR3-NEXT: lwp $16, 40($sp) > ; MMR3-NEXT: addiusp 48 > @@ -852,79 +858,80 @@ define signext i128 @ashr_i128(i128 signext %a, i128 signext %b) { > ; MMR6-NEXT: sw $16, 8($sp) # 4-byte Folded Spill > ; MMR6-NEXT: .cfi_offset 17, -4 > ; MMR6-NEXT: .cfi_offset 16, -8 > -; MMR6-NEXT: move $1, $7 > +; MMR6-NEXT: move $12, $7 > ; MMR6-NEXT: lw $3, 44($sp) > ; MMR6-NEXT: li16 $2, 64 > -; MMR6-NEXT: subu16 $7, $2, $3 > -; MMR6-NEXT: sllv $8, $5, $7 > -; MMR6-NEXT: andi16 $2, $7, 32 > -; MMR6-NEXT: selnez $9, $8, $2 > -; MMR6-NEXT: sllv $10, $4, $7 > -; MMR6-NEXT: not16 $7, $7 > -; MMR6-NEXT: srl16 $16, $5, 1 > -; MMR6-NEXT: srlv $7, $16, $7 > -; MMR6-NEXT: or $7, $10, $7 > -; MMR6-NEXT: seleqz $7, $7, $2 > -; MMR6-NEXT: or $7, $9, $7 > -; MMR6-NEXT: srlv $9, $1, $3 > -; MMR6-NEXT: not16 $16, $3 > -; MMR6-NEXT: sw $16, 4($sp) # 4-byte Folded Spill > +; MMR6-NEXT: subu16 $16, $2, $3 > +; MMR6-NEXT: sllv $1, $5, $16 > +; MMR6-NEXT: andi16 $2, $16, 32 > +; MMR6-NEXT: selnez $8, $1, $2 > +; MMR6-NEXT: sllv $9, $4, $16 > +; MMR6-NEXT: not16 $16, $16 > +; MMR6-NEXT: srl16 $17, $5, 1 > +; MMR6-NEXT: srlv $10, $17, $16 > +; MMR6-NEXT: or $9, $9, $10 > +; MMR6-NEXT: seleqz $9, $9, $2 > +; MMR6-NEXT: or $8, $8, $9 > +; MMR6-NEXT: srlv $9, $7, $3 > +; MMR6-NEXT: not16 $7, $3 > +; MMR6-NEXT: sw $7, 4($sp) # 4-byte Folded Spill > ; MMR6-NEXT: sll16 $17, $6, 1 > -; MMR6-NEXT: sllv $10, $17, $16 > +; MMR6-NEXT: sllv $10, $17, $7 > ; MMR6-NEXT: or $9, $10, $9 > ; MMR6-NEXT: andi16 $17, $3, 32 > ; MMR6-NEXT: seleqz $9, $9, $17 > ; MMR6-NEXT: srlv $10, $6, $3 > ; MMR6-NEXT: selnez $11, $10, $17 > ; MMR6-NEXT: seleqz $10, $10, $17 > -; MMR6-NEXT: or $10, $10, $7 > -; MMR6-NEXT: seleqz $12, $8, $2 > -; MMR6-NEXT: or $8, $11, $9 > +; MMR6-NEXT: or $8, $10, $8 > +; MMR6-NEXT: seleqz $1, $1, $2 > +; MMR6-NEXT: or $9, $11, $9 > ; MMR6-NEXT: addiu $2, $3, -64 > -; MMR6-NEXT: srlv $9, $5, $2 > +; MMR6-NEXT: srlv $10, $5, $2 > ; MMR6-NEXT: sll16 $7, $4, 1 > ; MMR6-NEXT: not16 $16, $2 > ; MMR6-NEXT: sllv $11, $7, $16 > ; MMR6-NEXT: sltiu $13, $3, 64 > -; MMR6-NEXT: or $8, $8, $12 > -; MMR6-NEXT: selnez $10, $10, $13 > -; MMR6-NEXT: or $9, $11, $9 > -; MMR6-NEXT: srav $11, $4, $2 > +; MMR6-NEXT: or $1, $9, $1 > +; MMR6-NEXT: selnez $8, $8, $13 > +; MMR6-NEXT: or $9, $11, $10 > +; MMR6-NEXT: srav $10, $4, $2 > ; MMR6-NEXT: andi16 $2, $2, 32 > -; MMR6-NEXT: seleqz $12, $11, $2 > +; MMR6-NEXT: seleqz $11, $10, $2 > ; MMR6-NEXT: sra $14, $4, 31 > ; MMR6-NEXT: selnez $15, $14, $2 > ; MMR6-NEXT: seleqz $9, $9, $2 > -; MMR6-NEXT: or $12, $15, $12 > -; MMR6-NEXT: seleqz $12, $12, $13 > -; MMR6-NEXT: selnez $2, $11, $2 > -; MMR6-NEXT: seleqz $11, $14, $13 > -; MMR6-NEXT: or $10, $10, $12 > -; MMR6-NEXT: selnez $10, $10, $3 > -; MMR6-NEXT: selnez $8, $8, $13 > +; MMR6-NEXT: or $11, $15, $11 > +; MMR6-NEXT: seleqz $11, $11, $13 > +; MMR6-NEXT: selnez $2, $10, $2 > +; MMR6-NEXT: seleqz $10, $14, $13 > +; MMR6-NEXT: or $8, $8, $11 > +; MMR6-NEXT: selnez $8, $8, $3 > +; MMR6-NEXT: selnez $1, $1, $13 > ; MMR6-NEXT: or $2, $2, $9 > ; MMR6-NEXT: srav $9, $4, $3 > ; MMR6-NEXT: seleqz $4, $9, $17 > -; MMR6-NEXT: selnez $12, $14, $17 > -; MMR6-NEXT: or $4, $12, $4 > -; MMR6-NEXT: selnez $12, $4, $13 > +; MMR6-NEXT: selnez $11, $14, $17 > +; MMR6-NEXT: or $4, $11, $4 > +; MMR6-NEXT: selnez $11, $4, $13 > ; MMR6-NEXT: seleqz $2, $2, $13 > ; MMR6-NEXT: seleqz $4, $6, $3 > -; MMR6-NEXT: seleqz $1, $1, $3 > -; MMR6-NEXT: or $2, $8, $2 > -; MMR6-NEXT: selnez $2, $2, $3 > +; MMR6-NEXT: seleqz $6, $12, $3 > ; MMR6-NEXT: or $1, $1, $2 > -; MMR6-NEXT: or $4, $4, $10 > -; MMR6-NEXT: or $2, $12, $11 > -; MMR6-NEXT: srlv $3, $5, $3 > -; MMR6-NEXT: lw $5, 4($sp) # 4-byte Folded Reload > -; MMR6-NEXT: sllv $5, $7, $5 > -; MMR6-NEXT: or $3, $5, $3 > -; MMR6-NEXT: seleqz $3, $3, $17 > -; MMR6-NEXT: selnez $5, $9, $17 > -; MMR6-NEXT: or $3, $5, $3 > -; MMR6-NEXT: selnez $3, $3, $13 > -; MMR6-NEXT: or $3, $3, $11 > +; MMR6-NEXT: selnez $1, $1, $3 > +; MMR6-NEXT: or $1, $6, $1 > +; MMR6-NEXT: or $4, $4, $8 > +; MMR6-NEXT: or $6, $11, $10 > +; MMR6-NEXT: srlv $2, $5, $3 > +; MMR6-NEXT: lw $3, 4($sp) # 4-byte Folded Reload > +; MMR6-NEXT: sllv $3, $7, $3 > +; MMR6-NEXT: or $2, $3, $2 > +; MMR6-NEXT: seleqz $2, $2, $17 > +; MMR6-NEXT: selnez $3, $9, $17 > +; MMR6-NEXT: or $2, $3, $2 > +; MMR6-NEXT: selnez $2, $2, $13 > +; MMR6-NEXT: or $3, $2, $10 > +; MMR6-NEXT: move $2, $6 > ; MMR6-NEXT: move $5, $1 > ; MMR6-NEXT: lw $16, 8($sp) # 4-byte Folded Reload > ; MMR6-NEXT: lw $17, 12($sp) # 4-byte Folded Reload > diff --git a/llvm/test/CodeGen/Mips/llvm-ir/lshr.ll b/llvm/test/CodeGen/Mips/llvm-ir/lshr.ll > index ed2bfc9fcf60..e4b4b3ae1d0f 100644 > --- a/llvm/test/CodeGen/Mips/llvm-ir/lshr.ll > +++ b/llvm/test/CodeGen/Mips/llvm-ir/lshr.ll > @@ -776,76 +776,77 @@ define signext i128 @lshr_i128(i128 signext %a, i128 signext %b) { > ; MMR3-NEXT: .cfi_offset 17, -4 > ; MMR3-NEXT: .cfi_offset 16, -8 > ; MMR3-NEXT: move $8, $7 > -; MMR3-NEXT: sw $6, 24($sp) # 4-byte Folded Spill > +; MMR3-NEXT: sw $5, 4($sp) # 4-byte Folded Spill > ; MMR3-NEXT: sw $4, 28($sp) # 4-byte Folded Spill > ; MMR3-NEXT: lw $16, 68($sp) > ; MMR3-NEXT: li16 $2, 64 > -; MMR3-NEXT: subu16 $7, $2, $16 > -; MMR3-NEXT: sllv $9, $5, $7 > -; MMR3-NEXT: move $17, $5 > -; MMR3-NEXT: sw $5, 0($sp) # 4-byte Folded Spill > -; MMR3-NEXT: andi16 $3, $7, 32 > +; MMR3-NEXT: subu16 $17, $2, $16 > +; MMR3-NEXT: sllv $9, $5, $17 > +; MMR3-NEXT: andi16 $3, $17, 32 > ; MMR3-NEXT: sw $3, 20($sp) # 4-byte Folded Spill > ; MMR3-NEXT: li16 $2, 0 > ; MMR3-NEXT: move $4, $9 > ; MMR3-NEXT: movn $4, $2, $3 > -; MMR3-NEXT: srlv $5, $8, $16 > +; MMR3-NEXT: srlv $5, $7, $16 > ; MMR3-NEXT: not16 $3, $16 > ; MMR3-NEXT: sw $3, 16($sp) # 4-byte Folded Spill > ; MMR3-NEXT: sll16 $2, $6, 1 > +; MMR3-NEXT: sw $6, 24($sp) # 4-byte Folded Spill > ; MMR3-NEXT: sllv $2, $2, $3 > ; MMR3-NEXT: or16 $2, $5 > -; MMR3-NEXT: srlv $5, $6, $16 > -; MMR3-NEXT: sw $5, 4($sp) # 4-byte Folded Spill > +; MMR3-NEXT: srlv $7, $6, $16 > ; MMR3-NEXT: andi16 $3, $16, 32 > ; MMR3-NEXT: sw $3, 12($sp) # 4-byte Folded Spill > -; MMR3-NEXT: movn $2, $5, $3 > +; MMR3-NEXT: movn $2, $7, $3 > ; MMR3-NEXT: addiu $3, $16, -64 > ; MMR3-NEXT: or16 $2, $4 > -; MMR3-NEXT: srlv $4, $17, $3 > -; MMR3-NEXT: sw $4, 8($sp) # 4-byte Folded Spill > -; MMR3-NEXT: lw $4, 28($sp) # 4-byte Folded Reload > -; MMR3-NEXT: sll16 $6, $4, 1 > -; MMR3-NEXT: not16 $5, $3 > -; MMR3-NEXT: sllv $5, $6, $5 > -; MMR3-NEXT: lw $17, 8($sp) # 4-byte Folded Reload > -; MMR3-NEXT: or16 $5, $17 > -; MMR3-NEXT: srlv $1, $4, $3 > -; MMR3-NEXT: andi16 $3, $3, 32 > +; MMR3-NEXT: lw $6, 4($sp) # 4-byte Folded Reload > +; MMR3-NEXT: srlv $3, $6, $3 > ; MMR3-NEXT: sw $3, 8($sp) # 4-byte Folded Spill > -; MMR3-NEXT: movn $5, $1, $3 > +; MMR3-NEXT: lw $3, 28($sp) # 4-byte Folded Reload > +; MMR3-NEXT: sll16 $4, $3, 1 > +; MMR3-NEXT: sw $4, 0($sp) # 4-byte Folded Spill > +; MMR3-NEXT: addiu $5, $16, -64 > +; MMR3-NEXT: not16 $5, $5 > +; MMR3-NEXT: sllv $5, $4, $5 > +; MMR3-NEXT: lw $4, 8($sp) # 4-byte Folded Reload > +; MMR3-NEXT: or16 $5, $4 > +; MMR3-NEXT: addiu $4, $16, -64 > +; MMR3-NEXT: srlv $1, $3, $4 > +; MMR3-NEXT: andi16 $4, $4, 32 > +; MMR3-NEXT: sw $4, 8($sp) # 4-byte Folded Spill > +; MMR3-NEXT: movn $5, $1, $4 > ; MMR3-NEXT: sltiu $10, $16, 64 > ; MMR3-NEXT: movn $5, $2, $10 > -; MMR3-NEXT: sllv $2, $4, $7 > -; MMR3-NEXT: not16 $3, $7 > -; MMR3-NEXT: lw $7, 0($sp) # 4-byte Folded Reload > -; MMR3-NEXT: srl16 $4, $7, 1 > +; MMR3-NEXT: sllv $2, $3, $17 > +; MMR3-NEXT: not16 $3, $17 > +; MMR3-NEXT: srl16 $4, $6, 1 > ; MMR3-NEXT: srlv $4, $4, $3 > ; MMR3-NEXT: or16 $4, $2 > -; MMR3-NEXT: srlv $2, $7, $16 > +; MMR3-NEXT: srlv $2, $6, $16 > ; MMR3-NEXT: lw $3, 16($sp) # 4-byte Folded Reload > +; MMR3-NEXT: lw $6, 0($sp) # 4-byte Folded Reload > ; MMR3-NEXT: sllv $3, $6, $3 > ; MMR3-NEXT: or16 $3, $2 > ; MMR3-NEXT: lw $2, 28($sp) # 4-byte Folded Reload > ; MMR3-NEXT: srlv $2, $2, $16 > -; MMR3-NEXT: lw $17, 12($sp) # 4-byte Folded Reload > -; MMR3-NEXT: movn $3, $2, $17 > +; MMR3-NEXT: lw $6, 12($sp) # 4-byte Folded Reload > +; MMR3-NEXT: movn $3, $2, $6 > ; MMR3-NEXT: movz $5, $8, $16 > -; MMR3-NEXT: li16 $6, 0 > -; MMR3-NEXT: movz $3, $6, $10 > -; MMR3-NEXT: lw $7, 20($sp) # 4-byte Folded Reload > -; MMR3-NEXT: movn $4, $9, $7 > -; MMR3-NEXT: lw $6, 4($sp) # 4-byte Folded Reload > -; MMR3-NEXT: li16 $7, 0 > -; MMR3-NEXT: movn $6, $7, $17 > -; MMR3-NEXT: or16 $6, $4 > +; MMR3-NEXT: li16 $17, 0 > +; MMR3-NEXT: movz $3, $17, $10 > +; MMR3-NEXT: lw $17, 20($sp) # 4-byte Folded Reload > +; MMR3-NEXT: movn $4, $9, $17 > +; MMR3-NEXT: li16 $17, 0 > +; MMR3-NEXT: movn $7, $17, $6 > +; MMR3-NEXT: or16 $7, $4 > ; MMR3-NEXT: lw $4, 8($sp) # 4-byte Folded Reload > -; MMR3-NEXT: movn $1, $7, $4 > -; MMR3-NEXT: li16 $7, 0 > -; MMR3-NEXT: movn $1, $6, $10 > +; MMR3-NEXT: movn $1, $17, $4 > +; MMR3-NEXT: li16 $17, 0 > +; MMR3-NEXT: movn $1, $7, $10 > ; MMR3-NEXT: lw $4, 24($sp) # 4-byte Folded Reload > ; MMR3-NEXT: movz $1, $4, $16 > -; MMR3-NEXT: movn $2, $7, $17 > +; MMR3-NEXT: movn $2, $17, $6 > ; MMR3-NEXT: li16 $4, 0 > ; MMR3-NEXT: movz $2, $4, $10 > ; MMR3-NEXT: move $4, $1 > @@ -855,98 +856,91 @@ define signext i128 @lshr_i128(i128 signext %a, i128 signext %b) { > ; > ; MMR6-LABEL: lshr_i128: > ; MMR6: # %bb.0: # %entry > -; MMR6-NEXT: addiu $sp, $sp, -32 > -; MMR6-NEXT: .cfi_def_cfa_offset 32 > -; MMR6-NEXT: sw $17, 28($sp) # 4-byte Folded Spill > -; MMR6-NEXT: sw $16, 24($sp) # 4-byte Folded Spill > +; MMR6-NEXT: addiu $sp, $sp, -24 > +; MMR6-NEXT: .cfi_def_cfa_offset 24 > +; MMR6-NEXT: sw $17, 20($sp) # 4-byte Folded Spill > +; MMR6-NEXT: sw $16, 16($sp) # 4-byte Folded Spill > ; MMR6-NEXT: .cfi_offset 17, -4 > ; MMR6-NEXT: .cfi_offset 16, -8 > ; MMR6-NEXT: move $1, $7 > -; MMR6-NEXT: move $7, $5 > -; MMR6-NEXT: lw $3, 60($sp) > +; MMR6-NEXT: move $7, $4 > +; MMR6-NEXT: lw $3, 52($sp) > ; MMR6-NEXT: srlv $2, $1, $3 > -; MMR6-NEXT: not16 $5, $3 > -; MMR6-NEXT: sw $5, 12($sp) # 4-byte Folded Spill > -; MMR6-NEXT: move $17, $6 > -; MMR6-NEXT: sw $6, 16($sp) # 4-byte Folded Spill > +; MMR6-NEXT: not16 $16, $3 > +; MMR6-NEXT: sw $16, 8($sp) # 4-byte Folded Spill > +; MMR6-NEXT: move $4, $6 > +; MMR6-NEXT: sw $6, 12($sp) # 4-byte Folded Spill > ; MMR6-NEXT: sll16 $6, $6, 1 > -; MMR6-NEXT: sllv $6, $6, $5 > +; MMR6-NEXT: sllv $6, $6, $16 > ; MMR6-NEXT: or $8, $6, $2 > -; MMR6-NEXT: addiu $5, $3, -64 > -; MMR6-NEXT: srlv $9, $7, $5 > -; MMR6-NEXT: move $6, $4 > -; MMR6-NEXT: sll16 $2, $4, 1 > -; MMR6-NEXT: sw $2, 8($sp) # 4-byte Folded Spill > -; MMR6-NEXT: not16 $16, $5 > +; MMR6-NEXT: addiu $6, $3, -64 > +; MMR6-NEXT: srlv $9, $5, $6 > +; MMR6-NEXT: sll16 $2, $7, 1 > +; MMR6-NEXT: sw $2, 4($sp) # 4-byte Folded Spill > +; MMR6-NEXT: not16 $16, $6 > ; MMR6-NEXT: sllv $10, $2, $16 > ; MMR6-NEXT: andi16 $16, $3, 32 > ; MMR6-NEXT: seleqz $8, $8, $16 > ; MMR6-NEXT: or $9, $10, $9 > -; MMR6-NEXT: srlv $10, $17, $3 > +; MMR6-NEXT: srlv $10, $4, $3 > ; MMR6-NEXT: selnez $11, $10, $16 > ; MMR6-NEXT: li16 $17, 64 > ; MMR6-NEXT: subu16 $2, $17, $3 > -; MMR6-NEXT: sllv $12, $7, $2 > -; MMR6-NEXT: move $17, $7 > +; MMR6-NEXT: sllv $12, $5, $2 > ; MMR6-NEXT: andi16 $4, $2, 32 > -; MMR6-NEXT: andi16 $7, $5, 32 > -; MMR6-NEXT: sw $7, 20($sp) # 4-byte Folded Spill > -; MMR6-NEXT: seleqz $9, $9, $7 > +; MMR6-NEXT: andi16 $17, $6, 32 > +; MMR6-NEXT: seleqz $9, $9, $17 > ; MMR6-NEXT: seleqz $13, $12, $4 > ; MMR6-NEXT: or $8, $11, $8 > ; MMR6-NEXT: selnez $11, $12, $4 > -; MMR6-NEXT: sllv $12, $6, $2 > -; MMR6-NEXT: move $7, $6 > -; MMR6-NEXT: sw $6, 4($sp) # 4-byte Folded Spill > +; MMR6-NEXT: sllv $12, $7, $2 > ; MMR6-NEXT: not16 $2, $2 > -; MMR6-NEXT: srl16 $6, $17, 1 > +; MMR6-NEXT: srl16 $6, $5, 1 > ; MMR6-NEXT: srlv $2, $6, $2 > ; MMR6-NEXT: or $2, $12, $2 > ; MMR6-NEXT: seleqz $2, $2, $4 > -; MMR6-NEXT: srlv $4, $7, $5 > -; MMR6-NEXT: or $11, $11, $2 > -; MMR6-NEXT: or $5, $8, $13 > -; MMR6-NEXT: srlv $6, $17, $3 > -; MMR6-NEXT: lw $2, 20($sp) # 4-byte Folded Reload > -; MMR6-NEXT: selnez $7, $4, $2 > -; MMR6-NEXT: sltiu $8, $3, 64 > -; MMR6-NEXT: selnez $12, $5, $8 > -; MMR6-NEXT: or $7, $7, $9 > -; MMR6-NEXT: lw $5, 12($sp) # 4-byte Folded Reload > +; MMR6-NEXT: addiu $4, $3, -64 > +; MMR6-NEXT: srlv $4, $7, $4 > +; MMR6-NEXT: or $12, $11, $2 > +; MMR6-NEXT: or $6, $8, $13 > +; MMR6-NEXT: srlv $5, $5, $3 > +; MMR6-NEXT: selnez $8, $4, $17 > +; MMR6-NEXT: sltiu $11, $3, 64 > +; MMR6-NEXT: selnez $13, $6, $11 > +; MMR6-NEXT: or $8, $8, $9 > ; MMR6-NEXT: lw $2, 8($sp) # 4-byte Folded Reload > -; MMR6-NEXT: sllv $9, $2, $5 > +; MMR6-NEXT: lw $6, 4($sp) # 4-byte Folded Reload > +; MMR6-NEXT: sllv $9, $6, $2 > ; MMR6-NEXT: seleqz $10, $10, $16 > -; MMR6-NEXT: li16 $5, 0 > -; MMR6-NEXT: or $10, $10, $11 > -; MMR6-NEXT: or $6, $9, $6 > -; MMR6-NEXT: seleqz $2, $7, $8 > -; MMR6-NEXT: seleqz $7, $5, $8 > -; MMR6-NEXT: lw $5, 4($sp) # 4-byte Folded Reload > -; MMR6-NEXT: srlv $9, $5, $3 > -; MMR6-NEXT: seleqz $11, $9, $16 > -; MMR6-NEXT: selnez $11, $11, $8 > +; MMR6-NEXT: li16 $2, 0 > +; MMR6-NEXT: or $10, $10, $12 > +; MMR6-NEXT: or $9, $9, $5 > +; MMR6-NEXT: seleqz $5, $8, $11 > +; MMR6-NEXT: seleqz $8, $2, $11 > +; MMR6-NEXT: srlv $7, $7, $3 > +; MMR6-NEXT: seleqz $2, $7, $16 > +; MMR6-NEXT: selnez $2, $2, $11 > ; MMR6-NEXT: seleqz $1, $1, $3 > -; MMR6-NEXT: or $2, $12, $2 > -; MMR6-NEXT: selnez $2, $2, $3 > -; MMR6-NEXT: or $5, $1, $2 > -; MMR6-NEXT: or $2, $7, $11 > -; MMR6-NEXT: seleqz $1, $6, $16 > -; MMR6-NEXT: selnez $6, $9, $16 > -; MMR6-NEXT: lw $16, 16($sp) # 4-byte Folded Reload > -; MMR6-NEXT: seleqz $9, $16, $3 > -; MMR6-NEXT: selnez $10, $10, $8 > -; MMR6-NEXT: lw $16, 20($sp) # 4-byte Folded Reload > -; MMR6-NEXT: seleqz $4, $4, $16 > -; MMR6-NEXT: seleqz $4, $4, $8 > -; MMR6-NEXT: or $4, $10, $4 > +; MMR6-NEXT: or $5, $13, $5 > +; MMR6-NEXT: selnez $5, $5, $3 > +; MMR6-NEXT: or $5, $1, $5 > +; MMR6-NEXT: or $2, $8, $2 > +; MMR6-NEXT: seleqz $1, $9, $16 > +; MMR6-NEXT: selnez $6, $7, $16 > +; MMR6-NEXT: lw $7, 12($sp) # 4-byte Folded Reload > +; MMR6-NEXT: seleqz $7, $7, $3 > +; MMR6-NEXT: selnez $9, $10, $11 > +; MMR6-NEXT: seleqz $4, $4, $17 > +; MMR6-NEXT: seleqz $4, $4, $11 > </cut>

3 years, 9 months

[TCWG CI] 482.sphinx3 slowed down by 4% after gcc: tree-optimization/65206 - dependence analysis on mixed pointer/array

by ci_notify＠linaro.org

After gcc commit f92901a508305f291fcf2acae0825379477724de Author: Richard Biener <rguenther(a)suse.de> tree-optimization/65206 - dependence analysis on mixed pointer/array the following benchmarks slowed down by more than 2%: - 482.sphinx3 slowed down by 4% from 20816 to 21661 perf samples Below reproducer instructions can be used to re-build both "first_bad" and "last_good" cross-toolchains used in this bisection. Naturally, the scripts will fail when triggerring benchmarking jobs if you don't have access to Linaro TCWG CI. For your convenience, we have uploaded tarballs with pre-processed source and assembly files at: - First_bad save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… - Last_good save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… - Baseline save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… Configuration: - Benchmark: SPEC CPU2006 - Toolchain: GCC + Glibc + GNU Linker - Version: all components were built from their tip of trunk - Target: aarch64-linux-gnu - Compiler flags: -O3 - Hardware: NVidia TX1 4x Cortex-A57 This benchmarking CI is work-in-progress, and we welcome feedback and suggestions at linaro-toolchain(a)lists.linaro.org . In our improvement plans is to add support for SPEC CPU2017 benchmarks and provide "perf report/annotate" data behind these reports. THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. This commit has regressed these CI configurations: - tcwg_bmk_gnu_tx1/gnu-master-aarch64-spec2k6-O3 First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… Reproduce builds: <cut> mkdir investigate-gcc-f92901a508305f291fcf2acae0825379477724de cd investigate-gcc-f92901a508305f291fcf2acae0825379477724de # Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts # Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tx1-gnu-master-aa… --fail chmod +x artifacts/test.sh # Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh # Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /gcc/ ./ ./bisect/baseline/ cd gcc # Reproduce first_bad build git checkout --detach f92901a508305f291fcf2acae0825379477724de ../artifacts/test.sh # Reproduce last_good build git checkout --detach abdf63d782cba82b5ecf264248518cbb065650ed ../artifacts/test.sh cd .. </cut> Full commit (up to 1000 lines): <cut> commit f92901a508305f291fcf2acae0825379477724de Author: Richard Biener <rguenther(a)suse.de> Date: Wed Sep 8 14:42:31 2021 +0200 tree-optimization/65206 - dependence analysis on mixed pointer/array This adds the capability to analyze the dependence of mixed pointer/array accesses. The example is from where using a masked load/store creates the pointer-based access when an otherwise unconditional access is array based. Other examples would include accesses to an array mixed with accesses from inlined helpers that work on pointers. The idea is quite simple and old - analyze the data-ref indices as if the reference was pointer-based. The following change does this by changing dr_analyze_indices to work on the indices sub-structure and storing an alternate indices substructure in each data reference. That alternate set of indices is analyzed lazily by initialize_data_dependence_relation when it fails to match-up the main set of indices of two data references. initialize_data_dependence_relation is refactored into a head and a tail worker and changed to work on one of the indices structures and thus away from using DR_* access macros which continue to reference the main indices substructure. There are quite some vectorization and loop distribution opportunities unleashed in SPEC CPU 2017, notably 520.omnetpp_r, 548.exchange2_r, 510.parest_r, 511.povray_r, 521.wrf_r, 526.blender_r, 527.cam4_r and 544.nab_r see amendments in what they report with -fopt-info-loop while the rest of the specrate set sees no changes there. Measuring runtime for the set where changes were reported reveals nothing off-noise besides 511.povray_r which seems to regress slightly for me (on a Zen2 machine with -Ofast -march=native). 2021-09-08 Richard Biener <rguenther(a)suse.de> PR tree-optimization/65206 * tree-data-ref.h (struct data_reference): Add alt_indices, order it last. * tree-data-ref.c (free_data_ref): Release alt_indices. (dr_analyze_indices): Work on struct indices and get DR_REF as tree. (create_data_ref): Adjust. (initialize_data_dependence_relation): Split into head and tail. When the base objects fail to match up try again with pointer-based analysis of indices. * tree-vectorizer.c (vec_info_shared::check_datarefs): Do not compare the lazily computed alternate set of indices. * gcc.dg/torture/20210916.c: New testcase. * gcc.dg/vect/pr65206.c: Likewise. --- gcc/testsuite/gcc.dg/torture/20210916.c | 20 ++++ gcc/testsuite/gcc.dg/vect/pr65206.c | 22 ++++ gcc/tree-data-ref.c | 174 +++++++++++++++++++++----------- gcc/tree-data-ref.h | 9 +- gcc/tree-vectorizer.c | 3 +- 5 files changed, 168 insertions(+), 60 deletions(-) diff --git a/gcc/testsuite/gcc.dg/torture/20210916.c b/gcc/testsuite/gcc.dg/torture/20210916.c new file mode 100644 index 00000000000..0ea6d45e463 --- /dev/null +++ b/gcc/testsuite/gcc.dg/torture/20210916.c @@ -0,0 +1,20 @@ +/* { dg-do compile } */ + +typedef union tree_node *tree; +struct tree_base { + unsigned : 1; + unsigned lang_flag_2 : 1; +}; +struct tree_type { + tree main_variant; +}; +union tree_node { + struct tree_base base; + struct tree_type type; +}; +tree finish_struct_t, finish_struct_x; +void finish_struct() +{ + for (; finish_struct_t->type.main_variant;) + finish_struct_x->base.lang_flag_2 = 0; +} diff --git a/gcc/testsuite/gcc.dg/vect/pr65206.c b/gcc/testsuite/gcc.dg/vect/pr65206.c new file mode 100644 index 00000000000..3b6262622c0 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/pr65206.c @@ -0,0 +1,22 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target vect_double } */ +/* { dg-additional-options "-fno-trapping-math -fno-allow-store-data-races" } */ +/* { dg-additional-options "-mavx" { target avx } } */ + +#define N 1024 + +double a[N], b[N]; + +void foo () +{ + for (int i = 0; i < N; ++i) + if (b[i] < 3.) + a[i] += b[i]; +} + +/* We get a .MASK_STORE because while the load of a[i] does not trap + the store would introduce store data races. Make sure we still + can handle the data dependence with zero distance. */ + +/* { dg-final { scan-tree-dump-not "versioning for alias required" "vect" { target { vect_masked_store || avx } } } } */ +/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" { target { vect_masked_store || avx } } } } */ diff --git a/gcc/tree-data-ref.c b/gcc/tree-data-ref.c index e061baa7c20..18307a554fc 100644 --- a/gcc/tree-data-ref.c +++ b/gcc/tree-data-ref.c @@ -99,6 +99,7 @@ along with GCC; see the file COPYING3. If not see #include "internal-fn.h" #include "vr-values.h" #include "range-op.h" +#include "tree-ssa-loop-ivopts.h" static struct datadep_stats { @@ -1300,22 +1301,18 @@ base_supports_access_fn_components_p (tree base) DR, analyzed in LOOP and instantiated before NEST. */ static void -dr_analyze_indices (struct data_reference *dr, edge nest, loop_p loop) +dr_analyze_indices (struct indices *dri, tree ref, edge nest, loop_p loop) { - vec<tree> access_fns = vNULL; - tree ref, op; - tree base, off, access_fn; - /* If analyzing a basic-block there are no indices to analyze and thus no access functions. */ if (!nest) { - DR_BASE_OBJECT (dr) = DR_REF (dr); - DR_ACCESS_FNS (dr).create (0); + dri->base_object = ref; + dri->access_fns.create (0); return; } - ref = DR_REF (dr); + vec<tree> access_fns = vNULL; /* REALPART_EXPR and IMAGPART_EXPR can be handled like accesses into a two element array with a constant index. The base is @@ -1338,8 +1335,8 @@ dr_analyze_indices (struct data_reference *dr, edge nest, loop_p loop) { if (TREE_CODE (ref) == ARRAY_REF) { - op = TREE_OPERAND (ref, 1); - access_fn = analyze_scalar_evolution (loop, op); + tree op = TREE_OPERAND (ref, 1); + tree access_fn = analyze_scalar_evolution (loop, op); access_fn = instantiate_scev (nest, loop, access_fn); access_fns.safe_push (access_fn); } @@ -1370,16 +1367,16 @@ dr_analyze_indices (struct data_reference *dr, edge nest, loop_p loop) analyzed nest, add it as an additional independent access-function. */ if (TREE_CODE (ref) == MEM_REF) { - op = TREE_OPERAND (ref, 0); - access_fn = analyze_scalar_evolution (loop, op); + tree op = TREE_OPERAND (ref, 0); + tree access_fn = analyze_scalar_evolution (loop, op); access_fn = instantiate_scev (nest, loop, access_fn); if (TREE_CODE (access_fn) == POLYNOMIAL_CHREC) { - tree orig_type; tree memoff = TREE_OPERAND (ref, 1); - base = initial_condition (access_fn); - orig_type = TREE_TYPE (base); + tree base = initial_condition (access_fn); + tree orig_type = TREE_TYPE (base); STRIP_USELESS_TYPE_CONVERSION (base); + tree off; split_constant_offset (base, &base, &off); STRIP_USELESS_TYPE_CONVERSION (base); /* Fold the MEM_REF offset into the evolutions initial @@ -1424,7 +1421,7 @@ dr_analyze_indices (struct data_reference *dr, edge nest, loop_p loop) base, memoff); MR_DEPENDENCE_CLIQUE (ref) = MR_DEPENDENCE_CLIQUE (old); MR_DEPENDENCE_BASE (ref) = MR_DEPENDENCE_BASE (old); - DR_UNCONSTRAINED_BASE (dr) = true; + dri->unconstrained_base = true; access_fns.safe_push (access_fn); } } @@ -1436,8 +1433,8 @@ dr_analyze_indices (struct data_reference *dr, edge nest, loop_p loop) build_int_cst (reference_alias_ptr_type (ref), 0)); } - DR_BASE_OBJECT (dr) = ref; - DR_ACCESS_FNS (dr) = access_fns; + dri->base_object = ref; + dri->access_fns = access_fns; } /* Extracts the alias analysis information from the memory reference DR. */ @@ -1463,6 +1460,8 @@ void free_data_ref (data_reference_p dr) { DR_ACCESS_FNS (dr).release (); + if (dr->alt_indices.base_object) + dr->alt_indices.access_fns.release (); free (dr); } @@ -1497,7 +1496,7 @@ create_data_ref (edge nest, loop_p loop, tree memref, gimple *stmt, dr_analyze_innermost (&DR_INNERMOST (dr), memref, nest != NULL ? loop : NULL, stmt); - dr_analyze_indices (dr, nest, loop); + dr_analyze_indices (&dr->indices, DR_REF (dr), nest, loop); dr_analyze_alias (dr); if (dump_file && (dump_flags & TDF_DETAILS)) @@ -3066,41 +3065,30 @@ access_fn_components_comparable_p (tree ref_a, tree ref_b) TREE_TYPE (TREE_OPERAND (ref_b, 0))); } -/* Initialize a data dependence relation between data accesses A and - B. NB_LOOPS is the number of loops surrounding the references: the - size of the classic distance/direction vectors. */ +/* Initialize a data dependence relation RES in LOOP_NEST. USE_ALT_INDICES + is true when the main indices of A and B were not comparable so we try again + with alternate indices computed on an indirect reference. */ struct data_dependence_relation * -initialize_data_dependence_relation (struct data_reference *a, - struct data_reference *b, - vec<loop_p> loop_nest) +initialize_data_dependence_relation (struct data_dependence_relation *res, + vec<loop_p> loop_nest, + bool use_alt_indices) { - struct data_dependence_relation *res; + struct data_reference *a = DDR_A (res); + struct data_reference *b = DDR_B (res); unsigned int i; - res = XCNEW (struct data_dependence_relation); - DDR_A (res) = a; - DDR_B (res) = b; - DDR_LOOP_NEST (res).create (0); - DDR_SUBSCRIPTS (res).create (0); - DDR_DIR_VECTS (res).create (0); - DDR_DIST_VECTS (res).create (0); - - if (a == NULL || b == NULL) + struct indices *indices_a = &a->indices; + struct indices *indices_b = &b->indices; + if (use_alt_indices) { - DDR_ARE_DEPENDENT (res) = chrec_dont_know; - return res; + if (TREE_CODE (DR_REF (a)) != MEM_REF) + indices_a = &a->alt_indices; + if (TREE_CODE (DR_REF (b)) != MEM_REF) + indices_b = &b->alt_indices; } - - /* If the data references do not alias, then they are independent. */ - if (!dr_may_alias_p (a, b, loop_nest.exists () ? loop_nest[0] : NULL)) - { - DDR_ARE_DEPENDENT (res) = chrec_known; - return res; - } - - unsigned int num_dimensions_a = DR_NUM_DIMENSIONS (a); - unsigned int num_dimensions_b = DR_NUM_DIMENSIONS (b); + unsigned int num_dimensions_a = indices_a->access_fns.length (); + unsigned int num_dimensions_b = indices_b->access_fns.length (); if (num_dimensions_a == 0 || num_dimensions_b == 0) { DDR_ARE_DEPENDENT (res) = chrec_dont_know; @@ -3125,9 +3113,9 @@ initialize_data_dependence_relation (struct data_reference *a, the a and b accesses have a single ARRAY_REF component reference [0] but have two subscripts. */ - if (DR_UNCONSTRAINED_BASE (a)) + if (indices_a->unconstrained_base) num_dimensions_a -= 1; - if (DR_UNCONSTRAINED_BASE (b)) + if (indices_b->unconstrained_base) num_dimensions_b -= 1; /* These structures describe sequences of component references in @@ -3210,6 +3198,10 @@ initialize_data_dependence_relation (struct data_reference *a, B: [3, 4] (i.e. s.e) */ while (index_a < num_dimensions_a && index_b < num_dimensions_b) { + /* The alternate indices form always has a single dimension + with unconstrained base. */ + gcc_assert (!use_alt_indices); + /* REF_A and REF_B must be one of the component access types allowed by dr_analyze_indices. */ gcc_checking_assert (access_fn_component_p (ref_a)); @@ -3280,11 +3272,12 @@ initialize_data_dependence_relation (struct data_reference *a, /* See whether FULL_SEQ ends at the base and whether the two bases are equal. We do not care about TBAA or alignment info so we can use OEP_ADDRESS_OF to avoid false negatives. */ - tree base_a = DR_BASE_OBJECT (a); - tree base_b = DR_BASE_OBJECT (b); + tree base_a = indices_a->base_object; + tree base_b = indices_b->base_object; bool same_base_p = (full_seq.start_a + full_seq.length == num_dimensions_a && full_seq.start_b + full_seq.length == num_dimensions_b - && DR_UNCONSTRAINED_BASE (a) == DR_UNCONSTRAINED_BASE (b) + && (indices_a->unconstrained_base + == indices_b->unconstrained_base) && operand_equal_p (base_a, base_b, OEP_ADDRESS_OF) && (types_compatible_p (TREE_TYPE (base_a), TREE_TYPE (base_b)) @@ -3323,7 +3316,7 @@ initialize_data_dependence_relation (struct data_reference *a, both lvalues are distinct from the object's declared type. */ if (same_base_p) { - if (DR_UNCONSTRAINED_BASE (a)) + if (indices_a->unconstrained_base) full_seq.length += 1; } else @@ -3332,8 +3325,41 @@ initialize_data_dependence_relation (struct data_reference *a, /* Punt if we didn't find a suitable sequence. */ if (full_seq.length == 0) { - DDR_ARE_DEPENDENT (res) = chrec_dont_know; - return res; + if (use_alt_indices + || (TREE_CODE (DR_REF (a)) == MEM_REF + && TREE_CODE (DR_REF (b)) == MEM_REF) + || may_be_nonaddressable_p (DR_REF (a)) + || may_be_nonaddressable_p (DR_REF (b))) + { + /* Fully exhausted possibilities. */ + DDR_ARE_DEPENDENT (res) = chrec_dont_know; + return res; + } + + /* Try evaluating both DRs as dereferences of pointers. */ + if (!a->alt_indices.base_object + && TREE_CODE (DR_REF (a)) != MEM_REF) + { + tree alt_ref = build2 (MEM_REF, TREE_TYPE (DR_REF (a)), + build1 (ADDR_EXPR, ptr_type_node, DR_REF (a)), + build_int_cst + (reference_alias_ptr_type (DR_REF (a)), 0)); + dr_analyze_indices (&a->alt_indices, alt_ref, + loop_preheader_edge (loop_nest[0]), + loop_containing_stmt (DR_STMT (a))); + } + if (!b->alt_indices.base_object + && TREE_CODE (DR_REF (b)) != MEM_REF) + { + tree alt_ref = build2 (MEM_REF, TREE_TYPE (DR_REF (b)), + build1 (ADDR_EXPR, ptr_type_node, DR_REF (b)), + build_int_cst + (reference_alias_ptr_type (DR_REF (b)), 0)); + dr_analyze_indices (&b->alt_indices, alt_ref, + loop_preheader_edge (loop_nest[0]), + loop_containing_stmt (DR_STMT (b))); + } + return initialize_data_dependence_relation (res, loop_nest, true); } if (!same_base_p) @@ -3381,8 +3407,8 @@ initialize_data_dependence_relation (struct data_reference *a, struct subscript *subscript; subscript = XNEW (struct subscript); - SUB_ACCESS_FN (subscript, 0) = DR_ACCESS_FN (a, full_seq.start_a + i); - SUB_ACCESS_FN (subscript, 1) = DR_ACCESS_FN (b, full_seq.start_b + i); + SUB_ACCESS_FN (subscript, 0) = indices_a->access_fns[full_seq.start_a + i]; + SUB_ACCESS_FN (subscript, 1) = indices_b->access_fns[full_seq.start_b + i]; SUB_CONFLICTS_IN_A (subscript) = conflict_fn_not_known (); SUB_CONFLICTS_IN_B (subscript) = conflict_fn_not_known (); SUB_LAST_CONFLICT (subscript) = chrec_dont_know; @@ -3393,6 +3419,40 @@ initialize_data_dependence_relation (struct data_reference *a, return res; } +/* Initialize a data dependence relation between data accesses A and + B. NB_LOOPS is the number of loops surrounding the references: the + size of the classic distance/direction vectors. */ + +struct data_dependence_relation * +initialize_data_dependence_relation (struct data_reference *a, + struct data_reference *b, + vec<loop_p> loop_nest) +{ + data_dependence_relation *res = XCNEW (struct data_dependence_relation); + DDR_A (res) = a; + DDR_B (res) = b; + DDR_LOOP_NEST (res).create (0); + DDR_SUBSCRIPTS (res).create (0); + DDR_DIR_VECTS (res).create (0); + DDR_DIST_VECTS (res).create (0); + + if (a == NULL || b == NULL) + { + DDR_ARE_DEPENDENT (res) = chrec_dont_know; + return res; + } + + /* If the data references do not alias, then they are independent. */ + if (!dr_may_alias_p (a, b, loop_nest.exists () ? loop_nest[0] : NULL)) + { + DDR_ARE_DEPENDENT (res) = chrec_known; + return res; + } + + return initialize_data_dependence_relation (res, loop_nest, false); +} + + /* Frees memory used by the conflict function F. */ static void diff --git a/gcc/tree-data-ref.h b/gcc/tree-data-ref.h index 685f33d85ae..74f579c9f3f 100644 --- a/gcc/tree-data-ref.h +++ b/gcc/tree-data-ref.h @@ -166,14 +166,19 @@ struct data_reference and runs to completion. */ bool is_conditional_in_stmt; + /* Alias information for the data reference. */ + struct dr_alias alias; + /* Behavior of the memory reference in the innermost loop. */ struct innermost_loop_behavior innermost; /* Subscripts of this data reference. */ struct indices indices; - /* Alias information for the data reference. */ - struct dr_alias alias; + /* Alternate subscripts initialized lazily and used by data-dependence + analysis only when the main indices of two DRs are not comparable. + Keep last to keep vec_info_shared::check_datarefs happy. */ + struct indices alt_indices; }; #define DR_STMT(DR) (DR)->stmt diff --git a/gcc/tree-vectorizer.c b/gcc/tree-vectorizer.c index 3aa3e2a6783..20daa31187d 100644 --- a/gcc/tree-vectorizer.c +++ b/gcc/tree-vectorizer.c @@ -507,7 +507,8 @@ vec_info_shared::check_datarefs () return; gcc_assert (datarefs.length () == datarefs_copy.length ()); for (unsigned i = 0; i < datarefs.length (); ++i) - if (memcmp (&datarefs_copy[i], datarefs[i], sizeof (data_reference)) != 0) + if (memcmp (&datarefs_copy[i], datarefs[i], + offsetof (data_reference, alt_indices)) != 0) gcc_unreachable (); } </cut>

3 years, 9 months

[TCWG CI] Regression caused by gcc: Factor predidacte analysis out of tree-ssa-uninit.c into its own module.

by ci_notify＠linaro.org

[TCWG CI] Regression caused by gcc: Factor predidacte analysis out of tree-ssa-uninit.c into its own module.: commit 94c12ffac234b29a702aa7b6730f2678265857c8 Author: Martin Sebor <msebor(a)redhat.com> Factor predidacte analysis out of tree-ssa-uninit.c into its own module. Results regressed to # reset_artifacts: -10 # build_abe binutils: -9 # build_abe stage1: -5 # build_abe qemu: -2 # linux_n_obj: 6240 # First few build errors in logs: from # reset_artifacts: -10 # build_abe binutils: -9 # build_abe stage1: -5 # build_abe qemu: -2 # linux_n_obj: 6999 # linux build successful: all # linux boot successful: boot THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. This commit has regressed these CI configurations: - tcwg_kernel/gnu-master-arm-mainline-defconfig First_bad build: https://ci.linaro.org/job/tcwg_kernel-gnu-bisect-gnu-master-arm-mainline-de… Last_good build: https://ci.linaro.org/job/tcwg_kernel-gnu-bisect-gnu-master-arm-mainline-de… Baseline build: https://ci.linaro.org/job/tcwg_kernel-gnu-bisect-gnu-master-arm-mainline-de… Even more details: https://ci.linaro.org/job/tcwg_kernel-gnu-bisect-gnu-master-arm-mainline-de… Reproduce builds: <cut> mkdir investigate-gcc-94c12ffac234b29a702aa7b6730f2678265857c8 cd investigate-gcc-94c12ffac234b29a702aa7b6730f2678265857c8 # Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts # Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_kernel-gnu-bisect-gnu-master-arm-mainline-de… --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_kernel-gnu-bisect-gnu-master-arm-mainline-de… --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_kernel-gnu-bisect-gnu-master-arm-mainline-de… --fail chmod +x artifacts/test.sh # Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_kernel-build.sh @@ artifacts/manifests/build-baseline.sh # Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /gcc/ ./ ./bisect/baseline/ cd gcc # Reproduce first_bad build git checkout --detach 94c12ffac234b29a702aa7b6730f2678265857c8 ../artifacts/test.sh # Reproduce last_good build git checkout --detach 51166eb2c534692c3c7779def24f83c8c3811b98 ../artifacts/test.sh cd .. </cut> Full commit (up to 1000 lines): <cut> commit 94c12ffac234b29a702aa7b6730f2678265857c8 Author: Martin Sebor <msebor(a)redhat.com> Date: Fri Sep 17 15:39:13 2021 -0600 Factor predidacte analysis out of tree-ssa-uninit.c into its own module. gcc/ChangeLog: * Makefile.in (OBJS): Add gimple-predicate-analysis.o. * tree-ssa-uninit.c (max_phi_args): Move to gimple-predicate-analysis. (MASK_SET_BIT, MASK_TEST_BIT, MASK_EMPTY): Same. (check_defs): Add comment. (can_skip_redundant_opnd): Update comment. (compute_uninit_opnds_pos): Adjust to namespace change. (find_pdom): Move to gimple-predicate-analysis.cc. (find_dom): Same. (struct uninit_undef_val_t): New. (is_non_loop_exit_postdominating): Move to gimple-predicate-analysis.cc. (find_control_equiv_block): Same. (MAX_NUM_CHAINS, MAX_CHAIN_LEN, MAX_POSTDOM_CHECK): Same. (MAX_SWITCH_CASES): Same. (compute_control_dep_chain): Same. (find_uninit_use): Use predicate analyzer. (struct pred_info): Move to gimple-predicate-analysis. (convert_control_dep_chain_into_preds): Same. (find_predicates): Same. (collect_phi_def_edges): Same. (warn_uninitialized_phi): Use predicate analyzer. (find_def_preds): Move to gimple-predicate-analysis. (dump_pred_info): Same. (dump_pred_chain): Same. (dump_predicates): Same. (destroy_predicate_vecs): Remove. (execute_late_warn_uninitialized): New. (get_cmp_code): Move to gimple-predicate-analysis. (is_value_included_in): Same. (value_sat_pred_p): Same. (find_matching_predicate_in_rest_chains): Same. (is_use_properly_guarded): Same. (prune_uninit_phi_opnds): Same. (find_var_cmp_const): Same. (use_pred_not_overlap_with_undef_path_pred): Same. (pred_equal_p): Same. (is_neq_relop_p): Same. (is_neq_zero_form_p): Same. (pred_expr_equal_p): Same. (is_pred_expr_subset_of): Same. (is_pred_chain_subset_of): Same. (is_included_in): Same. (is_superset_of): Same. (pred_neg_p): Same. (simplify_pred): Same. (simplify_preds_2): Same. (simplify_preds_3): Same. (simplify_preds_4): Same. (simplify_preds): Same. (push_pred): Same. (push_to_worklist): Same. (get_pred_info_from_cmp): Same. (is_degenerated_phi): Same. (normalize_one_pred_1): Same. (normalize_one_pred): Same. (normalize_one_pred_chain): Same. (normalize_preds): Same. (can_one_predicate_be_invalidated_p): Same. (can_chain_union_be_invalidated_p): Same. (uninit_uses_cannot_happen): Same. (pass_late_warn_uninitialized::execute): Define. * gimple-predicate-analysis.cc: New file. * gimple-predicate-analysis.h: New file. --- gcc/Makefile.in | 1 + gcc/gimple-predicate-analysis.cc | 2400 +++++++++++++++++++++++++++++++++++++ gcc/gimple-predicate-analysis.h | 158 +++ gcc/tree-ssa-uninit.c | 2431 +++----------------------------------- 4 files changed, 2741 insertions(+), 2249 deletions(-) diff --git a/gcc/Makefile.in b/gcc/Makefile.in index b8229adf580..f36ffa4740b 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -1394,6 +1394,7 @@ OBJS = \ gimple-loop-jam.o \ gimple-loop-versioning.o \ gimple-low.o \ + gimple-predicate-analysis.o \ gimple-pretty-print.o \ gimple-range.o \ gimple-range-cache.o \ diff --git a/gcc/gimple-predicate-analysis.cc b/gcc/gimple-predicate-analysis.cc new file mode 100644 index 00000000000..3404f2d630a --- /dev/null +++ b/gcc/gimple-predicate-analysis.cc @@ -0,0 +1,2400 @@ +/* Support for simple predicate analysis. + + Copyright (C) 2001-2021 Free Software Foundation, Inc. + Contributed by Xinliang David Li <davidxl(a)google.com> + Generalized by Martin Sebor <msebor(a)redhat.com> + + This file is part of GCC. + + GCC is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3, or (at your option) + any later version. + + GCC is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with GCC; see the file COPYING3. If not see + <http://www.gnu.org/licenses/>. */ + +#define INCLUDE_STRING +#include "config.h" +#include "system.h" +#include "coretypes.h" +#include "backend.h" +#include "tree.h" +#include "gimple.h" +#include "tree-pass.h" +#include "ssa.h" +#include "gimple-pretty-print.h" +#include "diagnostic-core.h" +#include "fold-const.h" +#include "gimple-iterator.h" +#include "tree-ssa.h" +#include "tree-cfg.h" +#include "cfghooks.h" +#include "attribs.h" +#include "builtins.h" +#include "calls.h" +#include "value-query.h" + +#include "gimple-predicate-analysis.h" + +#define DEBUG_PREDICATE_ANALYZER 1 + +/* Find the immediate postdominator of the specified basic block BB. */ + +static inline basic_block +find_pdom (basic_block bb) +{ + basic_block exit_bb = EXIT_BLOCK_PTR_FOR_FN (cfun); + if (bb == exit_bb) + return exit_bb; + + if (basic_block pdom = get_immediate_dominator (CDI_POST_DOMINATORS, bb)) + return pdom; + + return exit_bb; +} + +/* Find the immediate dominator of the specified basic block BB. */ + +static inline basic_block +find_dom (basic_block bb) +{ + basic_block entry_bb = ENTRY_BLOCK_PTR_FOR_FN (cfun); + if (bb == entry_bb) + return entry_bb; + + if (basic_block dom = get_immediate_dominator (CDI_DOMINATORS, bb)) + return dom; + + return entry_bb; +} + +/* Return true if BB1 is postdominating BB2 and BB1 is not a loop exit + bb. The loop exit bb check is simple and does not cover all cases. */ + +static bool +is_non_loop_exit_postdominating (basic_block bb1, basic_block bb2) +{ + if (!dominated_by_p (CDI_POST_DOMINATORS, bb2, bb1)) + return false; + + if (single_pred_p (bb1) && !single_succ_p (bb2)) + return false; + + return true; +} + +/* Find BB's closest postdominator that is its control equivalent (i.e., + that's controlled by the same predicate). */ + +static inline basic_block +find_control_equiv_block (basic_block bb) +{ + basic_block pdom = find_pdom (bb); + + /* Skip the postdominating bb that is also a loop exit. */ + if (!is_non_loop_exit_postdominating (pdom, bb)) + return NULL; + + /* If the postdominator is dominated by BB, return it. */ + if (dominated_by_p (CDI_DOMINATORS, pdom, bb)) + return pdom; + + return NULL; +} + +/* Return true if X1 is the negation of X2. */ + +static inline bool +pred_neg_p (const pred_info &x1, const pred_info &x2) +{ + if (!operand_equal_p (x1.pred_lhs, x2.pred_lhs, 0) + || !operand_equal_p (x1.pred_rhs, x2.pred_rhs, 0)) + return false; + + tree_code c1 = x1.cond_code, c2; + if (x1.invert == x2.invert) + c2 = invert_tree_comparison (x2.cond_code, false); + else + c2 = x2.cond_code; + + return c1 == c2; +} + +/* Return whether the condition (VAL CMPC BOUNDARY) is true. */ + +static bool +is_value_included_in (tree val, tree boundary, tree_code cmpc) +{ + /* Only handle integer constant here. */ + if (TREE_CODE (val) != INTEGER_CST || TREE_CODE (boundary) != INTEGER_CST) + return true; + + bool inverted = false; + if (cmpc == GE_EXPR || cmpc == GT_EXPR || cmpc == NE_EXPR) + { + cmpc = invert_tree_comparison (cmpc, false); + inverted = true; + } + + bool result; + if (cmpc == EQ_EXPR) + result = tree_int_cst_equal (val, boundary); + else if (cmpc == LT_EXPR) + result = tree_int_cst_lt (val, boundary); + else + { + gcc_assert (cmpc == LE_EXPR); + result = tree_int_cst_le (val, boundary); + } + + if (inverted) + result ^= 1; + + return result; +} + +/* Format the vector of edges EV as a string. */ + +static std::string +format_edge_vec (const vec<edge> &ev) +{ + std::string str; + + unsigned n = ev.length (); + for (unsigned i = 0; i < n; ++i) + { + char es[32]; + const_edge e = ev[i]; + sprintf (es, "%u", e->src->index); + str += es; + if (i + 1 < n) + str += " -> "; + } + return str; +} + +/* Format the first N elements of the array of vector of edges EVA as + a string. */ + +static std::string +format_edge_vecs (const vec<edge> eva[], unsigned n) +{ + std::string str; + + for (unsigned i = 0; i != n; ++i) + { + str += '{'; + str += format_edge_vec (eva[i]); + str += '}'; + if (i + 1 < n) + str += ", "; + } + return str; +} + +/* Dump a single pred_info to DUMP_FILE. */ + +static void +dump_pred_info (const pred_info &pred) +{ + if (pred.invert) + fprintf (dump_file, "NOT ("); + print_generic_expr (dump_file, pred.pred_lhs); + fprintf (dump_file, " %s ", op_symbol_code (pred.cond_code)); + print_generic_expr (dump_file, pred.pred_rhs); + if (pred.invert) + fputc (')', dump_file); +} + +/* Dump a pred_chain to DUMP_FILE. */ + +static void +dump_pred_chain (const pred_chain &chain) +{ + unsigned np = chain.length (); + if (np > 1) + fprintf (dump_file, "AND ("); + + for (unsigned j = 0; j < np; j++) + { + dump_pred_info (chain[j]); + if (j < np - 1) + fprintf (dump_file, ", "); + else if (j > 0) + fputc (')', dump_file); + } +} + +/* Dump the predicate chain PREDS for STMT, prefixed by MSG. */ + +static void +dump_predicates (gimple *stmt, const pred_chain_union &preds, const char *msg) +{ + fprintf (dump_file, "%s", msg); + if (stmt) + { + print_gimple_stmt (dump_file, stmt, 0); + fprintf (dump_file, "is guarded by:\n"); + } + + unsigned np = preds.length (); + if (np > 1) + fprintf (dump_file, "OR ("); + for (unsigned i = 0; i < np; i++) + { + dump_pred_chain (preds[i]); + if (i < np - 1) + fprintf (dump_file, ", "); + else if (i > 0) + fputc (')', dump_file); + } + fputc ('\n', dump_file); +} + +/* Dump the first NCHAINS elements of the DEP_CHAINS array into DUMP_FILE. */ + +static void +dump_dep_chains (const auto_vec<edge> dep_chains[], unsigned nchains) +{ + if (!dump_file) + return; + + for (unsigned i = 0; i != nchains; ++i) + { + const auto_vec<edge> &v = dep_chains[i]; + unsigned n = v.length (); + for (unsigned j = 0; j != n; ++j) + { + fprintf (dump_file, "%u", v[j]->src->index); + if (j + 1 < n) + fprintf (dump_file, " -> "); + } + fputc ('\n', dump_file); + } +} + +/* Return the 'normalized' conditional code with operand swapping + and condition inversion controlled by SWAP_COND and INVERT. */ + +static tree_code +get_cmp_code (tree_code orig_cmp_code, bool swap_cond, bool invert) +{ + tree_code tc = orig_cmp_code; + + if (swap_cond) + tc = swap_tree_comparison (orig_cmp_code); + if (invert) + tc = invert_tree_comparison (tc, false); + + switch (tc) + { + case LT_EXPR: + case LE_EXPR: + case GT_EXPR: + case GE_EXPR: + case EQ_EXPR: + case NE_EXPR: + break; + default: + return ERROR_MARK; + } + return tc; +} + +/* Return true if PRED is common among all predicate chains in PREDS + (and therefore can be factored out). */ + +static bool +find_matching_predicate_in_rest_chains (const pred_info &pred, + const pred_chain_union &preds) +{ + /* Trival case. */ + if (preds.length () == 1) + return true; + + for (unsigned i = 1; i < preds.length (); i++) + { + bool found = false; + const pred_chain &chain = preds[i]; + unsigned n = chain.length (); + for (unsigned j = 0; j < n; j++) + { + const pred_info &pred2 = chain[j]; + /* Can relax the condition comparison to not use address + comparison. However, the most common case is that + multiple control dependent paths share a common path + prefix, so address comparison should be ok. */ + if (operand_equal_p (pred2.pred_lhs, pred.pred_lhs, 0) + && operand_equal_p (pred2.pred_rhs, pred.pred_rhs, 0) + && pred2.invert == pred.invert) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + +/* Find a predicate to examine against paths of interest. If there + is no predicate of the "FLAG_VAR CMP CONST" form, try to find one + of that's the form "FLAG_VAR CMP FLAG_VAR" with value range info. + PHI is the phi node whose incoming (interesting) paths need to be + examined. On success, return the comparison code, set defintion + gimple of FLAG_DEF and BOUNDARY_CST. Otherwise return ERROR_MARK. */ + +static tree_code +find_var_cmp_const (pred_chain_union preds, gphi *phi, gimple **flag_def, + tree *boundary_cst) +{ + tree_code vrinfo_code = ERROR_MARK; + gimple *vrinfo_def = NULL; + tree vrinfo_cst = NULL; + + gcc_assert (preds.length () > 0); + pred_chain chain = preds[0]; + for (unsigned i = 0; i < chain.length (); i++) + { + bool use_vrinfo_p = false; + const pred_info &pred = chain[i]; + tree cond_lhs = pred.pred_lhs; + tree cond_rhs = pred.pred_rhs; + if (cond_lhs == NULL_TREE || cond_rhs == NULL_TREE) + continue; + + tree_code code = get_cmp_code (pred.cond_code, false, pred.invert); + if (code == ERROR_MARK) + continue; + + /* Convert to the canonical form SSA_NAME CMP CONSTANT. */ + if (TREE_CODE (cond_lhs) == SSA_NAME + && is_gimple_constant (cond_rhs)) + ; + else if (TREE_CODE (cond_rhs) == SSA_NAME + && is_gimple_constant (cond_lhs)) + { + std::swap (cond_lhs, cond_rhs); + if ((code = get_cmp_code (code, true, false)) == ERROR_MARK) + continue; + } + /* Check if we can take advantage of FLAG_VAR COMP FLAG_VAR predicate + with value range info. Note only first of such case is handled. */ + else if (vrinfo_code == ERROR_MARK + && TREE_CODE (cond_lhs) == SSA_NAME + && TREE_CODE (cond_rhs) == SSA_NAME) + { + gimple* lhs_def = SSA_NAME_DEF_STMT (cond_lhs); + if (!lhs_def || gimple_code (lhs_def) != GIMPLE_PHI + || gimple_bb (lhs_def) != gimple_bb (phi)) + { + std::swap (cond_lhs, cond_rhs); + if ((code = get_cmp_code (code, true, false)) == ERROR_MARK) + continue; + } + + /* Check value range info of rhs, do following transforms: + flag_var < [min, max] -> flag_var < max + flag_var > [min, max] -> flag_var > min + + We can also transform LE_EXPR/GE_EXPR to LT_EXPR/GT_EXPR: + flag_var <= [min, max] -> flag_var < [min, max+1] + flag_var >= [min, max] -> flag_var > [min-1, max] + if no overflow/wrap. */ + tree type = TREE_TYPE (cond_lhs); + value_range r; + if (!INTEGRAL_TYPE_P (type) + || !get_range_query (cfun)->range_of_expr (r, cond_rhs) + || r.kind () != VR_RANGE) + continue; + + wide_int min = r.lower_bound (); + wide_int max = r.upper_bound (); + if (code == LE_EXPR + && max != wi::max_value (TYPE_PRECISION (type), TYPE_SIGN (type))) + { + code = LT_EXPR; + max = max + 1; + } + if (code == GE_EXPR + && min != wi::min_value (TYPE_PRECISION (type), TYPE_SIGN (type))) + { + code = GT_EXPR; + min = min - 1; + } + if (code == LT_EXPR) + cond_rhs = wide_int_to_tree (type, max); + else if (code == GT_EXPR) + cond_rhs = wide_int_to_tree (type, min); + else + continue; + + use_vrinfo_p = true; + } + else + continue; + + if ((*flag_def = SSA_NAME_DEF_STMT (cond_lhs)) == NULL) + continue; + + if (gimple_code (*flag_def) != GIMPLE_PHI + || gimple_bb (*flag_def) != gimple_bb (phi) + || !find_matching_predicate_in_rest_chains (pred, preds)) + continue; + + /* Return if any "flag_var comp const" predicate is found. */ + if (!use_vrinfo_p) + { + *boundary_cst = cond_rhs; + return code; + } + /* Record if any "flag_var comp flag_var[vinfo]" predicate is found. */ + else if (vrinfo_code == ERROR_MARK) + { + vrinfo_code = code; + vrinfo_def = *flag_def; + vrinfo_cst = cond_rhs; + } + } + /* Return the "flag_var cmp flag_var[vinfo]" predicate we found. */ + if (vrinfo_code != ERROR_MARK) + { + *flag_def = vrinfo_def; + *boundary_cst = vrinfo_cst; + } + return vrinfo_code; +} + +/* Return true if all interesting opnds are pruned, false otherwise. + PHI is the phi node with interesting operands, OPNDS is the bitmap + of the interesting operand positions, FLAG_DEF is the statement + defining the flag guarding the use of the PHI output, BOUNDARY_CST + is the const value used in the predicate associated with the flag, + CMP_CODE is the comparison code used in the predicate, VISITED_PHIS + is the pointer set of phis visited, and VISITED_FLAG_PHIS is + the pointer to the pointer set of flag definitions that are also + phis. + + Example scenario: + + BB1: + flag_1 = phi <0, 1> // (1) + var_1 = phi <undef, some_val> + + + BB2: + flag_2 = phi <0, flag_1, flag_1> // (2) + var_2 = phi <undef, var_1, var_1> + if (flag_2 == 1) + goto BB3; + + BB3: + use of var_2 // (3) + + Because some flag arg in (1) is not constant, if we do not look into + the flag phis recursively, it is conservatively treated as unknown and + var_1 is thought to flow into use at (3). Since var_1 is potentially + uninitialized a false warning will be emitted. + Checking recursively into (1), the compiler can find out that only + some_val (which is defined) can flow into (3) which is OK. */ + +static bool +prune_phi_opnds (gphi *phi, unsigned opnds, gphi *flag_def, + tree boundary_cst, tree_code cmp_code, + predicate::func_t &eval, + hash_set<gphi *> *visited_phis, + bitmap *visited_flag_phis) +{ + /* The Boolean predicate guarding the PHI definition. Initialized + lazily from PHI in the first call to is_use_guarded() and cached + for subsequent iterations. */ + predicate def_preds (eval); + + unsigned n = MIN (eval.max_phi_args, gimple_phi_num_args (flag_def)); + for (unsigned i = 0; i < n; i++) + { + if (!MASK_TEST_BIT (opnds, i)) + continue; + + tree flag_arg = gimple_phi_arg_def (flag_def, i); + if (!is_gimple_constant (flag_arg)) + { + if (TREE_CODE (flag_arg) != SSA_NAME) + return false; + + gphi *flag_arg_def = dyn_cast<gphi *> (SSA_NAME_DEF_STMT (flag_arg)); + if (!flag_arg_def) + return false; + + tree phi_arg = gimple_phi_arg_def (phi, i); + if (TREE_CODE (phi_arg) != SSA_NAME) + return false; + + gphi *phi_arg_def = dyn_cast<gphi *> (SSA_NAME_DEF_STMT (phi_arg)); + if (!phi_arg_def) + return false; + + if (gimple_bb (phi_arg_def) != gimple_bb (flag_arg_def)) + return false; + + if (!*visited_flag_phis) + *visited_flag_phis = BITMAP_ALLOC (NULL); + + tree phi_result = gimple_phi_result (flag_arg_def); + if (bitmap_bit_p (*visited_flag_phis, SSA_NAME_VERSION (phi_result))) + return false; + + bitmap_set_bit (*visited_flag_phis, SSA_NAME_VERSION (phi_result)); + + /* Now recursively try to prune the interesting phi args. */ + unsigned opnds_arg_phi = eval.phi_arg_set (phi_arg_def); + if (!prune_phi_opnds (phi_arg_def, opnds_arg_phi, flag_arg_def, + boundary_cst, cmp_code, eval, visited_phis, + visited_flag_phis)) + return false; + + bitmap_clear_bit (*visited_flag_phis, SSA_NAME_VERSION (phi_result)); + continue; + } + + /* Now check if the constant is in the guarded range. */ + if (is_value_included_in (flag_arg, boundary_cst, cmp_code)) + { + /* Now that we know that this undefined edge is not pruned. + If the operand is defined by another phi, we can further + prune the incoming edges of that phi by checking + the predicates of this operands. */ + + tree opnd = gimple_phi_arg_def (phi, i); + gimple *opnd_def = SSA_NAME_DEF_STMT (opnd); + if (gphi *opnd_def_phi = dyn_cast <gphi *> (opnd_def)) + { + unsigned opnds2 = eval.phi_arg_set (opnd_def_phi); + if (!MASK_EMPTY (opnds2)) + { + edge opnd_edge = gimple_phi_arg_edge (phi, i); + if (def_preds.is_use_guarded (phi, opnd_edge->src, + opnd_def_phi, opnds2, + visited_phis)) + return false; + } + } + else + return false; + } + } + + return true; +} + +/* Recursively compute the set PHI's incoming edges with "uninteresting" + operands of a phi chain, i.e., those for which EVAL returns false. + CD_ROOT is the control dependence root from which edges are collected + up the CFG nodes that it's dominated by. *EDGES holds the result, and + VISITED is used for detecting cycles. */ + +static void +collect_phi_def_edges (gphi *phi, basic_block cd_root, auto_vec<edge> *edges, + predicate::func_t &eval, hash_set<gimple *> *visited) +{ + if (visited->elements () == 0 + && DEBUG_PREDICATE_ANALYZER + && dump_file) + { + fprintf (dump_file, "%s for cd_root %u and ", + __func__, cd_root->index); + print_gimple_stmt (dump_file, phi, 0); + + } + + if (visited->add (phi)) + return; + + unsigned n = gimple_phi_num_args (phi); + for (unsigned i = 0; i < n; i++) + { + edge opnd_edge = gimple_phi_arg_edge (phi, i); + tree opnd = gimple_phi_arg_def (phi, i); + + if (TREE_CODE (opnd) == SSA_NAME) + { + gimple *def = SSA_NAME_DEF_STMT (opnd); + + if (gimple_code (def) == GIMPLE_PHI + && dominated_by_p (CDI_DOMINATORS, gimple_bb (def), cd_root)) + collect_phi_def_edges (as_a<gphi *> (def), cd_root, edges, eval, + visited); + else if (!eval (opnd)) + { + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, + "\tFound def edge %i -> %i for cd_root %i " + "and operand %u of: ", + opnd_edge->src->index, opnd_edge->dest->index, + cd_root->index, i); + print_gimple_stmt (dump_file, phi, 0); + } + edges->safe_push (opnd_edge); + } + } + else + { + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, + "\tFound def edge %i -> %i for cd_root %i " + "and operand %u of: ", + opnd_edge->src->index, opnd_edge->dest->index, + cd_root->index, i); + print_gimple_stmt (dump_file, phi, 0); + } + + if (!eval (opnd)) + edges->safe_push (opnd_edge); + } + } +} + +/* Return an expression corresponding to the predicate PRED. */ + +static tree +build_pred_expr (const pred_info &pred) +{ + tree_code cond_code = pred.cond_code; + tree lhs = pred.pred_lhs; + tree rhs = pred.pred_rhs; + + if (pred.invert) + cond_code = invert_tree_comparison (cond_code, false); + + return build2 (cond_code, TREE_TYPE (lhs), lhs, rhs); +} + +/* Return an expression corresponding to PREDS. */ + +static tree +build_pred_expr (const pred_chain_union &preds, bool invert = false) +{ + tree_code code = invert ? TRUTH_AND_EXPR : TRUTH_OR_EXPR; + tree_code subcode = invert ? TRUTH_OR_EXPR : TRUTH_AND_EXPR; + + tree expr = NULL_TREE; + for (unsigned i = 0; i != preds.length (); ++i) + { + tree subexpr = NULL_TREE; + for (unsigned j = 0; j != preds[i].length (); ++j) + { + const pred_info &pi = preds[i][j]; + tree cond = build_pred_expr (pi); + if (invert) + cond = invert_truthvalue (cond); + subexpr = subexpr ? build2 (subcode, boolean_type_node, + subexpr, cond) : cond; + } + if (expr) + expr = build2 (code, boolean_type_node, expr, subexpr); + else + expr = subexpr; + } + + return expr; +} + +/* Return a bitset of all PHI arguments or zero if there are too many. */ + +unsigned +predicate::func_t::phi_arg_set (gphi *phi) +{ + unsigned n = gimple_phi_num_args (phi); + + if (max_phi_args < n) + return 0; + + /* Set the least significant N bits. */ + return (1U << n) - 1; +} + +/* Determine if the predicate set of the use does not overlap with that + of the interesting paths. The most common senario of guarded use is + in Example 1: + Example 1: + if (some_cond) + { + x = ...; // set x to valid + flag = true; + } + + ... some code ... + + if (flag) + use (x); // use when x is valid + + The real world examples are usually more complicated, but similar + and usually result from inlining: + + bool init_func (int * x) + { + if (some_cond) + return false; + *x = ...; // set *x to valid + return true; + } + + void foo (..) + { + int x; + + if (!init_func (&x)) + return; + + .. some_code ... + use (x); // use when x is valid + } + + Another possible use scenario is in the following trivial example: + + Example 2: + if (n > 0) + x = 1; + ... + if (n > 0) + { + if (m < 2) + ... = x; + } + + Predicate analysis needs to compute the composite predicate: + + 1) 'x' use predicate: (n > 0) .AND. (m < 2) + 2) 'x' default value (non-def) predicate: .NOT. (n > 0) + (the predicate chain for phi operand defs can be computed + starting from a bb that is control equivalent to the phi's + bb and is dominating the operand def.) + + and check overlapping: + (n > 0) .AND. (m < 2) .AND. (.NOT. (n > 0)) + <==> false + + This implementation provides a framework that can handle different + scenarios. (Note that many simple cases are handled properly without + the predicate analysis if jump threading eliminates the merge point + thus makes path-sensitive analysis unnecessary.) + + PHI is the phi node whose incoming (undefined) paths need to be + pruned, and OPNDS is the bitmap holding interesting operand + positions. VISITED is the pointer set of phi stmts being + checked. */ + +bool +predicate::overlap (gphi *phi, unsigned opnds, hash_set<gphi *> *visited) +{ + gimple *flag_def = NULL; + tree boundary_cst = NULL_TREE; + bitmap visited_flag_phis = NULL; + + /* Find within the common prefix of multiple predicate chains + a predicate that is a comparison of a flag variable against + a constant. */ + tree_code cmp_code = find_var_cmp_const (m_preds, phi, &flag_def, + &boundary_cst); + if (cmp_code == ERROR_MARK) + return true; + + /* Now check all the uninit incoming edges have a constant flag + value that is in conflict with the use guard/predicate. */ + gphi *phi_def = as_a<gphi *> (flag_def); + bool all_pruned = prune_phi_opnds (phi, opnds, phi_def, boundary_cst, + cmp_code, m_eval, visited, + &visited_flag_phis); + + if (visited_flag_phis) + BITMAP_FREE (visited_flag_phis); + + return !all_pruned; +} + +/* Return true if two predicates PRED1 and X2 are equivalent. Assume + the expressions have already properly re-associated. */ + +static inline bool +pred_equal_p (const pred_info &pred1, const pred_info &pred2) +{ + if (!operand_equal_p (pred1.pred_lhs, pred2.pred_lhs, 0) + || !operand_equal_p (pred1.pred_rhs, pred2.pred_rhs, 0)) + return false; + + tree_code c1 = pred1.cond_code, c2; + if (pred1.invert != pred2.invert + && TREE_CODE_CLASS (pred2.cond_code) == tcc_comparison) + c2 = invert_tree_comparison (pred2.cond_code, false); + else + c2 = pred2.cond_code; + + return c1 == c2; +} + +/* Return true if PRED tests inequality (i.e., X != Y). */ + +static inline bool +is_neq_relop_p (const pred_info &pred) +{ + + return ((pred.cond_code == NE_EXPR && !pred.invert) + || (pred.cond_code == EQ_EXPR && pred.invert)); +} + +/* Returns true if PRED is of the form X != 0. */ + +static inline bool +is_neq_zero_form_p (const pred_info &pred) +{ + if (!is_neq_relop_p (pred) || !integer_zerop (pred.pred_rhs) + || TREE_CODE (pred.pred_lhs) != SSA_NAME) + return false; + return true; +} + +/* Return true if PRED is equivalent to X != 0. */ + +static inline bool +pred_expr_equal_p (const pred_info &pred, tree expr) +{ + if (!is_neq_zero_form_p (pred)) + return false; + + return operand_equal_p (pred.pred_lhs, expr, 0); +} + +/* Return true if VAL satisfies (x CMPC BOUNDARY) predicate. CMPC can + be either one of the range comparison codes ({GE,LT,EQ,NE}_EXPR and + the like), or BIT_AND_EXPR. EXACT_P is only meaningful for the latter. + Modify the question from VAL & BOUNDARY != 0 to VAL & BOUNDARY == VAL. + For other values of CMPC, EXACT_P is ignored. */ + +static bool +value_sat_pred_p (tree val, tree boundary, tree_code cmpc, + bool exact_p = false) +{ + if (cmpc != BIT_AND_EXPR) + return is_value_included_in (val, boundary, cmpc); + + wide_int andw = wi::to_wide (val) & wi::to_wide (boundary); + if (exact_p) + return andw == wi::to_wide (val); + + return andw.to_uhwi (); +} + +/* Return true if the domain of single predicate expression PRED1 + is a subset of that of PRED2, and false if it cannot be proved. */ + +static bool +subset_of (const pred_info &pred1, const pred_info &pred2) +{ + if (pred_equal_p (pred1, pred2)) + return true; + </cut>

3 years, 9 months

clang-aarch64-full-2stage buildbot timeout

by Florian Hahn

Hi, It looks like a lot of the recent builds of clang-aarch64-full-2stage are timing out. E.g https://lab.llvm.org/buildbot/#/builders/179/builds/1078 while checking out sources https://lab.llvm.org/buildbot/#/builders/179/builds/1076 during building stage2 Is there anything that could be done to avoid such timeouts and avoid false positive failure emails? Cheers, Florian

3 years, 9 months

[TCWG CI] 403.gcc grew in size by 2% after llvm: Turn on the new pass manager by default

by ci_notify＠linaro.org

After llvm commit 669ddd1e9b1226432b003dbba05b99f8e992285b Author: Arthur Eubanks <aeubanks(a)google.com> Turn on the new pass manager by default the following benchmarks grew in size by more than 1%: - 403.gcc grew in size by 2% from 2586180 to 2648252 bytes Below reproducer instructions can be used to re-build both "first_bad" and "last_good" cross-toolchains used in this bisection. Naturally, the scripts will fail when triggerring benchmarking jobs if you don't have access to Linaro TCWG CI. For your convenience, we have uploaded tarballs with pre-processed source and assembly files at: - First_bad save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… - Last_good save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… - Baseline save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Configuration: - Benchmark: SPEC CPU2006 - Toolchain: Clang + Glibc + LLVM Linker - Version: all components were built from their latest release branch - Target: aarch64-linux-gnu - Compiler flags: -Os -flto - Hardware: APM Mustang 8x X-Gene1 This benchmarking CI is work-in-progress, and we welcome feedback and suggestions at linaro-toolchain(a)lists.linaro.org . In our improvement plans is to add support for SPEC CPU2017 benchmarks and provide "perf report/annotate" data behind these reports. THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. This commit has regressed these CI configurations: - tcwg_bmk_llvm_apm/llvm-release-aarch64-spec2k6-Os_LTO First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Reproduce builds: <cut> mkdir investigate-llvm-669ddd1e9b1226432b003dbba05b99f8e992285b cd investigate-llvm-669ddd1e9b1226432b003dbba05b99f8e992285b # Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts # Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… --fail chmod +x artifacts/test.sh # Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh # Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /llvm/ ./ ./bisect/baseline/ cd llvm # Reproduce first_bad build git checkout --detach 669ddd1e9b1226432b003dbba05b99f8e992285b ../artifacts/test.sh # Reproduce last_good build git checkout --detach b15cbaf5a03d0b32dbc32c37766e32ccf66e6c87 ../artifacts/test.sh cd .. </cut> Full commit (up to 1000 lines): <cut> commit 669ddd1e9b1226432b003dbba05b99f8e992285b Author: Arthur Eubanks <aeubanks(a)google.com> Date: Mon Jan 25 11:00:56 2021 -0800 Turn on the new pass manager by default This turns on the new pass manager by default for the optimization pipeline in Clang and ThinLTO in various LLD backends. This also makes uses of `opt -instcombine` use the new pass manager (unless specifically opted out). This does not affect the backend target-dependent codegen pipeline. If this causes regressions, you can opt out of the new pass manager either via the -DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=OFF CMake flag while building LLVM, or via various compiler flags, e.g. -flegacy-pass-manager for Clang or -Wl,--lto-legacy-pass-manager for ELF LLD. Please file bugs for any regressions. Major differences: * The inliner works slightly differently * -O1 does some amount of inlining * LCSSA and LoopSimplify are run before all loop passes * Loop unswitching is implemented slightly differently * A new SpeculateAroundPHIs pass is added to the pipeline https://lists.llvm.org/pipermail/llvm-dev/2021-January/148098.html Reviewed By: asbirlea, ychen, MaskRay, echristo Differential Revision: https://reviews.llvm.org/D95380 --- llvm/CMakeLists.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/llvm/CMakeLists.txt b/llvm/CMakeLists.txt index 1affc289e64b..f5298de9f7ca 100644 --- a/llvm/CMakeLists.txt +++ b/llvm/CMakeLists.txt @@ -688,8 +688,8 @@ else() endif() option(LLVM_ENABLE_PLUGINS "Enable plugin support" ${LLVM_ENABLE_PLUGINS_default}) -set(ENABLE_EXPERIMENTAL_NEW_PASS_MANAGER FALSE CACHE BOOL - "Enable the experimental new pass manager by default.") +set(ENABLE_EXPERIMENTAL_NEW_PASS_MANAGER TRUE CACHE BOOL + "Enable the new pass manager by default.") include(HandleLLVMOptions) </cut>

3 years, 9 months

[TCWG CI] Regression caused by linux: parisc: Declare pci_iounmap() parisc version only when CONFIG_PCI enabled

by ci_notify＠linaro.org

[TCWG CI] Regression caused by linux: parisc: Declare pci_iounmap() parisc version only when CONFIG_PCI enabled: commit 9caea0007601d3bc6debec04f8b4cd6f4c2394be Author: Helge Deller <deller(a)gmx.de> parisc: Declare pci_iounmap() parisc version only when CONFIG_PCI enabled Results regressed to # reset_artifacts: -10 # build_abe binutils: -9 # build_llvm: -5 # build_abe qemu: -2 # linux_n_obj: 37 # First few build errors in logs: from # reset_artifacts: -10 # build_abe binutils: -9 # build_llvm: -5 # build_abe qemu: -2 # linux_n_obj: 20151 THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. This commit has regressed these CI configurations: - tcwg_kernel/llvm-master-aarch64-mainline-allyesconfig First_bad build: https://ci.linaro.org/job/tcwg_kernel-llvm-bisect-llvm-master-aarch64-mainl… Last_good build: https://ci.linaro.org/job/tcwg_kernel-llvm-bisect-llvm-master-aarch64-mainl… Baseline build: https://ci.linaro.org/job/tcwg_kernel-llvm-bisect-llvm-master-aarch64-mainl… Even more details: https://ci.linaro.org/job/tcwg_kernel-llvm-bisect-llvm-master-aarch64-mainl… Reproduce builds: <cut> mkdir investigate-linux-9caea0007601d3bc6debec04f8b4cd6f4c2394be cd investigate-linux-9caea0007601d3bc6debec04f8b4cd6f4c2394be # Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts # Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_kernel-llvm-bisect-llvm-master-aarch64-mainl… --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_kernel-llvm-bisect-llvm-master-aarch64-mainl… --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_kernel-llvm-bisect-llvm-master-aarch64-mainl… --fail chmod +x artifacts/test.sh # Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_kernel-build.sh @@ artifacts/manifests/build-baseline.sh # Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /linux/ ./ ./bisect/baseline/ cd linux # Reproduce first_bad build git checkout --detach 9caea0007601d3bc6debec04f8b4cd6f4c2394be ../artifacts/test.sh # Reproduce last_good build git checkout --detach 31ad37bd6faf871c070650f72ac9488ceeeceeb0 ../artifacts/test.sh cd .. </cut> Full commit (up to 1000 lines): <cut> commit 9caea0007601d3bc6debec04f8b4cd6f4c2394be Author: Helge Deller <deller(a)gmx.de> Date: Sun Sep 19 10:36:09 2021 -0700 parisc: Declare pci_iounmap() parisc version only when CONFIG_PCI enabled Linus noticed odd declaration rules for pci_iounmap() in iomap.h and pci_iomap.h, where it dependend on either NO_GENERIC_PCI_IOPORT_MAP or GENERIC_IOMAP when CONFIG_PCI was disabled. Testing on parisc seems to indicate that we need pci_iounmap() only when CONFIG_PCI is enabled, so the declaration of pci_iounmap() can be moved cleanly into pci_iomap.h in sync with the declarations of pci_iomap(). Link: https://lore.kernel.org/all/CAHk-=wjRrh98pZoQ+AzfWmsTZacWxTJKXZ9eKU2X_0+jM=… Signed-off-by: Helge Deller <deller(a)gmx.de> Suggested-by: Linus Torvalds <torvalds(a)linux-foundation.org> Fixes: 97a29d59fc22 ("[PARISC] fix compile break caused by iomap: make IOPORT/PCI mapping functions conditional") Cc: Arnd Bergmann <arnd(a)arndb.de> Cc: Guenter Roeck <linux(a)roeck-us.net> Cc: Ulrich Teichert <krypton(a)ulrich-teichert.org> Cc: James Bottomley <James.Bottomley(a)hansenpartnership.com> Signed-off-by: Linus Torvalds <torvalds(a)linux-foundation.org> --- arch/parisc/lib/iomap.c | 4 +++- include/asm-generic/iomap.h | 10 ---------- include/asm-generic/pci_iomap.h | 3 +++ 3 files changed, 6 insertions(+), 11 deletions(-) diff --git a/arch/parisc/lib/iomap.c b/arch/parisc/lib/iomap.c index f03adb1999e7..367f6397bda7 100644 --- a/arch/parisc/lib/iomap.c +++ b/arch/parisc/lib/iomap.c @@ -513,12 +513,15 @@ void ioport_unmap(void __iomem *addr) } } +#ifdef CONFIG_PCI void pci_iounmap(struct pci_dev *dev, void __iomem * addr) { if (!INDIRECT_ADDR(addr)) { iounmap(addr); } } +EXPORT_SYMBOL(pci_iounmap); +#endif EXPORT_SYMBOL(ioread8); EXPORT_SYMBOL(ioread16); @@ -544,4 +547,3 @@ EXPORT_SYMBOL(iowrite16_rep); EXPORT_SYMBOL(iowrite32_rep); EXPORT_SYMBOL(ioport_map); EXPORT_SYMBOL(ioport_unmap); -EXPORT_SYMBOL(pci_iounmap); diff --git a/include/asm-generic/iomap.h b/include/asm-generic/iomap.h index 9b3eb6d86200..08237ae8b840 100644 --- a/include/asm-generic/iomap.h +++ b/include/asm-generic/iomap.h @@ -110,16 +110,6 @@ static inline void __iomem *ioremap_np(phys_addr_t offset, size_t size) } #endif -#ifdef CONFIG_PCI -/* Destroy a virtual mapping cookie for a PCI BAR (memory or IO) */ -struct pci_dev; -extern void pci_iounmap(struct pci_dev *dev, void __iomem *); -#elif defined(CONFIG_GENERIC_IOMAP) -struct pci_dev; -static inline void pci_iounmap(struct pci_dev *dev, void __iomem *addr) -{ } -#endif - #include <asm-generic/pci_iomap.h> #endif diff --git a/include/asm-generic/pci_iomap.h b/include/asm-generic/pci_iomap.h index df636c6d8e6c..5a2f9bf53384 100644 --- a/include/asm-generic/pci_iomap.h +++ b/include/asm-generic/pci_iomap.h @@ -18,6 +18,7 @@ extern void __iomem *pci_iomap_range(struct pci_dev *dev, int bar, extern void __iomem *pci_iomap_wc_range(struct pci_dev *dev, int bar, unsigned long offset, unsigned long maxlen); +extern void pci_iounmap(struct pci_dev *dev, void __iomem *); /* Create a virtual mapping cookie for a port on a given PCI device. * Do not call this directly, it exists to make it easier for architectures * to override */ @@ -50,6 +51,8 @@ static inline void __iomem *pci_iomap_wc_range(struct pci_dev *dev, int bar, { return NULL; } +static inline void pci_iounmap(struct pci_dev *dev, void __iomem *addr) +{ } #endif #endif /* __ASM_GENERIC_PCI_IOMAP_H */ </cut>

3 years, 9 months

[TCWG CI] 447.dealII:libstdc++.so.6.0.29 grew in size by 12% after gcc: libstdc++: Add floating-point std::to_chars implementation

by ci_notify＠linaro.org

After gcc commit 3c57e692357c79ee7623dfc1586652aee2aefb8f Author: Patrick Palka <ppalka(a)redhat.com> libstdc++: Add floating-point std::to_chars implementation the following hot functions grew in size by more than 10% (but their benchmarks grew in size by less than 1%): - 447.dealII:libstdc++.so.6.0.29 grew in size by 12% from 1245370 to 1391240 bytes Below reproducer instructions can be used to re-build both "first_bad" and "last_good" cross-toolchains used in this bisection. Naturally, the scripts will fail when triggerring benchmarking jobs if you don't have access to Linaro TCWG CI. For your convenience, we have uploaded tarballs with pre-processed source and assembly files at: - First_bad save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… - Last_good save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… - Baseline save-temps: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Configuration: - Benchmark: SPEC CPU2006 - Toolchain: Clang + Glibc + LLVM Linker - Version: all components were built from their latest release branch - Target: arm-linux-gnueabihf - Compiler flags: -Os -mthumb - Hardware: APM Mustang 8x X-Gene1 This benchmarking CI is work-in-progress, and we welcome feedback and suggestions at linaro-toolchain(a)lists.linaro.org . In our improvement plans is to add support for SPEC CPU2017 benchmarks and provide "perf report/annotate" data behind these reports. THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. This commit has regressed these CI configurations: - tcwg_bmk_llvm_apm/llvm-release-arm-spec2k6-Os First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… Reproduce builds: <cut> mkdir investigate-gcc-3c57e692357c79ee7623dfc1586652aee2aefb8f cd investigate-gcc-3c57e692357c79ee7623dfc1586652aee2aefb8f # Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts # Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-release… --fail chmod +x artifacts/test.sh # Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh # Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /gcc/ ./ ./bisect/baseline/ cd gcc # Reproduce first_bad build git checkout --detach 3c57e692357c79ee7623dfc1586652aee2aefb8f ../artifacts/test.sh # Reproduce last_good build git checkout --detach 5033506993ef92589373270a8e8dbbf50e3ebef1 ../artifacts/test.sh cd .. </cut> Full commit (up to 1000 lines): <cut> commit 3c57e692357c79ee7623dfc1586652aee2aefb8f Author: Patrick Palka <ppalka(a)redhat.com> Date: Thu Dec 17 23:11:34 2020 -0500 libstdc++: Add floating-point std::to_chars implementation This implements the floating-point std::to_chars overloads for float, double and long double. We use the Ryu library to compute the shortest round-trippable fixed and scientific forms for float, double and long double. We also use Ryu for performing explicit-precision fixed and scientific formatting for float and double. For explicit-precision formatting for long double we fall back to using printf. Hexadecimal formatting for float, double and long double is implemented from scratch. The supported long double binary formats are binary64, binary80 (x86 80-bit extended precision), binary128 and ibm128. Much of the complexity of the implementation is in computing the exact output length before handing it off to Ryu (which doesn't do bounds checking). In some cases it's hard to compute the output length beforehand, so in these cases we instead compute an upper bound on the output length and use a sufficiently-sized intermediate buffer only if necessary. Another source of complexity is in the general-with-precision formatting mode, where we need to do zero-trimming of the string returned by Ryu, and where we also take care to avoid having to format the number through Ryu a second time when the general formatting mode resolves to fixed (which we determine by doing a scientific formatting first and inspecting the scientific exponent). We avoid going through Ryu twice by instead transforming the scientific form to the corresponding fixed form via in-place string manipulation. This implementation is non-conforming in a couple of ways: 1. For the shortest hexadecimal formatting, we currently follow the Microsoft implementation's decision to be consistent with the output of printf's '%a' specifier at the expense of sometimes not printing the shortest representation. For example, the shortest hex form for the number 1.08p+0 is 2.1p-1, but we output the former instead of the latter, as does printf. 2. The Ryu routine generic_binary_to_decimal that we use for performing shortest formatting for large floating point types is implemented using the __int128 type, but some targets with a large long double type lack __int128 (e.g. i686), so we can't perform shortest formatting of long double on such targets through Ryu. As a temporary stopgap this patch makes the long double to_chars overloads just dispatch to the double overloads on these targets, which means we lose precision in the output. (We could potentially fix this by writing a specialized version of Ryu's generic_binary_to_decimal routine that uses uint64_t instead of __int128.) [Though I wonder if there's a better way to work around the lack of __int128 on i686 specifically?] 3. Our shortest formatting for __ibm128 doesn't guarantee the round-trip property if the difference between the high- and low-order exponent is large. This is because we treat __ibm128 as if it has a contiguous 105-bit mantissa by merging the mantissas of the high- and low-order parts (using code extracted from glibc), so we potentially lose precision from the low-order part. This seems to be consistent with how glibc printf formats __ibm128. libstdc++-v3/ChangeLog: * config/abi/pre/gnu.ver: Add new exports. * include/std/charconv (to_chars): Declare the floating-point overloads for float, double and long double. * src/c++17/Makefile.am (sources): Add floating_to_chars.cc. * src/c++17/Makefile.in: Regenerate. * src/c++17/floating_to_chars.cc: New file. (to_chars): Define for float, double and long double. * testsuite/20_util/to_chars/long_double.cc: New test. --- libstdc++-v3/config/abi/pre/gnu.ver | 7 + libstdc++-v3/include/std/charconv | 24 + libstdc++-v3/src/c++17/Makefile.am | 1 + libstdc++-v3/src/c++17/Makefile.in | 3 +- libstdc++-v3/src/c++17/floating_to_chars.cc | 1563 ++++++++++++++++++++ .../testsuite/20_util/to_chars/long_double.cc | 199 +++ 6 files changed, 1796 insertions(+), 1 deletion(-) diff --git a/libstdc++-v3/config/abi/pre/gnu.ver b/libstdc++-v3/config/abi/pre/gnu.ver index 4b4bd8ab6da..05e0a512247 100644 --- a/libstdc++-v3/config/abi/pre/gnu.ver +++ b/libstdc++-v3/config/abi/pre/gnu.ver @@ -2393,6 +2393,13 @@ GLIBCXX_3.4.29 { # std::once_flag::_M_finish(bool) _ZNSt9once_flag9_M_finishEb; + # std::to_chars(char*, char*, [float|double|long double]) + _ZSt8to_charsPcS_[defg]; + # std::to_chars(char*, char*, [float|double|long double], chars_format) + _ZSt8to_charsPcS_[defg]St12chars_format; + # std::to_chars(char*, char*, [float|double|long double], chars_format, int) + _ZSt8to_charsPcS_[defg]St12chars_formati; + } GLIBCXX_3.4.28; # Symbols in the support library (libsupc++) have their own tag. diff --git a/libstdc++-v3/include/std/charconv b/libstdc++-v3/include/std/charconv index dd1ebdf8322..b57b0a16db2 100644 --- a/libstdc++-v3/include/std/charconv +++ b/libstdc++-v3/include/std/charconv @@ -702,6 +702,30 @@ namespace __detail chars_format __fmt = chars_format::general) noexcept; #endif + // Floating-point std::to_chars + + // Overloads for float. + to_chars_result to_chars(char* __first, char* __last, float __value) noexcept; + to_chars_result to_chars(char* __first, char* __last, float __value, + chars_format __fmt) noexcept; + to_chars_result to_chars(char* __first, char* __last, float __value, + chars_format __fmt, int __precision) noexcept; + + // Overloads for double. + to_chars_result to_chars(char* __first, char* __last, double __value) noexcept; + to_chars_result to_chars(char* __first, char* __last, double __value, + chars_format __fmt) noexcept; + to_chars_result to_chars(char* __first, char* __last, double __value, + chars_format __fmt, int __precision) noexcept; + + // Overloads for long double. + to_chars_result to_chars(char* __first, char* __last, long double __value) + noexcept; + to_chars_result to_chars(char* __first, char* __last, long double __value, + chars_format __fmt) noexcept; + to_chars_result to_chars(char* __first, char* __last, long double __value, + chars_format __fmt, int __precision) noexcept; + _GLIBCXX_END_NAMESPACE_VERSION } // namespace std #endif // C++14 diff --git a/libstdc++-v3/src/c++17/Makefile.am b/libstdc++-v3/src/c++17/Makefile.am index 37cdb53c076..2ec5ed621ca 100644 --- a/libstdc++-v3/src/c++17/Makefile.am +++ b/libstdc++-v3/src/c++17/Makefile.am @@ -51,6 +51,7 @@ endif sources = \ floating_from_chars.cc \ + floating_to_chars.cc \ fs_dir.cc \ fs_ops.cc \ fs_path.cc \ diff --git a/libstdc++-v3/src/c++17/Makefile.in b/libstdc++-v3/src/c++17/Makefile.in index ccae721ab3f..9b36b7a916c 100644 --- a/libstdc++-v3/src/c++17/Makefile.in +++ b/libstdc++-v3/src/c++17/Makefile.in @@ -124,7 +124,7 @@ LTLIBRARIES = $(noinst_LTLIBRARIES) libc__17convenience_la_LIBADD = @ENABLE_DUAL_ABI_TRUE@am__objects_1 = cow-fs_dir.lo cow-fs_ops.lo \ @ENABLE_DUAL_ABI_TRUE@ cow-fs_path.lo -am__objects_2 = floating_from_chars.lo fs_dir.lo fs_ops.lo fs_path.lo \ +am__objects_2 = floating_from_chars.lo floating_to_chars.lo fs_dir.lo fs_ops.lo fs_path.lo \ memory_resource.lo $(am__objects_1) @ENABLE_DUAL_ABI_TRUE@am__objects_3 = cow-string-inst.lo @ENABLE_EXTERN_TEMPLATE_TRUE@am__objects_4 = ostream-inst.lo \ @@ -440,6 +440,7 @@ headers = sources = \ floating_from_chars.cc \ + floating_to_chars.cc \ fs_dir.cc \ fs_ops.cc \ fs_path.cc \ diff --git a/libstdc++-v3/src/c++17/floating_to_chars.cc b/libstdc++-v3/src/c++17/floating_to_chars.cc new file mode 100644 index 00000000000..dd83f5eea93 --- /dev/null +++ b/libstdc++-v3/src/c++17/floating_to_chars.cc @@ -0,0 +1,1563 @@ +// std::to_chars implementation for floating-point types -*- C++ -*- + +// Copyright (C) 2020 Free Software Foundation, Inc. +// +// This file is part of the GNU ISO C++ Library. This library is free +// software; you can redistribute it and/or modify it under the +// terms of the GNU General Public License as published by the +// Free Software Foundation; either version 3, or (at your option) +// any later version. + +// This library is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. + +// Under Section 7 of GPL version 3, you are granted additional +// permissions described in the GCC Runtime Library Exception, version +// 3.1, as published by the Free Software Foundation. + +// You should have received a copy of the GNU General Public License and +// a copy of the GCC Runtime Library Exception along with this program; +// see the files COPYING3 and COPYING.RUNTIME respectively. If not, see +// <http://www.gnu.org/licenses/>. + +// Activate __glibcxx_assert within this file to shake out any bugs. +#define _GLIBCXX_ASSERTIONS 1 + +#include <charconv> + +#include <bit> +#include <cfenv> +#include <cassert> +#include <cmath> +#include <cstdio> +#include <cstring> +#include <langinfo.h> +#include <optional> +#include <string_view> +#include <type_traits> + +// Determine the binary format of 'long double'. + +// We support the binary64, float80 (i.e. x86 80-bit extended precision), +// binary128, and ibm128 formats. +#define LDK_UNSUPPORTED 0 +#define LDK_BINARY64 1 +#define LDK_FLOAT80 2 +#define LDK_BINARY128 3 +#define LDK_IBM128 4 + +#if __LDBL_MANT_DIG__ == __DBL_MANT_DIG__ +# define LONG_DOUBLE_KIND LDK_BINARY64 +#elif defined(__SIZEOF_INT128__) +// The Ryu routines need a 128-bit integer type in order to do shortest +// formatting of types larger than 64-bit double, so without __int128 we can't +// support any large long double format. This is the case for e.g. i386. +# if __LDBL_MANT_DIG__ == 64 +# define LONG_DOUBLE_KIND LDK_FLOAT80 +# elif __LDBL_MANT_DIG__ == 113 +# define LONG_DOUBLE_KIND LDK_BINARY128 +# elif __LDBL_MANT_DIG__ == 106 +# define LONG_DOUBLE_KIND LDK_IBM128 +# endif +#endif +#if !defined(LONG_DOUBLE_KIND) +# define LONG_DOUBLE_KIND LDK_UNSUPPORTED +#endif + +namespace +{ + namespace ryu + { +#include "ryu/common.h" +#include "ryu/digit_table.h" +#include "ryu/d2s_intrinsics.h" +#include "ryu/d2s_full_table.h" +#include "ryu/d2fixed_full_table.h" +#include "ryu/f2s_intrinsics.h" +#include "ryu/d2s.c" +#include "ryu/d2fixed.c" +#include "ryu/f2s.c" + +#ifdef __SIZEOF_INT128__ + namespace generic128 + { + // Put the generic Ryu bits in their own namespace to avoid name conflicts. +# include "ryu/generic_128.h" +# include "ryu/ryu_generic_128.h" +# include "ryu/generic_128.c" + } // namespace generic128 + + using generic128::floating_decimal_128; + using generic128::generic_binary_to_decimal; + + int + to_chars(const floating_decimal_128 v, char* const result) + { return generic128::generic_to_chars(v, result); } +#endif + } // namespace ryu + + // A traits class that contains pertinent information about the binary + // format of each of the floating-point types we support. + template<typename T> + struct floating_type_traits + { }; + + template<> + struct floating_type_traits<float> + { + // We (and Ryu) assume float has the IEEE binary32 format. + static_assert(__FLT_MANT_DIG__ == 24); + static constexpr int mantissa_bits = 23; + static constexpr int exponent_bits = 8; + static constexpr bool has_implicit_leading_bit = true; + using mantissa_t = uint32_t; + using shortest_scientific_t = ryu::floating_decimal_32; + + static constexpr uint64_t pow10_adjustment_tab[] + = { 0b0000000000011101011100110101100101101110000000000000000000000000 }; + }; + + template<> + struct floating_type_traits<double> + { + // We (and Ryu) assume double has the IEEE binary64 format. + static_assert(__DBL_MANT_DIG__ == 53); + static constexpr int mantissa_bits = 52; + static constexpr int exponent_bits = 11; + static constexpr bool has_implicit_leading_bit = true; + using mantissa_t = uint64_t; + using shortest_scientific_t = ryu::floating_decimal_64; + + static constexpr uint64_t pow10_adjustment_tab[] + = { 0b0000000000000000000000011000110101110111000001100101110000111100, + 0b0111100011110101011000011110000000110110010101011000001110011111, + 0b0101101100000000011100100100111100110110110100010001010101110000, + 0b0011110010111000101111110101100011101100010001010000000101100111, + 0b0001010000011001011100100001010000010101101000001101000000000000 }; + }; + +#if LONG_DOUBLE_KIND == LDK_BINARY64 + // When long double is equivalent to double, we just forward the long double + // overloads to the double overloads, so we don't need to define a a + // floating_type_traits<long double> specialization in this case. +#elif LONG_DOUBLE_KIND == LDK_FLOAT80 + template<> + struct floating_type_traits<long double> + { + static constexpr int mantissa_bits = 64; + static constexpr int exponent_bits = 15; + static constexpr bool has_implicit_leading_bit = false; + using mantissa_t = uint64_t; + using shortest_scientific_t = ryu::floating_decimal_128; + + static constexpr uint64_t pow10_adjustment_tab[] + = { 0b0000000000000000000000000000110101011111110100010100110000011101, + 0b1001100101001111010011011111101000101111110001011001011101110000, + 0b0000101111111011110010001000001010111101011110111111010100011001, + 0b0011100000011111001101101011111001111100100010000101001111101001, + 0b0100100100000000100111010010101110011000110001101101110011001010, + 0b0111100111100010100000010011000010010110101111110101000011110100, + 0b1010100111100010011110000011011101101100010110000110101010101010, + 0b0000001111001111000000101100111011011000101000110011101100110010, + 0b0111000011100100101101010100001101111110101111001000010011111111, + 0b0010111000100110100100100010101100111010110001101010010111001000, + 0b0000100000010110000011001001000111000001111010100101101000001111, + 0b0010101011101000111100001011000010011101000101010010010000101111, + 0b1011111011101101110010101011010001111000101000101101011001100011, + 0b1010111011011011110111110011001010000010011001110100101101000101, + 0b0011000001110110011010010000011100100011001011001100001101010110, + 0b0100011111011000111111101000011110000010111110101001000000001001, + 0b1110000001110001001101101110011000100000001010000111100010111010, + 0b1110001001010011101000111000001000010100110000010110100011110000, + 0b0000011010110000110001111000011111000011001101001101001001000110, + 0b1010010111001000101001100101010110100100100010010010000101000010, + 0b1011001110000111100010100110000011100011111001110111001100000101, + 0b0110101001001000010110001000010001010101110101100001111100011001, + 0b1111100011110101011110011010101001010010100011000010110001101001, + 0b0100000100001000111101011100010011011111011001000000001100011000, + 0b1110111111000111100101110111110000000011001110011100011011011001, + 0b1100001100100000010001100011011000111011110000110011010101000011, + 0b1111111011100111011101001111111000010000001111010111110010000100, + 0b1110111001111110101111000101000000001010001110011010001000111010, + 0b1000010001011000101111111010110011111101110101101001111000111010, + 0b0100000111101001000111011001101000001010111011101001101111000100, + 0b0000011100110001000111011100111100110001101111111010110111100000, + 0b0000011101011100100110010011110101010100010011110010010111010000, + 0b0011011001100111110101111100001001101110101101001110110011110110, + 0b1011000101000001110100111001100100111100110011110000000001101000, + 0b1011100011110100001001110101010110111001000000001011101001011110, + 0b1111001010010010100000010110101010101011101000101000000000001100, + 0b1000001111100100111001110101100001010011111111000001000011110000, + 0b0001011101001000010000101101111000001110101100110011001100110111, + 0b1110011100000010101011011111001010111101111110100000011100000011, + 0b1001110110011100101010011110100010110001001110110000101011100110, + 0b1001101000100011100111010000011011100001000000110101100100001001, + 0b1010111000101000101101010111000010001100001010100011111100000100, + 0b0111101000100011000101101011111011100010001101110111001111001011, + 0b1110100111010110001110110110000000010110100011110000010001111100, + 0b1100010100011010001011001000111001010101011110100101011001000000, + 0b0000110001111001100110010110111010101101001101000000000010010101, + 0b0001110111101000001111101010110010010000111110111100000111110100, + 0b0111110111001001111000110001101101001010101110110101111110000100, + 0b0000111110111010101111100010111010011100010110011011011001000001, + 0b1010010100100100101110111111111000101100000010111111101101000110, + 0b1000100111111101100011001101000110001000000100010101010100001101, + 0b1100101010101000111100101100001000110001110010100000000010110101, + 0b1010000100111101100100101010010110100010000000110101101110000100, + 0b1011111011110001110000100100000000001010111010001101100000100100, + 0b0111101101100011001110011100000001000101101101111000100111011111, + 0b0100111010010011011001010011110100001100111010010101111111100011, + 0b0010001001011000111000001100110111110111110010100011000110110110, + 0b0101010110000000010000100000110100111011111101000100000111010010, + 0b0110000011011101000001010100110101101110011100110101000000001001, + 0b1101100110100000011000001111000100100100110001100110101010101100, + 0b0010100101010110010010001010101000011111111111001011001010001111, + 0b0111001010001111001100111001010101001000110101000011110000001000, + 0b0110010011001001001111110001010010001011010010001101110110110011, + 0b0110010100111011000100111000001001101011111001110010111110111111, + 0b0101110111001001101100110100101001110010101110011001101110001000, + 0b0100110101010111011010001100010111100011010011111001010100111000, + 0b0111000110110111011110100100010111000110000110110110110001111110, + 0b1000101101010100100100111110100011110110110010011001110011110101, + 0b1001101110101001010100111101101011000101000010110101101111110000, + 0b0100100101001011011001001011000010001101001010010001010110101000, + 0b0010100001001011100110101000010110000111000111000011100101011011, + 0b0110111000011001111101101011111010001000000010101000101010011110, + 0b1000110110100001111011000001111100001001000000010110010100100100, + 0b1001110100011111100111101011010000010101011100101000010010100110, + 0b0001010110101110100010101010001110110110100011101010001001111100, + 0b1010100101101100000010110011100110100010010000100100001110000100, + 0b0001000000010000001010000010100110000001110100111001110111101101, + 0b1100000000000000000000000000000000000000000000000000000000000000 }; + }; +#elif LONG_DOUBLE_KIND == LDK_BINARY128 + template<> + struct floating_type_traits<long double> + { + static constexpr int mantissa_bits = 112; + static constexpr int exponent_bits = 15; + static constexpr bool has_implicit_leading_bit = true; + using mantissa_t = unsigned __int128; + using shortest_scientific_t = ryu::floating_decimal_128; + + static constexpr uint64_t pow10_adjustment_tab[] + = { 0b0000000000000000000000000000000000000000000000000100000010000000, + 0b1011001111110100000100010101101110011100100110000110010110011000, + 0b1010100010001101111111000000001101010010100010010000111011110111, + 0b1011111001110001111000011111000010110111000111110100101010100101, + 0b0110100110011110011011000011000010011001110001001001010011100011, + 0b0000011111110010101111101011101010000110011111100111001110100111, + 0b0100010101010110000010111011110100000010011001001010001110111101, + 0b1101110111000010001101100000110100000111001001101011000101011011, + 0b0100111011101101010000001101011000101100101110010010110000101011, + 0b0100000110111000000110101000010011101000110100010110000011101101, + 0b1011001101001000100001010001100100001111011101010101110001010110, + 0b1000000001000000101001110010110010001111101101010101001100000110, + 0b0101110110100110000110000001001010111110001110010000111111010011, + 0b1010001111100111000100011100100100111100100101000001011001000111, + 0b1010011000011100110101100111001011100101111111100001110100000100, + 0b1100011100100010100000110001001010000000100000001001010111011101, + 0b0101110000100011001111101101000000100110000010010111010001111010, + 0b0100111100011010110111101000100110000111001001101100000001111100, + 0b1100100100111110101011000100000101011010110111000111110100110101, + 0b0110010000010111010100110011000000111010000010111011010110000100, + 0b0101001001010010110111010111000101011100000111100111000001110010, + 0b1101111111001011101010110001000111011010111101001011010110100100, + 0b0001000100110000011111101011001101110010110110010000000011100100, + 0b0001000000000101001001001000000000011000100011001110101001001110, + 0b0010010010001000111010011011100001000110011011011110110100111000, + 0b0000100110101100000111100010100100011100110111011100001111001100, + 0b1011111010001110001100000011110111111111100000001011111111101100, + 0b0000011100001111010101110000100110111100101101110111101001000001, + 0b1100010001110110111100001001001101101000011100000010110101001011, + 0b0100101001101011111001011110101101100011011111011100101010101111, + 0b0001101001111001110000101101101100001011010001011110011101000010, + 0b1111000000101001101111011010110011101110100001011011001011100010, + 0b0101001010111101101100001111100010010110001101001000001101100100, + 0b0101100101011110001100101011111000111001111001001001101101100001, + 0b1111001101010010100100011011000110110010001111000111010001001101, + 0b0001110010011000000001000110110111011000011100001000011001110111, + 0b0100001011011011011011110011101100100101111111101100101000001110, + 0b0101011110111101010111100111101111000101111111111110100011011010, + 0b1110101010001001110100000010110111010111111010111110100110010110, + 0b1010001111100001001100101000110100001100011100110010000011010111, + 0b1111111101101111000100111100000101011000001110011011101010111001, + 0b1111101100001110100101111101011001000100000101110000110010100011, + 0b1001010110110101101101000101010001010000101011011111010011010000, + 0b0111001110110011101001100111000001000100001010110000010000001101, + 0b0101111100111110100111011001111001111011011110010111010011101010, + 0b1110111000000001100100111001100100110001011011001110101111110111, + 0b0001010001001101010111101010011111000011110001101101011001111111, + 0b0101000011100011010010001101100001011101011010100110101100100010, + 0b0001000101011000100101111100110110000101101101111000110001001011, + 0b0101100101001011011000010101000000010100011100101101000010011111, + 0b1000010010001011101001011010100010111011110100110011011000100111, + 0b1000011011100001010111010111010011101100100010010010100100101001, + 0b1001001001010111110101000010111010000000101111010100001010010010, + 0b0011011110110010010101111011000001000000000011011111000011111011, + 0b1011000110100011001110000001000100000001011100010111010010011110, + 0b0111101110110101110111110000011000000100011100011000101101101110, + 0b1001100101111011011100011110101011001111100111101010101010110111, + 0b1100110010010001100011001111010000000100011101001111011101001111, + 0b1000111001111010100101000010000100000001001100101010001011001101, + 0b0011101011110000110010100101010100110010100001000010101011111101, + 0b1100000000000110000010101011000000011101000110011111100010111111, + 0b0010100110000011011100010110111100010110101100110011101110001101, + 0b0010111101010011111000111001111100110111111100100011110001101110, + 0b1001110111001001101001001001011000010100110001000000100011010110, + 0b0011110101100111011011111100001000011001010100111100100101111010, + 0b0010001101000011000010100101110000010101101000100110000100001010, + 0b0010000010100110010101100101110011101111000111111111001001100001, + 0b0100111111011011011011100111111011000010011101101111011111110110, + 0b1111111111010110101011101000100101110100001110001001101011100111, + 0b1011111101000101110000111100100010111010100001010000010010110010, + 0b1111010101001011101011101010000100110110001110111100100110111111, + 0b1011001101000001001101000010101010010110010001100001011100011010, + 0b0101001011011101010001110100010000010001111100100100100001001101, + 0b0010100000111001100011000101100101000001111100111001101000000010, + 0b1011001111010101011001000100100110100100110111110100000110111000, + 0b0101011111010011100011010010111101110010100001111111100010001001, + 0b0010111011101100100000000000001111111010011101100111100001001101, + 0b1101000000000000000000000000000000000000000000000000000000000000 }; + }; +#elif LONG_DOUBLE_KIND == LDK_IBM128 + template<> + struct floating_type_traits<long double> + { + static constexpr int mantissa_bits = 105; + static constexpr int exponent_bits = 11; + static constexpr bool has_implicit_leading_bit = true; + using mantissa_t = unsigned __int128; + using shortest_scientific_t = ryu::floating_decimal_128; + + static constexpr uint64_t pow10_adjustment_tab[] + = { 0b0000000000000000000000000000000000000000000000001000000100000000, + 0b0000000000000000000100000000000000000000001000000000000000000010, + 0b0000100000000000000000001001000000000000000001100100000000000000, + 0b0011000000000000000000000000000001110000010000000000000000000000, + 0b0000100000000000001000000000000000000000000000100000000000000000 }; + }; +#endif + + // An IEEE-style decomposition of a floating-point value of type T. + template<typename T> + struct ieee_t + { + typename floating_type_traits<T>::mantissa_t mantissa; + uint32_t biased_exponent; + bool sign; + }; + + // Decompose the floating-point value into its IEEE components. + template<typename T> + ieee_t<T> + get_ieee_repr(const T value) + { + constexpr int mantissa_bits = floating_type_traits<T>::mantissa_bits; + constexpr int exponent_bits = floating_type_traits<T>::exponent_bits; + constexpr int total_bits = mantissa_bits + exponent_bits + 1; + + constexpr auto get_uint_t = [] { + if constexpr (total_bits <= 32) + return uint32_t{}; + else if constexpr (total_bits <= 64) + return uint64_t{}; +#ifdef __SIZEOF_INT128__ + else if constexpr (total_bits <= 128) + return (unsigned __int128){}; +#endif + }; + using uint_t = decltype(get_uint_t()); + uint_t value_bits = 0; + memcpy(&value_bits, &value, sizeof(value)); + + ieee_t<T> ieee_repr; + ieee_repr.mantissa = value_bits & ((uint_t{1} << mantissa_bits) - 1u); + ieee_repr.biased_exponent + = (value_bits >> mantissa_bits) & ((uint_t{1} << exponent_bits) - 1u); + ieee_repr.sign = (value_bits >> (mantissa_bits + exponent_bits)) & 1; + return ieee_repr; + } + +#if LONG_DOUBLE_KIND == LDK_IBM128 + template<> + ieee_t<long double> + get_ieee_repr(const long double value) + { + // The layout of __ibm128 isn't compatible with the standard IEEE format. + // So we transform it into an IEEE-compatible format, suitable for + // consumption by the generic Ryu API, with an 11-bit exponent and 105-bit + // mantissa (plus an implicit leading bit). We use the exponent and sign + // of the high part, and we merge the mantissa of the high part with the + // mantissa (and the implicit leading bit) of the low part. + using uint_t = unsigned __int128; + uint_t value_bits = 0; + memcpy(&value_bits, &value, sizeof(value_bits)); + + const uint64_t value_hi = value_bits; + const uint64_t value_lo = value_bits >> 64; + + uint64_t mantissa_hi = value_hi & ((1ull << 52) - 1); + unsigned exponent_hi = (value_hi >> 52) & ((1ull << 11) - 1); + const int sign_hi = (value_hi >> 63) & 1; + + uint64_t mantissa_lo = value_lo & ((1ull << 52) - 1); + const unsigned exponent_lo = (value_lo >> 52) & ((1ull << 11) - 1); + const int sign_lo = (value_lo >> 63) & 1; + + { + // The following code for adjusting the low-part mantissa to combine + // it with the high-part mantissa is taken from the glibc source file + // sysdeps/ieee754/ldbl-128ibm/printf_fphex.c. + mantissa_lo <<= 7; + if (exponent_lo != 0) + mantissa_lo |= (1ull << (52 + 7)); + else + mantissa_lo <<= 1; + + const int ediff = exponent_hi - exponent_lo - 53; + if (ediff > 63) + mantissa_lo = 0; + else if (ediff > 0) + mantissa_lo >>= ediff; + else if (ediff < 0) + mantissa_lo <<= -ediff; + + if (sign_lo != sign_hi && mantissa_lo != 0) + { + mantissa_lo = (1ull << 60) - mantissa_lo; + if (mantissa_hi == 0) + { + mantissa_hi = 0xffffffffffffeLL | (mantissa_lo >> 59); + mantissa_lo = 0xfffffffffffffffLL & (mantissa_lo << 1); + exponent_hi--; + } + else + mantissa_hi--; + } + } + + ieee_t<long double> ieee_repr; + ieee_repr.mantissa = ((uint_t{mantissa_hi} << 64) + | (uint_t{mantissa_lo} << 4)) >> 11; + ieee_repr.biased_exponent = exponent_hi; + ieee_repr.sign = sign_hi; + return ieee_repr; + } +#endif + + // Invoke Ryu to obtain the shortest scientific form for the given + // floating-point number. + template<typename T> + typename floating_type_traits<T>::shortest_scientific_t + floating_to_shortest_scientific(const T value) + { + if constexpr (std::is_same_v<T, float>) + return ryu::floating_to_fd32(value); + else if constexpr (std::is_same_v<T, double>) + return ryu::floating_to_fd64(value); +#ifdef __SIZEOF_INT128__ + else if constexpr (std::is_same_v<T, long double>) + { + constexpr int mantissa_bits + = floating_type_traits<T>::mantissa_bits; + constexpr int exponent_bits + = floating_type_traits<T>::exponent_bits; + constexpr bool has_implicit_leading_bit + = floating_type_traits<T>::has_implicit_leading_bit; + + const auto [mantissa, exponent, sign] = get_ieee_repr(value); + return ryu::generic_binary_to_decimal(mantissa, exponent, sign, + mantissa_bits, exponent_bits, + !has_implicit_leading_bit); + } +#endif + } + + // This subroutine returns true if the shortest scientific form fd is a + // positive power of 10, and the floating-point number that has this shortest + // scientific form is smaller than this power of 10. + // + // For instance, the exactly-representable 64-bit number + // 99999999999999991611392.0 has the shortest scientific form 1e23, so its + // exact value is smaller than its shortest scientific form. + // + // For these powers of 10 the length of the fixed form is one digit less + // than what the scientific exponent suggests. + // + // This subroutine inspects a lookup table to detect when fd is such a + // "rounded up" power of 10. + template<typename T> + bool + is_rounded_up_pow10_p(const typename + floating_type_traits<T>::shortest_scientific_t fd) + { + if (fd.exponent < 0 || fd.mantissa != 1) [[likely]] + return false; + + constexpr auto& pow10_adjustment_tab + = floating_type_traits<T>::pow10_adjustment_tab; + __glibcxx_assert(fd.exponent/64 < (int)std::size(pow10_adjustment_tab)); + return (pow10_adjustment_tab[fd.exponent/64] + & (1ull << (63 - fd.exponent%64))); + } + + int + get_mantissa_length(const ryu::floating_decimal_32 fd) + { return ryu::decimalLength9(fd.mantissa); } + + int + get_mantissa_length(const ryu::floating_decimal_64 fd) + { return ryu::decimalLength17(fd.mantissa); } + +#ifdef __SIZEOF_INT128__ + int + get_mantissa_length(const ryu::floating_decimal_128 fd) + { return ryu::generic128::decimalLength(fd.mantissa); } +#endif +} // anon namespace + +namespace std _GLIBCXX_VISIBILITY(default) +{ +_GLIBCXX_BEGIN_NAMESPACE_VERSION + +// This subroutine of __floating_to_chars_* handles writing nan, inf and 0 in +// all formatting modes. +template<typename T> + static optional<to_chars_result> + __handle_special_value(char* first, char* const last, const T value, + const chars_format fmt, const int precision) + { + __glibcxx_assert(precision >= 0); + + string_view str; + switch (__builtin_fpclassify(FP_NAN, FP_INFINITE, FP_NORMAL, FP_SUBNORMAL, + FP_ZERO, value)) + { + case FP_INFINITE: + str = "-inf"; + break; + + case FP_NAN: + str = "-nan"; + break; + + case FP_ZERO: + break; + + default: + case FP_SUBNORMAL: + case FP_NORMAL: [[likely]] + return nullopt; + } + + if (!str.empty()) + { + // We're formatting +-inf or +-nan. + if (!__builtin_signbit(value)) + str.remove_prefix(strlen("-")); + + if (last - first < (int)str.length()) + return {{last, errc::value_too_large}}; + + memcpy(first, &str[0], str.length()); + first += str.length(); + return {{first, errc{}}}; + } + + // We're formatting 0. + __glibcxx_assert(value == 0); + const auto orig_first = first; + const bool sign = __builtin_signbit(value); + int expected_output_length; + switch (fmt) + { + case chars_format::fixed: + case chars_format::scientific: + case chars_format::hex: + expected_output_length = sign + 1; + if (precision) + expected_output_length += strlen(".") + precision; + if (fmt == chars_format::scientific) + expected_output_length += strlen("e+00"); + else if (fmt == chars_format::hex) + expected_output_length += strlen("p+0"); + if (last - first < expected_output_length) + return {{last, errc::value_too_large}}; + + if (sign) + *first++ = '-'; + *first++ = '0'; + if (precision) + { + *first++ = '.'; + memset(first, '0', precision); + first += precision; + } + if (fmt == chars_format::scientific) + { + memcpy(first, "e+00", 4); + first += 4; + } + else if (fmt == chars_format::hex) + { + memcpy(first, "p+0", 3); + first += 3; + } + break; + + case chars_format::general: + default: // case chars_format{}: + expected_output_length = sign + 1; + if (last - first < expected_output_length) + return {{last, errc::value_too_large}}; + + if (sign) + *first++ = '-'; + *first++ = '0'; + break; + } + __glibcxx_assert(first - orig_first == expected_output_length); + return {{first, errc{}}}; + } + +// This subroutine of the floating-point to_chars overloads performs +// hexadecimal formatting. +template<typename T> + static to_chars_result + __floating_to_chars_hex(char* first, char* const last, const T value, + const optional<int> precision) + { + if (precision.has_value() && precision.value() < 0) [[unlikely]] + // A negative precision argument is treated as if it were omitted. + return __floating_to_chars_hex(first, last, value, nullopt); + + __glibcxx_requires_valid_range(first, last); + + constexpr int mantissa_bits = floating_type_traits<T>::mantissa_bits; + constexpr bool has_implicit_leading_bit + = floating_type_traits<T>::has_implicit_leading_bit; + constexpr int exponent_bits = floating_type_traits<T>::exponent_bits; + constexpr int exponent_bias = (1u << (exponent_bits - 1)) - 1; + using mantissa_t = typename floating_type_traits<T>::mantissa_t; + constexpr int mantissa_t_width = sizeof(mantissa_t) * __CHAR_BIT__; + + if (auto result = __handle_special_value(first, last, value, + chars_format::hex, + precision.value_or(0))) + return *result; + + // Extract the sign, mantissa and exponent from the value. + const auto [ieee_mantissa, biased_exponent, sign] = get_ieee_repr(value); + const bool is_normal_number = (biased_exponent != 0); + + // Calculate the unbiased exponent. + const int32_t unbiased_exponent = (is_normal_number + ? biased_exponent - exponent_bias + : 1 - exponent_bias); + + // Shift the mantissa so that its bitwidth is a multiple of 4. + constexpr unsigned rounded_mantissa_bits = (mantissa_bits + 3) / 4 * 4; + static_assert(mantissa_t_width >= rounded_mantissa_bits); + mantissa_t effective_mantissa + = ieee_mantissa << (rounded_mantissa_bits - mantissa_bits); + if (is_normal_number) + { + if constexpr (has_implicit_leading_bit) + // Restore the mantissa's implicit leading bit. + effective_mantissa |= mantissa_t{1} << rounded_mantissa_bits; + else + // The explicit mantissa bit should already be set. + __glibcxx_assert(effective_mantissa & (mantissa_t{1} << (mantissa_bits + - 1u))); + } + + // Compute the shortest precision needed to print this value exactly, + // disregarding trailing zeros. + constexpr int full_hex_precision = (has_implicit_leading_bit + ? (mantissa_bits + 3) / 4 + // With an explicit leading bit, we + // use the four leading nibbles as the + // hexit before the decimal point. + : (mantissa_bits - 4 + 3) / 4); + const int trailing_zeros = __countr_zero(effective_mantissa) / 4; + const int shortest_full_precision = full_hex_precision - trailing_zeros; + __glibcxx_assert(shortest_full_precision >= 0); + + int written_exponent = unbiased_exponent; + const int effective_precision = precision.value_or(shortest_full_precision); + if (effective_precision < shortest_full_precision) + { + // When limiting the precision, we need to determine how to round the + // least significant printed hexit. The following branchless + // bit-level-parallel technique computes whether to round up the + // mantissa bit at index N (according to round-to-nearest rules) when + // dropping N bits of precision, for each index N in the bit vector. + // This technique is borrowed from the MSVC implementation. + using bitvec = mantissa_t; + const bitvec round_bit = effective_mantissa << 1; + const bitvec has_tail_bits = round_bit - 1; + const bitvec lsb_bit = effective_mantissa; + const bitvec should_round = round_bit & (has_tail_bits | lsb_bit); + + const int dropped_bits = 4*(full_hex_precision - effective_precision); + // Mask out the dropped nibbles. + effective_mantissa >>= dropped_bits; + effective_mantissa <<= dropped_bits; + if (should_round & (mantissa_t{1} << dropped_bits)) + { + // Round up the least significant nibble. + effective_mantissa += mantissa_t{1} << dropped_bits; + // Check and adjust for overflow of the leading nibble. When the + // type has an implicit leading bit, then the leading nibble + // before rounding is either 0 or 1, so it can't overflow. + if constexpr (!has_implicit_leading_bit) + { + // The only supported floating-point type with explicit + // leading mantissa bit is LDK_FLOAT80, i.e. x86 80-bit + // extended precision, and so we hardcode the below overflow + // check+adjustment for this type. + static_assert(mantissa_t_width == 64 + && rounded_mantissa_bits == 64); + if (effective_mantissa == 0) + { + // We rounded up the least significant nibble and the + // mantissa overflowed, e.g f.fcp+10 with precision=1 + // became 10.0p+10. Absorb this extra hexit into the + // exponent to obtain 1.0p+14. + effective_mantissa + = mantissa_t{1} << (rounded_mantissa_bits - 4); + written_exponent += 4; + } + } + } + } + + // Compute the leading hexit and mask it out from the mantissa. + char leading_hexit; + if constexpr (has_implicit_leading_bit) + { + const unsigned nibble = effective_mantissa >> rounded_mantissa_bits; + __glibcxx_assert(nibble <= 2); + leading_hexit = '0' + nibble; + effective_mantissa &= ~(mantissa_t{0b11} << rounded_mantissa_bits); + } + else + { + const unsigned nibble = effective_mantissa >> (rounded_mantissa_bits-4); + __glibcxx_assert(nibble < 16); + leading_hexit = "0123456789abcdef"[nibble]; + effective_mantissa &= ~(mantissa_t{0b1111} << (rounded_mantissa_bits-4)); + written_exponent -= 3; + } + + // Now before we start writing the string, determine the total length of + // the output string and perform a single bounds check. + int expected_output_length = sign + 1; + if (effective_precision != 0) + expected_output_length += strlen(".") + effective_precision; + const int abs_written_exponent = abs(written_exponent); + expected_output_length += (abs_written_exponent >= 10000 ? strlen("p+ddddd") + : abs_written_exponent >= 1000 ? strlen("p+dddd") + : abs_written_exponent >= 100 ? strlen("p+ddd") + : abs_written_exponent >= 10 ? strlen("p+dd") + : strlen("p+d")); + if (last - first < expected_output_length) + return {last, errc::value_too_large}; + + const auto saved_first = first; + // Write the negative sign and the leading hexit. + if (sign) + *first++ = '-'; + *first++ = leading_hexit; + + if (effective_precision > 0) + { + *first++ = '.'; + int written_hexits = 0; + // Extract and mask out the leading nibble after the decimal point, + // write its corresponding hexit, and repeat until the mantissa is + // empty. + int nibble_offset = rounded_mantissa_bits; + if constexpr (!has_implicit_leading_bit) + // We already printed the entire leading hexit. + nibble_offset -= 4; + while (effective_mantissa != 0) + { + nibble_offset -= 4; + const unsigned nibble = effective_mantissa >> nibble_offset; + __glibcxx_assert(nibble < 16); + *first++ = "0123456789abcdef"[nibble]; + ++written_hexits; + effective_mantissa &= ~(mantissa_t{0b1111} << nibble_offset); + } + __glibcxx_assert(nibble_offset >= 0); + __glibcxx_assert(written_hexits <= effective_precision); + // Since the mantissa is now empty, every hexit hereafter must be '0'. + if (int remaining_hexits = effective_precision - written_hexits) + { + memset(first, '0', remaining_hexits); + first += remaining_hexits; + } + } + + // Finally, write the exponent. + *first++ = 'p'; + if (written_exponent >= 0) + *first++ = '+'; + const to_chars_result result = to_chars(first, last, written_exponent); + __glibcxx_assert(result.ec == errc{} + && result.ptr == saved_first + expected_output_length); + return result; + } + +template<typename T> + static to_chars_result + __floating_to_chars_shortest(char* first, char* const last, const T value, + chars_format fmt) + { + if (fmt == chars_format::hex) + return __floating_to_chars_hex(first, last, value, nullopt); + + __glibcxx_assert(fmt == chars_format::fixed + || fmt == chars_format::scientific </cut>

3 years, 9 months

[TCWG CI] Regression caused by llvm: Recommit "Revert "[CVP] processSwitch: Remove default case when switch cover all possible values.""

by ci_notify＠linaro.org

After llvm commit 8ba2adcf9e54b34ba8efa73ac0d81a1192e4f614 Author: Jun Ma <JunMa(a)linux.alibaba.com> Recommit "Revert "[CVP] processSwitch: Remove default case when switch cover all possible values."" the following benchmarks grew in size by more than 1%: - 401.bzip2 grew in size by 4% from 36214 to 37510 bytes - [.] BZ2_decompress grew in size by 19%,401.bzip2,[.] BZ2_decompress grew in size by 19% from 7260 to 8660 bytes Results regressed to # reset_artifacts: -10 # build_abe binutils: -9 # build_abe stage1 -- --set gcc_override_configure=--with-mode=thumb --set gcc_override_configure=--disable-libsanitizer: -8 # build_abe linux: -7 # build_abe glibc: -6 # build_abe stage2 -- --set gcc_override_configure=--with-mode=thumb --set gcc_override_configure=--disable-libsanitizer: -5 # build_llvm true: -3 # true: 0 # benchmark -- -Oz_mthumb artifacts/build-8ba2adcf9e54b34ba8efa73ac0d81a1192e4f614/results_id: 1 from # reset_artifacts: -10 # build_abe binutils: -9 # build_abe stage1 -- --set gcc_override_configure=--with-mode=thumb --set gcc_override_configure=--disable-libsanitizer: -8 # build_abe linux: -7 # build_abe glibc: -6 # build_abe stage2 -- --set gcc_override_configure=--with-mode=thumb --set gcc_override_configure=--disable-libsanitizer: -5 # build_llvm true: -3 # true: 0 # benchmark -- -Oz_mthumb artifacts/build-baseline/results_id: 1 THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. This commit has regressed these CI configurations: - tcwg_bmk_llvm_apm/llvm-master-arm-spec2k6-Oz First_bad build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-master-… Last_good build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-master-… Baseline build: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-master-… Even more details: https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-master-… Reproduce builds: <cut> mkdir investigate-llvm-8ba2adcf9e54b34ba8efa73ac0d81a1192e4f614 cd investigate-llvm-8ba2adcf9e54b34ba8efa73ac0d81a1192e4f614 # Fetch scripts git clone https://git.linaro.org/toolchain/jenkins-scripts # Fetch manifests and test.sh script mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-master-… --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-master-… --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_apm-llvm-master-… --fail chmod +x artifacts/test.sh # Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh # Save baseline build state (which is then restored in artifacts/test.sh) mkdir -p ./bisect rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ --exclude /llvm/ ./ ./bisect/baseline/ cd llvm # Reproduce first_bad build git checkout --detach 8ba2adcf9e54b34ba8efa73ac0d81a1192e4f614 ../artifacts/test.sh # Reproduce last_good build git checkout --detach d1280f6967db1ca8fa4e0c39414003e717b40feb ../artifacts/test.sh cd .. </cut> Full commit (up to 1000 lines): <cut> commit 8ba2adcf9e54b34ba8efa73ac0d81a1192e4f614 Author: Jun Ma <JunMa(a)linux.alibaba.com> Date: Fri Aug 20 17:27:00 2021 +0800 Recommit "Revert "[CVP] processSwitch: Remove default case when switch cover all possible values."" Differential Revision: https://reviews.llvm.org/D106056 --- llvm/include/llvm/Transforms/Utils/Local.h | 5 ++++ .../Scalar/CorrelatedValuePropagation.cpp | 27 +++++++++++++++++++++- llvm/lib/Transforms/Utils/Local.cpp | 20 ++++++++++++++++ llvm/lib/Transforms/Utils/SimplifyCFG.cpp | 20 ---------------- .../Transforms/CorrelatedValuePropagation/basic.ll | 11 +++++---- 5 files changed, 57 insertions(+), 26 deletions(-) diff --git a/llvm/include/llvm/Transforms/Utils/Local.h b/llvm/include/llvm/Transforms/Utils/Local.h index 97686d7d5f2f..f003615eca78 100644 --- a/llvm/include/llvm/Transforms/Utils/Local.h +++ b/llvm/include/llvm/Transforms/Utils/Local.h @@ -55,6 +55,7 @@ class MDNode; class MemorySSAUpdater; class PHINode; class StoreInst; +class SwitchInst; class TargetLibraryInfo; class TargetTransformInfo; @@ -236,6 +237,10 @@ CallInst *createCallMatchingInvoke(InvokeInst *II); /// This function converts the specified invoek into a normall call. void changeToCall(InvokeInst *II, DomTreeUpdater *DTU = nullptr); +/// This function removes the default destination from the specified switch. +void createUnreachableSwitchDefault(SwitchInst *Switch, + DomTreeUpdater *DTU = nullptr); + ///===---------------------------------------------------------------------===// /// Dbg Intrinsic utilities /// diff --git a/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp b/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp index 36cbd42a5fdd..cd38ce96e287 100644 --- a/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp +++ b/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp @@ -341,7 +341,13 @@ static bool processSwitch(SwitchInst *I, LazyValueInfo *LVI, // ConstantFoldTerminator() as the underlying SwitchInst can be changed. SwitchInstProfUpdateWrapper SI(*I); - for (auto CI = SI->case_begin(), CE = SI->case_end(); CI != CE;) { + APInt Low = + APInt::getSignedMaxValue(Cond->getType()->getScalarSizeInBits()); + APInt High = + APInt::getSignedMinValue(Cond->getType()->getScalarSizeInBits()); + + SwitchInst::CaseIt CI = SI->case_begin(); + for (auto CE = SI->case_end(); CI != CE;) { ConstantInt *Case = CI->getCaseValue(); LazyValueInfo::Tristate State = LVI->getPredicateAt(CmpInst::ICMP_EQ, Cond, Case, I, @@ -374,9 +380,28 @@ static bool processSwitch(SwitchInst *I, LazyValueInfo *LVI, break; } + // Get Lower/Upper bound from switch cases. + Low = APIntOps::smin(Case->getValue(), Low); + High = APIntOps::smax(Case->getValue(), High); + // Increment the case iterator since we didn't delete it. ++CI; } + + // Try to simplify default case as unreachable + if (CI == SI->case_end() && SI->getNumCases() != 0 && + !isa<UnreachableInst>(SI->getDefaultDest()->getFirstNonPHIOrDbg())) { + const ConstantRange SIRange = + LVI->getConstantRange(SI->getCondition(), SI); + + // If the numbered switch cases cover the entire range of the condition, + // then the default case is not reachable. + if (SIRange.getSignedMin() == Low && SIRange.getSignedMax() == High && + SI->getNumCases() == High - Low + 1) { + createUnreachableSwitchDefault(SI, &DTU); + Changed = true; + } + } } if (Changed) diff --git a/llvm/lib/Transforms/Utils/Local.cpp b/llvm/lib/Transforms/Utils/Local.cpp index 3d6ffded9b19..6d7eca8e3678 100644 --- a/llvm/lib/Transforms/Utils/Local.cpp +++ b/llvm/lib/Transforms/Utils/Local.cpp @@ -2182,6 +2182,26 @@ void llvm::changeToCall(InvokeInst *II, DomTreeUpdater *DTU) { DTU->applyUpdates({{DominatorTree::Delete, BB, UnwindDestBB}}); } +void llvm::createUnreachableSwitchDefault(SwitchInst *Switch, + DomTreeUpdater *DTU) { + LLVM_DEBUG(dbgs() << "SimplifyCFG: switch default is dead.\n"); + auto *BB = Switch->getParent(); + auto *OrigDefaultBlock = Switch->getDefaultDest(); + OrigDefaultBlock->removePredecessor(BB); + BasicBlock *NewDefaultBlock = BasicBlock::Create( + BB->getContext(), BB->getName() + ".unreachabledefault", BB->getParent(), + OrigDefaultBlock); + new UnreachableInst(Switch->getContext(), NewDefaultBlock); + Switch->setDefaultDest(&*NewDefaultBlock); + if (DTU) { + SmallVector<DominatorTree::UpdateType, 2> Updates; + Updates.push_back({DominatorTree::Insert, BB, &*NewDefaultBlock}); + if (!is_contained(successors(BB), OrigDefaultBlock)) + Updates.push_back({DominatorTree::Delete, BB, &*OrigDefaultBlock}); + DTU->applyUpdates(Updates); + } +} + BasicBlock *llvm::changeToInvokeAndSplitBasicBlock(CallInst *CI, BasicBlock *UnwindEdge, DomTreeUpdater *DTU) { diff --git a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp index 737b4f97a97a..70297e471a7a 100644 --- a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp +++ b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp @@ -4743,26 +4743,6 @@ static bool CasesAreContiguous(SmallVectorImpl<ConstantInt *> &Cases) { return true; } -static void createUnreachableSwitchDefault(SwitchInst *Switch, - DomTreeUpdater *DTU) { - LLVM_DEBUG(dbgs() << "SimplifyCFG: switch default is dead.\n"); - auto *BB = Switch->getParent(); - auto *OrigDefaultBlock = Switch->getDefaultDest(); - OrigDefaultBlock->removePredecessor(BB); - BasicBlock *NewDefaultBlock = BasicBlock::Create( - BB->getContext(), BB->getName() + ".unreachabledefault", BB->getParent(), - OrigDefaultBlock); - new UnreachableInst(Switch->getContext(), NewDefaultBlock); - Switch->setDefaultDest(&*NewDefaultBlock); - if (DTU) { - SmallVector<DominatorTree::UpdateType, 2> Updates; - Updates.push_back({DominatorTree::Insert, BB, &*NewDefaultBlock}); - if (!is_contained(successors(BB), OrigDefaultBlock)) - Updates.push_back({DominatorTree::Delete, BB, &*OrigDefaultBlock}); - DTU->applyUpdates(Updates); - } -} - /// Turn a switch with two reachable destinations into an integer range /// comparison and branch. bool SimplifyCFGOpt::TurnSwitchRangeIntoICmp(SwitchInst *SI, diff --git a/llvm/test/Transforms/CorrelatedValuePropagation/basic.ll b/llvm/test/Transforms/CorrelatedValuePropagation/basic.ll index 5abbcbc90e01..a620c8468d4d 100644 --- a/llvm/test/Transforms/CorrelatedValuePropagation/basic.ll +++ b/llvm/test/Transforms/CorrelatedValuePropagation/basic.ll @@ -382,7 +382,7 @@ define i32 @switch_range(i32 %cond) { ; CHECK-NEXT: entry: ; CHECK-NEXT: [[S:%.*]] = urem i32 [[COND:%.*]], 3 ; CHECK-NEXT: [[S1:%.*]] = add nuw nsw i32 [[S]], 1 -; CHECK-NEXT: switch i32 [[S1]], label [[UNREACHABLE:%.*]] [ +; CHECK-NEXT: switch i32 [[S1]], label [[ENTRY_UNREACHABLEDEFAULT:%.*]] [ ; CHECK-NEXT: i32 1, label [[EXIT1:%.*]] ; CHECK-NEXT: i32 2, label [[EXIT2:%.*]] ; CHECK-NEXT: i32 3, label [[EXIT1]] @@ -391,6 +391,8 @@ define i32 @switch_range(i32 %cond) { ; CHECK-NEXT: ret i32 1 ; CHECK: exit2: ; CHECK-NEXT: ret i32 2 +; CHECK: entry.unreachabledefault: +; CHECK-NEXT: unreachable ; CHECK: unreachable: ; CHECK-NEXT: ret i32 0 ; @@ -453,10 +455,9 @@ define i8 @switch_defaultdest_multipleuse(i8 %t0) { ; CHECK-NEXT: entry: ; CHECK-NEXT: [[O:%.*]] = or i8 [[T0:%.*]], 1 ; CHECK-NEXT: [[R:%.*]] = srem i8 1, [[O]] -; CHECK-NEXT: switch i8 [[R]], label [[EXIT:%.*]] [ -; CHECK-NEXT: i8 0, label [[EXIT]] -; CHECK-NEXT: i8 1, label [[EXIT]] -; CHECK-NEXT: ] +; CHECK-NEXT: br label [[EXIT:%.*]] +; CHECK: entry.unreachabledefault: +; CHECK-NEXT: unreachable ; CHECK: exit: ; CHECK-NEXT: ret i8 0 ; </cut>

3 years, 9 months

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

linaro-toolchain