Commit Graph
1515 Commits
Author SHA1 Message Date
Alexey Bataev ac254fc055 [SLP]Improve tryToGatherExtractElements by using per-register analysis.
Currently tryToGatherExtractElements function analyzes the whole vector,
regrdless number of actual registers, used in this vector. It may
prevent some optimizations, because per-register analysis may allow to
simplify the final code by reusing more already emitted vectors and
better shuffles.

Differential Revision: https://reviews.llvm.org/D148855
2023-11-06 07:29:27 -08:00
Hans Wennborg 046c57e705 Revert "[SLP]Improve tryToGatherExtractElements by using per-register analysis."
This causes asserts:

  llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10082:
  Value *llvm::slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::adjustExtracts(
    const TreeEntry *, MutableArrayRef<int>, unsigned int, bool &):
  Assertion `Part == 0 && "Expected firs part."' failed.

See comment on the code review.

> Currently tryToGatherExtractElements function analyzes the whole vector,
> regrdless number of actual registers, used in this vector. It may
> prevent some optimizations, because per-register analysis may allow to
> simplify the final code by reusing more already emitted vectors and
> better shuffles.
>
> Differential Revision: https://reviews.llvm.org/D148855

This reverts commit 9dfdbd7887.
2023-11-06 13:56:42 +01:00
Alexey Bataev 9dfdbd7887 [SLP]Improve tryToGatherExtractElements by using per-register analysis.
Currently tryToGatherExtractElements function analyzes the whole vector,
regrdless number of actual registers, used in this vector. It may
prevent some optimizations, because per-register analysis may allow to
simplify the final code by reusing more already emitted vectors and
better shuffles.

Differential Revision: https://reviews.llvm.org/D148855
2023-11-03 10:43:58 -07:00
Martin Storsjö 66152f4eed Revert "[SLP]Improve tryToGatherExtractElements by using per-register analysis."
This reverts commit 3e6d7c6d98.

That commit caused miscompilation of ffmpeg's libavcodec/vp9dsp_8bpp.o
on aarch64; the file still compiles correctly, but no longer produces
the right result - see https://reviews.llvm.org/D148855#4655968
for details.
2023-11-03 00:08:17 +02:00
Alexey Bataev 3026c13612 [SLP][NFC]Remove commented out code, NFC. 2023-11-02 11:06:56 -07:00
Alexey Bataev 495ed8d8c8 [SLP]Fix PR70507: freeze poisonous insts to avoid poison propagation.
If the reduction instruction is not bool logical op, but reduced within bool logical op reduction list, need to freeze to avoid poison propagation.
2023-11-02 10:37:38 -07:00
Alexey Bataev 3e6d7c6d98 [SLP]Improve tryToGatherExtractElements by using per-register analysis.
Currently tryToGatherExtractElements function analyzes the whole vector,
regrdless number of actual registers, used in this vector. It may
prevent some optimizations, because per-register analysis may allow to
simplify the final code by reusing more already emitted vectors and
better shuffles.

Differential Revision: https://reviews.llvm.org/D148855
2023-11-01 10:42:35 -07:00
Alexey Bataev 6e8d957a22 Revert "[SLP]Improve tryToGatherExtractElements by using per-register analysis."
This reverts commit 0a34aaedd8 to fix
fails reported in https://lab.llvm.org/buildbot/#/builders/265/builds/40
2023-11-01 08:52:31 -07:00
Alexey Bataev c28b7eb496 [SLP]Fix handling of -slp-vectorize-hor-store for values with many uses. 2023-11-01 08:41:54 -07:00
Alexey Bataev 0a34aaedd8 [SLP]Improve tryToGatherExtractElements by using per-register analysis.
Currently tryToGatherExtractElements function analyzes the whole vector,
regrdless number of actual registers, used in this vector. It may
prevent some optimizations, because per-register analysis may allow to
simplify the final code by reusing more already emitted vectors and
better shuffles.

Differential Revision: https://reviews.llvm.org/D148855
2023-11-01 07:44:49 -07:00
Nikita Popov 79bd1d8a95 [SLPVectorizer] Avoid use of ConstantExpr::getIntegerCast() (NFC)
We are working on a ConstantInt here, so folding will always
succeed. This just avoids use of the ConstantExpr API.
2023-11-01 12:25:55 +01:00
Alexey Bataev 4c997e1536 [SLP]Fix PR70507: emit freeeze whenever required for bool logical ops in
the middle of reduction ops.

Need to emit freeze instruction not only in the case, where the root is
bool logical op, but also if we reduce several scalars, but unable to
say precisely, if the root is bool logical op.
2023-10-31 12:23:12 -07:00
Alexey Bataev 9da19e4340 [SLP]Fix PR70507: correctly handle bool logical ops in reductions.
If the very first reduction operation is not bool logical op, but some
others are, still need to emit the boo logic op for all the extra
reduction operations to avoid incorrect poison propagation.
2023-10-30 14:09:08 -07:00
Alexey Bataev af15c46777 [SLP]Do not crash if number of vector registers does not feet the vector
type.

Need to check, if the number of vector registers, returned by TTI, is
not greater than total number of mask element and not zero, before
trying to perform any operations. TTI still may return non-valid number
of registers.
2023-10-30 07:30:52 -07:00
Alexey Bataev 196d154ab7 [SLP]Improve isGatherShuffledEntry by trying per-register shuffle.
Currently when building gather/buildvector node, we try to build nodes
shuffles without taking into account separate vector registers. We can
improve final codegen and the whole vectorization process by including
this info into the analysis and the vector code emission, allows to emit
better vectorized code.

Differential Revision: https://reviews.llvm.org/D149742
2023-10-26 08:51:37 -07:00
Alexey Bataev c65ec9d919 Revert "[SLP]Improve isGatherShuffledEntry by trying per-register shuffle."
This reverts commit 560bad013e to fix
a bug reported in https://lab.llvm.org/buildbot/#/builders/5/builds/37763.
2023-10-26 08:36:50 -07:00
Alexey Bataev 560bad013e [SLP]Improve isGatherShuffledEntry by trying per-register shuffle.
Currently when building gather/buildvector node, we try to build nodes
shuffles without taking into account separate vector registers. We can
improve final codegen and the whole vectorization process by including
this info into the analysis and the vector code emission, allows to emit
better vectorized code.

Differential Revision: https://reviews.llvm.org/D149742
2023-10-26 05:57:03 -07:00
Kazu Hirata f9306f6de3 [ADT] Rename llvm::erase_value to llvm::erase (NFC) (#70156)
C++20 comes with std::erase to erase a value from std::vector.  This
patch renames llvm::erase_value to llvm::erase for consistency with
C++20.

We could make llvm::erase more similar to std::erase by having it
return the number of elements removed, but I'm not doing that for now
because nobody seems to care about that in our code base.

Since there are only 50 occurrences of erase_value in our code base,
this patch replaces all of them with llvm::erase and deprecates
llvm::erase_value.
2023-10-24 23:03:13 -07:00
Valery Dmitriev 3324776d9c [SLP] Improve gather tree nodes matching when users are PHIs. (#70111)
This is re-commit of #69392 and also fixes issue #69670 which was
uncovered with the prior commit.
For delayed gather emission it may be incorrect to use stab instruction
as insertion point if it is a PHI operand. For that case insertion point
is adjusted to be at the end of block, ensuring that prior dependecy
vector code is emitted earlier.
2023-10-24 16:39:36 -07:00
Alexey Bataev a3c68754b0 [SLP][NFC]Remove unused variables, NFC. 2023-10-24 14:36:33 -07:00
Alexey Bataev d79051f894 [SLP]Fix PR70004: Do not change insert point for reduction gather nodes.
No need to change the insert point for reduction gather node, we can use
the ReductionRoot as insert point instead to avoid possible crashes.
2023-10-24 09:19:59 -07:00
Alexey Bataev 8d307f59ee [SLP]Fix PR69246: do not treat resizing maskas identity.
If the mask is resizing and the mask size is greater than than the
length of the vector, being reused from extractelement instructions, the
mask for undefs cannot be treated as identity, must be treated as
a broadcast.
2023-10-24 08:14:13 -07:00
Alexey Bataev 254558ac53 [SLP]Fix PR69976: Check for multi-node uses during node building.
Need to check if there is already a node created for the multi-node
instruction before ending up with creating a new node for such
instructions.
2023-10-24 07:01:46 -07:00
Douglas Yung 734b016b66 Revert "[SLP] Improve gather tree nodes matching when users are PHIs. (#69392)"
This reverts commit c80b503496.

This change causes a fatal error in the backend and is filed as issue #69670.
2023-10-20 10:59:07 -07:00
Alexey Bataev 4a06332e45 [SLP][NFC]Use MutableArrayRef instead of SmallVectorImpl&, rename
function, NFC.
2023-10-18 13:09:20 -07:00
Alexey Bataev 3ef271c3d6 [SLP][NFC]Use MutableArrayRef instead of SmallVectorImpl& in param, NFC. 2023-10-18 09:47:07 -07:00
Valery Dmitriev c80b503496 [SLP] Improve gather tree nodes matching when users are PHIs. (#69392) 2023-10-18 09:05:11 -07:00
Valery Dmitriev 9aa571f080 [SLP][NFC] Try to cleanup and better document some isGatherShuffledEntry code. (#69384)
Outline some often used common code to dedicated variables in order
to make code compact. Rename variables to more accurately reflect
their purpose. Apply const qualifier where appropriate.
Fix and add bit more explanation comment for the existing code.
2023-10-17 14:59:36 -07:00
Alexey Bataev 66775f8ccd [SLP]Fix PR69196: Instruction does not dominate all uses
During emission of the postponed gathers, need to insert them before
user instruction to avoid use before definition crash.
2023-10-17 10:43:59 -07:00
Alexey Bataev 119b0f3895 Revert "[SLP]Fix PR69196: Instruction does not dominate all uses"
This reverts commit 8e2b2c4181 to fix
a crash reported in https://lab.llvm.org/buildbot/#/builders/230/builds/19993.
2023-10-16 13:29:17 -07:00
Alexey Bataev 8e2b2c4181 [SLP]Fix PR69196: Instruction does not dominate all uses
During emission of the postponed gathers, need to insert them before
user instruction to avoid use before definition crash.
2023-10-16 12:57:18 -07:00
Fangrui Song 2d854dd3e7 Move global namespace cl::opt inside llvm:: or internalize them 2023-10-10 19:58:03 -07:00
Alexey Bataev e22818d5c9 [IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.

Differential Revision: https://reviews.llvm.org/D158449
2023-10-05 06:17:07 -07:00
Arthur Eubanks 07389535a7 Revert "[IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst."
This reverts commit b186f1f68b.

Causes crashes, see https://reviews.llvm.org/D158449.
2023-10-04 14:37:16 -07:00
Alexey Bataev b186f1f68b [IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.

Differential Revision: https://reviews.llvm.org/D158449
2023-10-04 07:53:30 -07:00
Alexey Bataev 1129dec778 Revert "[IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst."
This reverts commit 6f43d28f34 to fix
a crash reported in https://reviews.llvm.org/D158449.
2023-10-03 13:02:16 -07:00
Alexey Bataev 6f43d28f34 [IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.

Differential Revision: https://reviews.llvm.org/D158449
2023-10-03 10:26:11 -07:00
Alexey Bataev d0d608383e [SLP][NFC]Fix assert message, NFC. 2023-10-02 13:38:54 -07:00
Alexey Bataev ebcb5d59fc Revert "[IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst."
This reverts commit 9f5960e004 to fix
buildbots reported here https://lab.llvm.org/buildbot/#/builders/230/builds/19412.
2023-09-29 15:03:46 -07:00
Alexey Bataev 9f5960e004 [IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.

Differential Revision: https://reviews.llvm.org/D158449
2023-09-29 13:16:03 -07:00
Alexey Bataev 019aee8327 [SLP]Improve costs in computeExtractCost() to avoid crash after D158449.
Need to consider the length of the original vector for extractelements,
not the length, matched number of the scalars. It fixes 2 issues: 1)
improves cost estimation; 2) Fixes crashes after D158449.
2023-09-29 07:48:02 -07:00
Hans Wennborg 06f3b0ed43 Revert "[SLP]Improve costs in computeExtractCost() to avoid crash after D158449."
This caused asserts:

  Assertion failed: NumElts > 1 && "Expected at least 2-element fixed length vector(s).",
  file C:\b\s\w\ir\cache\builder\src\third_party\llvm\llvm\lib\Transforms\Vectorize\SLPVectorizer.cpp, line 7096

see comment on https://github.com/llvm/llvm-project/commit/59a67ea35d608480257fc64ec3e5106ef50de740

> Need to consider the length of the original vector for extractelements,
> not the length, matched number of the scalars. It fixes 2 issues: 1)
> improves cost estimation; 2) Fixes crashes after D158449.

This reverts commit 59a67ea35d.
2023-09-29 10:42:19 +02:00
Alexey Bataev 3204f88a8b Revert "[IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst."
This reverts commit c88c281cf1 to fix the
crash revealed by https://lab.llvm.org/buildbot/#/builders/230/builds/19353.
2023-09-28 11:57:32 -07:00
Alexey Bataev c88c281cf1 [IR]Add NumSrcElts param to is..Mask static function in ShuffleVectorInst.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.

Differential Revision: https://reviews.llvm.org/D158449
2023-09-28 11:03:21 -07:00
Alexey Bataev 59a67ea35d [SLP]Improve costs in computeExtractCost() to avoid crash after D158449.
Need to consider the length of the original vector for extractelements,
not the length, matched number of the scalars. It fixes 2 issues: 1)
improves cost estimation; 2) Fixes crashes after D158449.
2023-09-28 09:36:08 -07:00
Alexey Bataev 9eeb0293e2 [SLP]Cleanup MultiNodeScalars when tree deleted.
Need to clear MultiNodeScalars map to avoid compiler crash when tree is
deleted.
2023-09-27 07:48:53 -07:00
Alexey Bataev ea7f43ec14 [SLP]Do not gather node, if the instruction, that does not require
scheduling, is previously vectorized.

If the main node was vectorized already, but does not require
scheduling, we still can try to vectorize it in this new node instead of
gathering.
2023-09-26 11:57:35 -07:00
alexfh 5d86176f48 Revert "[SLP]Do not gather node, if the instruction, that does not require" (#67386)
This reverts commit 7705342122, which
introduces a
clang crash (test case: https://gcc.godbolt.org/z/zn5n4KWPY).
2023-09-26 02:45:11 +02:00
Kazu Hirata e7497570d8 [Vectorize] Use range-based for loops (NFC) 2023-09-22 17:43:06 -07:00
Alexey Bataev 7ff83ed6cd [SLP]Do not try to reorder possible strided nodes.
Reordering of possible strided nodes in bottom-to-top order requires
top-to-bottom reordering of the operands of such nodes, which is not
supported. Need to disable reordering of strided operands to avoid
compiler crashes.
2023-09-22 07:55:43 -07:00