Commit Graph
142 Commits
Author SHA1 Message Date
Joseph Huber 4e80bc7d71 [Clang] Introduce scoped variants of GNU atomic functions (#72280)
Summary:
The standard GNU atomic operations are a very common way to target
hardware atomics on the device. With more heterogenous devices being
introduced, the concept of memory scopes has been in the LLVM language
for awhile via the `syncscope` modifier. For targets, such as the GPU,
this can change code generation depending on whether or not we only need
to be consistent with the memory ordering with the entire system, the
single GPU device, or lower.

Previously these scopes were only exported via the `opencl` and `hip`
variants of these functions. However, this made it difficult to use
outside of those languages and the semantics were different from the
standard GNU versions. This patch introduces a `__scoped_atomic` variant
for the common functions. There was some discussion over whether or not
these should be overloads of the existing ones, or simply new variants.
I leant towards new variants to be less disruptive.

The scope here can be one of the following

```
__MEMORY_SCOPE_SYSTEM // All devices and systems
__MEMORY_SCOPE_DEVICE // Just this device
__MEMORY_SCOPE_WRKGRP // A 'work-group' AKA CUDA block
__MEMORY_SCOPE_WVFRNT // A 'wavefront' AKA CUDA warp
__MEMORY_SCOPE_SINGLE // A single thread.
```
Naming consistency was attempted, but it is difficult to capture to full
spectrum with no many names. Suggestions appreciated.
2023-12-07 13:40:25 -06:00
James Y Knight 4d4c30a37c Use Address for CGBuilder's CreateAtomicRMW and CreateAtomicCmpXchg. (#74349)
Update all callers to pass through the Address.

For the older builtins such as `__sync_*` and MSVC `_Interlocked*`,
natural alignment of the atomic access is _assumed_. This change
preserves that behavior. It will pass through greater-than-required
alignments, however.
2023-12-04 13:37:04 -05:00
Logikable 752c21be68 [clang][NFC] Reorder Atomic builtins to be consistent. (#72718) 2023-11-21 16:00:31 -05:00
Vlad Serebrennikov 49fd28d960 [clang][NFC] Refactor ArrayType::ArraySizeModifier
This patch moves `ArraySizeModifier` before `Type` declaration so that it's complete at `ArrayTypeBitfields` declaration. It's also converted to scoped enum along the way.
2023-10-31 18:06:34 +03:00
Björn Pettersson b4858c634e [clang][CodeGen] Simplify code based on opaque pointers (#65624)
- Update CodeGenTypeCache to use a single union for all pointers in
  address space zero.
- Introduce a UnqualPtrTy in CodeGenTypeCache, and use that (for
  example instead of llvm::PointerType::getUnqual) in some places.
- Drop some redundant bit/pointers casts from ptr to ptr.
2023-09-25 11:21:24 +02:00
Sergei Barannikov 2348902268 [clang][CodeGen] Remove no-op EmitCastToVoidPtr (NFC)
Reviewed By: JOE1994

Differential Revision: https://reviews.llvm.org/D153694
2023-06-29 20:29:38 +03:00
Youngsuk Kim 474ec69419 [clang] Replace uses of CGBuilderTy::CreateElementBitCast (NFC)
Partial progress towards replacing `CreateElementBitCast`, as it no
longer does what its name suggests. Either replace its uses with
`Address::withElementType()`, or remove them if no longer needed.

Reviewed By: barannikov88, nikic

Differential Revision: https://reviews.llvm.org/D153314
2023-06-27 10:38:54 -04:00
Youngsuk Kim 0f4d48d73d [clang] Replace use of Type::getPointerTo() (NFC)
Partial progress towards replacing in-tree uses of `Type::getPointerTo()`.
This needs to be done before deprecating the API.

Reviewed By: nikic, barannikov88

Differential Revision: https://reviews.llvm.org/D152321
2023-06-16 22:07:32 +03:00
Nikita Popov 8a19af513d [Clang] Remove uses of PointerType::getWithSamePointeeType (NFC)
No longer relevant with opaque pointers.
2023-06-12 12:18:28 +02:00
Yaxun (Sam) Liu 00448a548c [clang] Allow fp in atomic fetch max/min builtins
LLVM IR already allows floating point type in atomicrmw.
Update clang atomic fetch max/min builtins to accept
floating point type like we did for fetch add/sub.

Reviewed by: Artem Belevich

Differential Revision: https://reviews.llvm.org/D150985

Fixes: SWDEV-401056
2023-05-31 15:19:31 -04:00
Luke Drummond e3fbede7f3 [HIP] Add missing __hip_atomic_fetch_sub support
The rest of the fetch/op intrinsics were added in e13246a2ec but sub
was conspicuous by its absence.

Reviewed By: yaxunl

Differential Revision: https://reviews.llvm.org/D151701
2023-05-30 22:22:43 +01:00
David Majnemer 2c923b8863 [clang-cl] Expose the /volatile:{iso,ms} choice via _ISO_VOLATILE
MSVC allows interpreting volatile loads and stores, when combined with
/volatile:iso, as having acquire/release semantics. MSVC also exposes a
define, _ISO_VOLATILE, which allows users to enquire if this feature is
enabled or disabled.
2022-08-23 14:29:52 +00:00
Fangrui Song 3f18f7c007 [clang] LLVM_FALLTHROUGH => [[fallthrough]]. NFC
With C++17 there is no Clang pedantic warning or MSVC C5051.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D131346
2022-08-08 09:12:46 -07:00
Akira Hatanaka d112cc2756 [NFC][Clang][OpaquePtr] Remove the call to Address::deprecated in
CreatePointerBitCastOrAddrSpaceCast

Differential Revision: https://reviews.llvm.org/D120757
2022-03-02 08:58:00 -08:00
Nikita Popov 5065076698 [CodeGen] Rename deprecated Address constructor
To make uses of the deprecated constructor easier to spot, and to
ensure that no new uses are introduced, rename it to
Address::deprecated().

While doing the rename, I've filled in element types in cases
where it was relatively obvious, but we're still left with 135
calls to the deprecated constructor.
2022-02-17 11:26:42 +01:00
Arthur Eubanks e487ddc5c6 [clang][OpaquePtr] Use proper Address constructor in AtomicInfo::getAtomicAddress() 2022-02-10 18:29:51 -08:00
Nikita Popov 99adacbcb7 [clang] Remove some getPointerElementType() uses
Same cases where the call can be removed in a straightforward way.
2022-01-25 12:09:06 +01:00
Serge Guelton d2cc6c2d0c Use a sorted array instead of a map to store AttrBuilder string attributes
Using and std::map<SmallString, SmallString> for target dependent attributes is
inefficient: it makes its constructor slightly heavier, and involves extra
allocation for each new string attribute. Storing the attribute key/value as
strings implies extra allocation/copy step.

Use a sorted vector instead. Given the low number of attributes generally
involved, this is cheaper, as showcased by

https://llvm-compile-time-tracker.com/compare.php?from=5de322295f4ade692dc4f1823ae4450ad3c48af2&to=05bc480bf641a9e3b466619af43a2d123ee3f71d&stat=instructions

Differential Revision: https://reviews.llvm.org/D116599
2022-01-10 14:49:53 +01:00
Nikita Popov 481de0ed80 [CodeGen] Prefer CreateElementBitCast() where possible
CreateElementBitCast() can preserve the pointer element type in
the presence of opaque pointers, so use it in place of CreateBitCast()
in some places. This also sometimes simplifies the code a bit.
2021-12-15 11:48:39 +01:00
Nikita Popov b4f46555d7 [CodeGen] Avoid some pointer element type accesses 2021-12-15 09:29:27 +01:00
Anshil Gandhi df0560ca00 [HIP] Add atomic load, atomic store and atomic cmpxchng_weak builtin support in HIP-clang
Introduce `__hip_atomic_load`, `__hip_atomic_store` and `__hip_atomic_compare_exchange_weak`
builtins in HIP.

Reviewed By: yaxunl

Differential Revision: https://reviews.llvm.org/D114553
2021-11-29 12:07:13 -07:00
Yaxun (Sam) Liu e13246a2ec [HIP] Add HIP scope atomic operations
Add an AtomicScopeModel for HIP and support for OpenCL builtins
that are missing in HIP.

Patch by: Michael Liao

Revised by: Anshil Ghandi

Reviewed by: Yaxun Liu

Differential Revision: https://reviews.llvm.org/D113925
2021-11-23 10:13:37 -05:00
Kai Luo 6ea2431d3f [clang][compiler-rt][atomics] Add __c11_atomic_fetch_nand builtin and support __atomic_fetch_nand libcall
Add `__c11_atomic_fetch_nand` builtin to language extensions and support `__atomic_fetch_nand` libcall in compiler-rt.

Reviewed By: theraven

Differential Revision: https://reviews.llvm.org/D112400
2021-10-28 02:18:43 +00:00
Nikita Popov 357756ecf6 [OpaquePtr] Remove uses of CreateConstGEP1_64() without element type
Remove uses of to-be-deprecated API.
2021-07-17 16:43:20 +02:00
Eli Friedman 577fea4e1a [CGAtomic] Delete outdated code comparing success/failure ordering for cmpxchg.
This doesn't actually have any effect: we only call this code with
SequentiallyConsistent orderings.  But delete it anyway for consistency
with other recent changes.
2021-05-28 15:36:01 -07:00
Bruno Cardoso Lopes 819e0d105e [CGAtomic] Lift strong requirement for remaining compare_exchange combinations
Follow up on 431e3138a and complete the other possible combinations.

Besides enforcing the new behavior, it also mitigates TSAN false positives when
combining orders that used to be stronger.
2021-05-06 21:05:20 -07:00
Yaxun (Sam) Liu 61d065e21f Let clang atomic builtins fetch add/sub support floating point types
Recently atomicrmw started to support fadd/fsub:

https://reviews.llvm.org/D53965

However clang atomic builtins fetch add/sub still does not support
emitting atomicrmw fadd/fsub.

This patch adds that.

Reviewed by: John McCall, Artem Belevich, Matt Arsenault, JF Bastien,
James Y Knight, Louis Dionne, Olivier Giroux

Differential Revision: https://reviews.llvm.org/D71726
2021-04-06 15:44:00 -04:00
Bruno Cardoso Lopes 431e3138a1 [CGAtomic] Lift stronger requirements on cmpxch and support acquire failure mode
- Fix `emitAtomicCmpXchgFailureSet` to support release/acquire (succ/fail) memory order.
- Remove stronger checks for cmpxch.

Effectively, this addresses http://wg21.link/p0418

Differential Revision: https://reviews.llvm.org/D98995
2021-03-23 16:45:37 -07:00
Bevin Hansson 101309fe04 [AST] Change return type of getTypeInfoInChars to a proper struct instead of std::pair.
Followup to D85191.

This changes getTypeInfoInChars to return a TypeInfoChars
struct instead of a std::pair of CharUnits. This lets the
interface match getTypeInfo more closely.

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D86447
2020-10-13 13:26:56 +02:00
Christopher Tetreault 19e883fc59 [SVE] Remove calls to VectorType::getNumElements from clang
Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D82582
2020-08-26 11:12:26 -07:00
Gui Andrade 909a851dbf [CGAtomic] Mark atomic libcall functions nounwind
These functions won't ever unwind. This is useful for MemorySanitizer
as it simplifies handling __atomic_load in particular.

Differential Revision: https://reviews.llvm.org/D85573
2020-08-14 07:46:43 +00:00
Thorsten Schuett e18c6ef6b4 [clang] improve diagnostics for misaligned and large atomics
"Listing the alignment and access size (== expected alignment) in the warning
seems like a good idea."

solves PR 46947

  struct Foo {
    struct Bar {
      void * a;
      void * b;
    };
    Bar bar;
  };

  struct ThirtyTwo {
    struct Large {
      void * a;
      void * b;
      void * c;
      void * d;
    };
    Large bar;
  };

  void braz(Foo *foo, ThirtyTwo *braz) {
    Foo::Bar bar;
    __atomic_load(&foo->bar, &bar, __ATOMIC_RELAXED);

    ThirtyTwo::Large foobar;
    __atomic_load(&braz->bar, &foobar, __ATOMIC_RELAXED);
  }

repro.cpp:21:3: warning: misaligned atomic operation may incur significant performance penalty; the expected (16 bytes) exceeds the actual alignment (8 bytes) [-Watomic-alignment]
  __atomic_load(&foo->bar, &bar, __ATOMIC_RELAXED);
  ^
repro.cpp:24:3: warning: misaligned atomic operation may incur significant performance penalty; the expected (32 bytes) exceeds the actual alignment (8 bytes) [-Watomic-alignment]
  __atomic_load(&braz->bar, &foobar, __ATOMIC_RELAXED);
  ^
repro.cpp:24:3: warning: large atomic operation may incur significant performance penalty; the access size (32 bytes) exceeds the max lock-free size (16  bytes) [-Watomic-alignment]
3 warnings generated.

Differential Revision: https://reviews.llvm.org/D85102
2020-08-04 11:10:29 -07:00
Christopher Tetreault f22fbe3a15 Clean up usages of asserting vector getters in Type
Summary:
Remove usages of asserting vector getters in Type in preparation for the
VectorType refactor. The existence of these functions complicates the
refactor while adding little value.

Reviewers: sdesmalen, efriedma, krememek

Reviewed By: sdesmalen, efriedma

Subscribers: dexonsmith, Charusso, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77257
2020-04-13 13:01:40 -07:00
Alexey Bataev e8e05de08b [OPENMP50]Add codegen for acq_rel clause in atomic|flush directives.
Added codegen support for atomic|flush directives with acq_rel clause.
2020-02-07 15:05:09 -05:00
Guillaume Chatelet 1b2842bf90 [Alignment][NFC] CreateMemSet use MaybeAlign
Summary:
This is patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html
See this patch for the introduction of the type: https://reviews.llvm.org/D64790

Reviewers: courbet

Subscribers: arsenm, jvesely, nhaehnle, hiraditya, cfe-commits, llvm-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D71213
2019-12-10 15:17:44 +01:00
Akira Hatanaka f139ae3d93 [NFC] Pass a reference to CodeGenFunction to methods of LValue and
AggValueSlot

This reapplies 8a5b7c3570 after a null
dereference bug in CGOpenMPRuntime::emitUserDefinedMapper.

Original commit message:

This is needed for the pointer authentication work we plan to do in the
near future.

https://github.com/apple/llvm-project/blob/a63a81bd9911f87a0b5dcd5bdd7ccdda7124af87/clang/docs/PointerAuthentication.rst
2019-12-03 15:22:13 -08:00
Akira Hatanaka 9f37c0e703 Revert "[NFC] Pass a reference to CodeGenFunction to methods of LValue and"
This reverts commit 8a5b7c3570. This seems
to have broken UBSan because of a null dereference.
2019-12-03 13:08:01 -08:00
Akira Hatanaka 8a5b7c3570 [NFC] Pass a reference to CodeGenFunction to methods of LValue and
AggValueSlot

This is needed for the pointer authentication work we plan to do in the
near future.

https://github.com/apple/llvm-project/blob/a63a81bd9911f87a0b5dcd5bdd7ccdda7124af87/clang/docs/PointerAuthentication.rst
2019-12-03 11:30:09 -08:00
Tim Northover 5cf58768cb Atomics: support min/max orthogonally
We seem to have been gradually growing support for atomic min/max operations
(exposing longstanding IR atomicrmw instructions). But until now there have
been gaps in the expected intrinsics. This adds support for the C11-style
intrinsics (i.e. taking _Atomic, rather than individually blessed by C11
standard), and the variants that return the new value instead of the original
one.

That way, people won't be misled by trying one form and it not working, and the
front-end is more friendly to people using _Atomic types, as we recommend.
2019-11-21 10:37:56 +00:00
Simon Pilgrim 7e38f0c408 Codegen - silence static analyzer getAs<> null dereference warnings. NFCI.
The static analyzer is warning about potential null dereferences, but in these cases we should be able to use castAs<> directly and if not assert will fire for us.

llvm-svn: 373918
2019-10-07 16:42:25 +00:00
Richard Smith 772e266fbf Properly handle instantiation-dependent array bounds.
We previously failed to treat an array with an instantiation-dependent
but not value-dependent bound as being an instantiation-dependent type.
We now track the array bound expression as part of a constant array type
if it's an instantiation-dependent expression.

llvm-svn: 373685
2019-10-04 01:25:59 +00:00
Simon Pilgrim e0712019f2 Silence static analyzer getAs<VectorType> null dereference warnings. NFCI.
The static analyzer is warning about potential null dereferences, but in these cases we should be able to use castAs<VectorType> directly and if not assert will fire for us.

llvm-svn: 373478
2019-10-02 15:31:25 +00:00
Amy Huang 144a43a2fd add periods
llvm-svn: 357643
2019-04-03 22:19:07 +00:00
Fangrui Song 9d568e29b7 [CodeGen] Delete never used LValueAlign
It was added by rC176658 but never used since then.

llvm-svn: 357001
2019-03-26 15:39:45 +00:00
Konstantin Zhuravlyov ec28a1dcef AMDGPU: Add support for cross address space synchronization scopes (clang)
Differential Revision: https://reviews.llvm.org/D59494

llvm-svn: 356947
2019-03-25 20:54:00 +00:00
James Y Knight 751fe286dc [opaque pointer types] Cleanup CGBuilder's Create*GEP.
The various EltSize, Offset, DataLayout, and StructLayout arguments
are all computable from the Address's element type and the DataLayout
which the CGBuilder already has access to.

After having previously asserted that the computed values are the same
as those passed in, now remove the redundant arguments from
CGBuilder's Create*GEP functions.

Differential Revision: https://reviews.llvm.org/D57767

llvm-svn: 353629
2019-02-09 22:22:28 +00:00
James Y Knight 9871db064d [opaque pointer types] Pass function types for runtime function calls.
Emit{Nounwind,}RuntimeCall{,OrInvoke} have been modified to take a
FunctionCallee as an argument, and CreateRuntimeFunction has been
modified to return a FunctionCallee. All callers have been updated.

Additionally, CreateBuiltinFunction is removed, as it was redundant
with CreateRuntimeFunction after some previous changes.

Differential Revision: https://reviews.llvm.org/D57668

llvm-svn: 353184
2019-02-05 16:42:33 +00:00
Chandler Carruth 2946cd7010 Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636
2019-01-19 08:50:56 +00:00
Richard Trieu 5337c74825 Remove CodeGen dependencies on Sema.
Move diagnostics from Sema to Frontend (or Common) so that CodeGen no longer
needs to include the Sema diagnostic IDs.

llvm-svn: 348458
2018-12-06 06:12:20 +00:00
Richard Smith da3729d1e6 Do not use optimized atomic libcalls for misaligned atomics.
Summary:
The optimized (__atomic_foo_<n>) libcalls assume that the atomic object
is properly aligned, so should never be called on an underaligned
object.

This addresses one of several problems identified in PR38846.

Reviewers: jyknight, t.p.northover

Subscribers: jfb, cfe-commits

Differential Revision: https://reviews.llvm.org/D51817

llvm-svn: 341734
2018-09-07 23:57:54 +00:00