[clang][sycl][nvlink] Share static library linking in Frontend/Offloading (#201253)
Move clang-nvlink-wrapper's archive member selection engine into a new shared library in llvm/lib/Frontend/Offloading (ArchiveLinker.h/.cpp) and use it from both clang-nvlink-wrapper and clang-sycl-linker, adding static library (.a) and -l support to the SYCL linker. The shared llvm::offloading::resolveArchiveMembers() API: - Searches -L paths for -l library names (lib<name>.a or :<name>) - Expands archives, honouring --whole-archive/--no-whole-archive - Runs a symbol-driven fixed-point loop to extract only the archive members that resolve undefined symbols - Returns the resolved MemoryBuffers and symbol table; the symbol table is consumed by clang-nvlink-wrapper's LTO resolution pass clang-sycl-linker gains -l, --whole-archive/--no-whole-archive, and -u options (added to SYCLLinkOpts.td). The previous --bc-library option has been removed in favor of the standard -l mechanism. Bug fixes included: * Fix dangling StringRef UB: Args.getAllArgValues() returns a temporary vector; retain it in ForcedUndefStorage so the StringRefs remain valid through the resolveArchiveMembers call (both tools). * Fix assert crash in clang-sycl-linker when all positional inputs are non-existent: return a proper error instead of propagating an empty buffer vector to linkInputs. * Fixed forced undefined symbol handling: corrected -u option processing to properly handle symbols. Co-Authored-By: Claude
This commit is contained in:
Alexey BaderandClaude authored and GitHub committed
1 parent
1dae5dee50
commit
8aafa50c7a
12 files changed
+804
-406
No files matched your search
@@ -50,7 +50,10 @@ be passed down to downstream AOT compilation tools like 'ocloc' and 'opencl-aot'
|
||||
-help-hidden Display all available options
|
||||
-help Display available options (--help-hidden for more)
|
||||
-L <dir> Add <dir> to the library search path
|
||||
--bc-library <name> Add LLVM bitcode library <name> (with extension) to the link. A relative <name> is resolved against -L paths; an absolute path is taken as-is.
|
||||
-l <libname> Search for library <libname>
|
||||
--whole-archive Include all archive members in the link
|
||||
--no-whole-archive Only include archive members that resolve undefined symbols (default)
|
||||
-u <symbol> Force undefined symbol during linking
|
||||
--module-split-mode=<mode> Module split mode: 'source' (default), 'kernel', or 'none'
|
||||
--ocloc-options=<value> Options passed to ocloc for Intel GPU AOT compilation
|
||||
--opencl-aot-options=<value> Options passed to opencl-aot for Intel CPU AOT compilation
|
||||
@@ -61,8 +64,58 @@ be passed down to downstream AOT compilation tools like 'ocloc' and 'opencl-aot'
|
||||
-v Print verbose information
|
||||
-spirv-dump-device-code=<dir> Directory to dump SPIR-V IR code into
|
||||
|
||||
Example
|
||||
=======
|
||||
Library Linking
|
||||
===============
|
||||
|
||||
Device bitcode libraries can be packaged into archive libraries (``.a`` files)
|
||||
using ``llvm-ar`` and linked using the ``-l`` option:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
llvm-ar rc libdevice.a func1.bc func2.bc func3.bc
|
||||
clang-sycl-linker input.bc -l device -L /path/to/libs
|
||||
|
||||
The linker supports standard archive library search semantics:
|
||||
|
||||
* ``-l <name>`` searches for ``lib<name>.a`` in the directories specified by ``-L``
|
||||
* ``-l :<exact-name>`` searches for the exact filename in the ``-L`` paths
|
||||
* Absolute paths can be passed as positional arguments: ``clang-sycl-linker input.bc /path/to/libdevice.a``
|
||||
|
||||
By default, archive linking is **lazy** - only archive members (individual ``.bc`` files)
|
||||
that resolve undefined symbols are extracted and linked. This happens at file
|
||||
granularity: if any symbol in a ``.bc`` file is needed, all symbols in that file
|
||||
are included. The linker uses a symbol-driven fixed-point algorithm: it
|
||||
repeatedly scans archives to extract members that resolve currently undefined
|
||||
symbols until no more extractions occur.
|
||||
|
||||
To force extraction of all archive members regardless of symbol resolution, use
|
||||
``--whole-archive``:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
clang-sycl-linker input.bc --whole-archive -l device --no-whole-archive -l other
|
||||
|
||||
The ``-u <symbol>`` option can be used to force a symbol to be undefined, which
|
||||
can trigger extraction of archive members that define that symbol:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
clang-sycl-linker input.bc -u my_init_function -l device
|
||||
|
||||
Implementation
|
||||
--------------
|
||||
|
||||
Archive linking in ``clang-sycl-linker`` is implemented using the shared
|
||||
``llvm::offloading::resolveArchiveMembers()`` API from
|
||||
``llvm/lib/Frontend/Offloading/ArchiveLinker.cpp``. This same infrastructure is
|
||||
also used by ``clang-nvlink-wrapper``, ensuring consistent archive linking
|
||||
semantics across offloading tools.
|
||||
|
||||
Examples
|
||||
========
|
||||
|
||||
Basic Usage
|
||||
-----------
|
||||
|
||||
This tool is intended to be invoked when targeting any of the target offloading
|
||||
toolchains. When the --sycl-link option is passed to the clang driver, the
|
||||
@@ -74,3 +127,24 @@ generate the final executable.
|
||||
.. code-block:: console
|
||||
|
||||
clang-sycl-linker --triple spirv64 --arch bmg_g21 input.bc
|
||||
|
||||
Linking with Device Libraries
|
||||
------------------------------
|
||||
|
||||
To link device bitcode libraries, first package them into archive files:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# Create device library archives
|
||||
llvm-ar rc libmath.a sin.bc cos.bc tan.bc
|
||||
llvm-ar rc libutils.a helper1.bc helper2.bc
|
||||
|
||||
# Link with lazy loading (only needed members extracted)
|
||||
clang-sycl-linker --triple spirv64 kernel.bc -l math -l utils -L /path/to/libs -o kernel.spv
|
||||
|
||||
# Force all members to be included from libmath.a
|
||||
clang-sycl-linker --triple spirv64 kernel.bc --whole-archive -l math --no-whole-archive -l utils -L /path/to/libs -o kernel.spv
|
||||
|
||||
# Use exact archive filename or absolute path
|
||||
clang-sycl-linker --triple spirv64 kernel.bc -l :libmath.a -L /path/to/libs -o kernel.spv
|
||||
clang-sycl-linker --triple spirv64 kernel.bc /absolute/path/libmath.a -o kernel.spv
|
||||
@@ -22,76 +22,100 @@
|
||||
;
|
||||
; Test non-existent input file
|
||||
; RUN: not clang-sycl-linker %t-missing.bc -o %t.out 2>&1 | FileCheck %s --check-prefix=MISSING
|
||||
; MISSING: Input file '{{.*}}-missing.bc' does not exist
|
||||
; MISSING: input file not found: '{{.*}}-missing.bc'
|
||||
;
|
||||
; Test the dry run of a simple case to link two input files.
|
||||
; Test that IMG_SPIRV image kind is set for non-AOT compilation.
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc %t/input2.bc -o %t/spirv.out 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=SIMPLE-FO
|
||||
; SIMPLE-FO: link: inputs: {{.*}}.bc, {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
|
||||
; SIMPLE-FO: link: inputs: {{.*}}.bc, {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
|
||||
; SIMPLE-FO-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: {{.*}}_0.spv
|
||||
; SIMPLE-FO-NEXT: sycl-bundle: image kind: spv, triple: spirv64, arch: {{$}}
|
||||
; SIMPLE-FO-NOT: {{.+}}
|
||||
;
|
||||
; Test the dry run of a simple case with device library files specified.
|
||||
; Test the dry run of a simple case with device library archive specified using --whole-archive.
|
||||
; RUN: mkdir -p %t/libs
|
||||
; RUN: touch %t/libs/lib1.bc
|
||||
; RUN: touch %t/libs/lib2.bc
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc %t/input2.bc --library-path=%t/libs --bc-library lib1.bc --bc-library lib2.bc -o a.spv 2>&1 \
|
||||
; RUN: llvm-as %t/lib1.ll -o %t/libs/lib1.bc
|
||||
; RUN: llvm-as %t/lib2.ll -o %t/libs/lib2.bc
|
||||
; RUN: rm -f %t/libs/libdevice.a
|
||||
; RUN: llvm-ar rc %t/libs/libdevice.a %t/libs/lib1.bc %t/libs/lib2.bc
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc %t/input2.bc --library-path=%t/libs --whole-archive -l device -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=DEVLIBS
|
||||
; DEVLIBS: link: inputs: {{.*}}.bc libfiles: {{.*}}lib1.bc, {{.*}}lib2.bc output: [[LLVMLINKOUT:.*]].bc
|
||||
; DEVLIBS: link: inputs: {{.*}}.bc, {{.*}}.bc, {{.*}}libdevice.a(lib1.bc), {{.*}}libdevice.a(lib2.bc) output: [[LLVMLINKOUT:.*]].bc
|
||||
; DEVLIBS-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: a_0.spv
|
||||
; DEVLIBS-NEXT: sycl-bundle: image kind: spv, triple: spirv64, arch: {{$}}
|
||||
; DEVLIBS-NOT: {{.+}}
|
||||
;
|
||||
; Test -L short form (joined) and --bc-library= joined form.
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L%t/libs --bc-library=lib1.bc -o a.spv 2>&1 \
|
||||
; Test -L short form (joined) and -l with archive using --whole-archive.
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L%t/libs --whole-archive -l device -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=DEVLIBS-SHORT
|
||||
; DEVLIBS-SHORT: link: inputs: {{.*}}.bc libfiles: {{.*}}libs{{[/\\]}}lib1.bc output: {{.*}}.bc
|
||||
; DEVLIBS-SHORT: link: inputs: {{.*}}.bc, {{.*}}libdevice.a(lib1.bc), {{.*}}libdevice.a(lib2.bc) output: {{.*}}.bc
|
||||
;
|
||||
; Test that search continues past the first -L when the library is not found there. lib1.bc exists only in %t/libs (the second -L).
|
||||
; Test that search continues past the first -L when the library is not found there. libdevice.a exists only in %t/libs (the second -L).
|
||||
; RUN: mkdir -p %t/empty
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L %t/empty -L %t/libs --bc-library lib1.bc -o a.spv 2>&1 \
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L %t/empty -L %t/libs --whole-archive -l device -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=DEVLIBS-FALLTHROUGH
|
||||
; DEVLIBS-FALLTHROUGH: link: inputs: {{.*}}.bc libfiles: {{.*}}libs{{[/\\]}}lib1.bc output: {{.*}}.bc
|
||||
;
|
||||
; Test that -L paths are searched in order: when the same name exists in multiple -L dirs, the first one wins.
|
||||
; RUN: mkdir -p %t/libs2
|
||||
; RUN: touch %t/libs/shadow.bc %t/libs2/shadow.bc
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L %t/libs2 -L %t/libs --bc-library shadow.bc -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=DEVLIBS-ORDER
|
||||
; DEVLIBS-ORDER: link: inputs: {{.*}}.bc libfiles: {{.*}}libs2{{[/\\]}}shadow.bc output: {{.*}}.bc
|
||||
; DEVLIBS-FALLTHROUGH: link: inputs: {{.*}}.bc, {{.*}}libdevice.a(lib1.bc), {{.*}}libdevice.a(lib2.bc) output: {{.*}}.bc
|
||||
;
|
||||
; Test a simple case with a random file (not bitcode) as input.
|
||||
; RUN: touch %t/dummy.o
|
||||
; RUN: not clang-sycl-linker %t/dummy.o -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=FILETYPEERROR
|
||||
; FILETYPEERROR: Unsupported file type
|
||||
; FILETYPEERROR: Unsupported file type: '{{.*}}dummy.o'
|
||||
;
|
||||
; Test that unsupported file type error includes buffer identifier when found inside an archive.
|
||||
; Create an archive containing an unsupported file (text file instead of bitcode).
|
||||
; RUN: echo "not bitcode" > %t/invalid.txt
|
||||
; RUN: rm -f %t/libinvalid.a
|
||||
; RUN: llvm-ar rc %t/libinvalid.a %t/invalid.txt
|
||||
; RUN: not clang-sycl-linker --dry-run %t/input1.bc -L %t --whole-archive -l invalid -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=ARCHIVE-INVALID-MEMBER
|
||||
; ARCHIVE-INVALID-MEMBER: Unsupported file type: '{{.*}}libinvalid.a(invalid.txt)'
|
||||
;
|
||||
; Test mixed archive: valid bitcode member + invalid member.
|
||||
; The error should clearly identify which member is invalid.
|
||||
; RUN: rm -f %t/libmixed.a
|
||||
; RUN: llvm-ar rc %t/libmixed.a %t/libs/lib1.bc %t/invalid.txt
|
||||
; RUN: not clang-sycl-linker --dry-run %t/input1.bc -L %t --whole-archive -l mixed -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=ARCHIVE-MIXED-INVALID
|
||||
; ARCHIVE-MIXED-INVALID: Unsupported file type: '{{.*}}libmixed.a(invalid.txt)'
|
||||
;
|
||||
; Test to see if device library related errors are emitted.
|
||||
; RUN: not clang-sycl-linker --dry-run %t/input1.bc %t/input2.bc --library-path=%t/libs --bc-library lib1.bc --bc-library lib2.bc --bc-library lib3.bc -o a.spv 2>&1 \
|
||||
; RUN: not clang-sycl-linker --dry-run %t/input1.bc %t/input2.bc --library-path=%t/libs -l device -l nonexistent -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=DEVLIBSERR
|
||||
; DEVLIBSERR: '{{.*}}lib3.bc' library file not found
|
||||
; DEVLIBSERR: unable to find library -lnonexistent
|
||||
;
|
||||
; Test that there is no implicit CWD search: a bare bitcode name without any -L
|
||||
; Test that there is no implicit CWD search: a bare library name without any -L
|
||||
; must fail to resolve, even if a same-named file exists in the CWD.
|
||||
; RUN: cd %t && not clang-sycl-linker --dry-run input1.bc --bc-library input1.bc -o a.spv 2>&1 \
|
||||
; RUN: cd %t && not clang-sycl-linker --dry-run input1.bc -l mixed -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=NO-CWD-SEARCH
|
||||
; NO-CWD-SEARCH: 'input1.bc' library file not found
|
||||
; NO-CWD-SEARCH: unable to find library -lmixed
|
||||
;
|
||||
; Test that a directory matching the requested name is not accepted as a library:
|
||||
; %t/libs is a directory created above; resolving --bc-library libs against -L %t
|
||||
; would otherwise pick it up and fail later with a confusing bitcode-reader error.
|
||||
; RUN: not clang-sycl-linker --dry-run %t/input1.bc -L %t --bc-library libs -o a.spv 2>&1 \
|
||||
; %t/libs is a directory created above; resolving -l:libs against -L %t
|
||||
; would detect it's a directory and error with the filename in the message.
|
||||
; RUN: not clang-sycl-linker --dry-run %t/input1.bc -L %t -l :libs -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=NO-DIR-AS-LIB
|
||||
; NO-DIR-AS-LIB: 'libs' library file not found
|
||||
; NO-DIR-AS-LIB: '{{.*}}libs': Is a directory
|
||||
;
|
||||
; Test that providing only an empty archive results in "No input files could be resolved" error
|
||||
; RUN: rm -f %t/empty.a
|
||||
; RUN: llvm-ar rc %t/empty.a
|
||||
; RUN: not clang-sycl-linker --dry-run --whole-archive %t/empty.a -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=NO-RESOLVED-INPUT
|
||||
; NO-RESOLVED-INPUT: No input files could be resolved
|
||||
;
|
||||
; Test that providing only a lazy archive with no extracted members results in "No input files could be resolved" error
|
||||
; RUN: not clang-sycl-linker --dry-run %t/libs/libdevice.a -o a.spv 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=NO-RESOLVED-LAZY
|
||||
; NO-RESOLVED-LAZY: No input files could be resolved
|
||||
;
|
||||
; Test AOT compilation for an Intel GPU.
|
||||
; Test that IMG_Object image kind is set for AOT compilation (Intel GPU).
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none -arch=bmg_g21 %t/input1.bc %t/input2.bc -o %t/aot-gpu.out 2>&1 \
|
||||
; RUN: --ocloc-options="-a -b" \
|
||||
; RUN: | FileCheck %s --check-prefix=AOT-INTEL-GPU
|
||||
; AOT-INTEL-GPU: link: inputs: {{.*}}.bc, {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
|
||||
; AOT-INTEL-GPU: link: inputs: {{.*}}.bc, {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
|
||||
; AOT-INTEL-GPU-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: [[SPIRVTRANSLATIONOUT:.*]]_0.spv
|
||||
; AOT-INTEL-GPU-NEXT: "{{.*}}ocloc{{.*}}" {{.*}}-device bmg_g21 -a -b {{.*}}-output [[SPIRVTRANSLATIONOUT]]_0.out -file [[SPIRVTRANSLATIONOUT]]_0.spv
|
||||
; AOT-INTEL-GPU-NEXT: sycl-bundle: image kind: o, triple: spirv64, arch: bmg_g21
|
||||
@@ -102,7 +126,7 @@
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none -arch=graniterapids %t/input1.bc %t/input2.bc -o %t/aot-cpu.out 2>&1 \
|
||||
; RUN: --opencl-aot-options="-a -b" \
|
||||
; RUN: | FileCheck %s --check-prefix=AOT-INTEL-CPU
|
||||
; AOT-INTEL-CPU: link: inputs: {{.*}}.bc, {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
|
||||
; AOT-INTEL-CPU: link: inputs: {{.*}}.bc, {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
|
||||
; AOT-INTEL-CPU-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: [[SPIRVTRANSLATIONOUT:.*]]_0.spv
|
||||
; AOT-INTEL-CPU-NEXT: "{{.*}}opencl-aot{{.*}}" {{.*}}--device=cpu -a -b {{.*}}-o [[SPIRVTRANSLATIONOUT]]_0.out [[SPIRVTRANSLATIONOUT]]_0.spv
|
||||
; AOT-INTEL-CPU-NEXT: sycl-bundle: image kind: o, triple: spirv64, arch: graniterapids
|
||||
@@ -164,3 +188,19 @@ target triple = "spirv64"
|
||||
define spir_func i32 @helper() {
|
||||
ret i32 0
|
||||
}
|
||||
|
||||
;--- lib1.ll
|
||||
target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n8:16:32:64-G1"
|
||||
target triple = "spirv64"
|
||||
|
||||
define spir_func i32 @lib1_func() {
|
||||
ret i32 1
|
||||
}
|
||||
|
||||
;--- lib2.ll
|
||||
target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n8:16:32:64-G1"
|
||||
target triple = "spirv64"
|
||||
|
||||
define spir_func i32 @lib2_func() {
|
||||
ret i32 2
|
||||
}
|
||||
@@ -7,6 +7,11 @@
|
||||
; RUN: llvm-as %t/bar.ll -o %t/bar.bc
|
||||
; RUN: llvm-as %t/baz.ll -o %t/baz.bc
|
||||
; RUN: llvm-as %t/libfoo.ll -o %t/libfoo.bc
|
||||
; RUN: llvm-as %t/addFive.ll -o %t/addFive.bc
|
||||
; RUN: llvm-as %t/unusedFunc.ll -o %t/unusedFunc.bc
|
||||
; RUN: rm -f %t/libfoo.a %t/libdevice.a
|
||||
; RUN: llvm-ar rc %t/libfoo.a %t/libfoo.bc
|
||||
; RUN: llvm-ar rc %t/libdevice.a %t/addFive.bc %t/unusedFunc.bc
|
||||
;
|
||||
; Test linking two input files.
|
||||
; RUN: clang-sycl-linker %t/foo.bc %t/bar.bc --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
@@ -22,18 +27,101 @@
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-MULTIPLE-DEFS
|
||||
; CHECK-MULTIPLE-DEFS: error: Linking globals named {{.*}}bar_func1{{.*}} symbol multiply defined!
|
||||
;
|
||||
; Test linking with a BC library file resolved through -L.
|
||||
; RUN: clang-sycl-linker %t/foo.bc %t/bar.bc --bc-library libfoo.bc -L %t --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; Test lazy linking with an archive library: only needed members are extracted.
|
||||
; foo.bc references addFive, so addFive.bc is extracted from libdevice.a.
|
||||
; unusedFunc.bc is not needed, so it should NOT be extracted.
|
||||
; RUN: clang-sycl-linker %t/foo.bc %t/bar.bc -l device -L %t --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-LAZY-LINK
|
||||
; CHECK-LAZY-LINK: define {{.*}}foo_func1{{.*}}
|
||||
; CHECK-LAZY-LINK: define {{.*}}foo_func2{{.*}}
|
||||
; CHECK-LAZY-LINK: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-LAZY-LINK: define {{.*}}addFive{{.*}}
|
||||
; CHECK-LAZY-LINK-NOT: define {{.*}}unusedFunc{{.*}}
|
||||
;
|
||||
; Test linking with an archive library file using -l:libname.a syntax.
|
||||
; Archive linking extracts members at file granularity, so all functions in libfoo.bc are included.
|
||||
; RUN: clang-sycl-linker %t/foo.bc %t/bar.bc -l :libfoo.a -L %t --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-DEVICE-LIB
|
||||
; CHECK-DEVICE-LIB: define {{.*}}foo_func1{{.*}}
|
||||
; CHECK-DEVICE-LIB: define {{.*}}foo_func2{{.*}}
|
||||
; CHECK-DEVICE-LIB: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-DEVICE-LIB: define {{.*}}addFive{{.*}}
|
||||
; CHECK-DEVICE-LIB-NOT: define {{.*}}unusedFunc{{.*}}
|
||||
; CHECK-DEVICE-LIB: define {{.*}}unusedFunc{{.*}}
|
||||
;
|
||||
; Test that an absolute path to --bc-library is taken as-is, with no -L required.
|
||||
; RUN: clang-sycl-linker %t/foo.bc %t/bar.bc --bc-library %t/libfoo.bc --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-DEVICE-LIB
|
||||
; Test that an absolute path as a positional argument still performs lazy member extraction.
|
||||
; libdevice.a has two members (addFive.bc and unusedFunc.bc).
|
||||
; Since foo.bc needs addFive, only addFive.bc member is extracted; unusedFunc.bc is not.
|
||||
; RUN: clang-sycl-linker %t/foo.bc %t/bar.bc %t/libdevice.a --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-DEVICE-LIB-POS
|
||||
; CHECK-DEVICE-LIB-POS: define {{.*}}foo_func1{{.*}}
|
||||
; CHECK-DEVICE-LIB-POS: define {{.*}}foo_func2{{.*}}
|
||||
; CHECK-DEVICE-LIB-POS: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-DEVICE-LIB-POS: define {{.*}}addFive{{.*}}
|
||||
; CHECK-DEVICE-LIB-POS-NOT: define {{.*}}unusedFunc{{.*}}
|
||||
;
|
||||
; Test that -L paths are searched in order: when the same name exists in multiple -L dirs, the first one wins.
|
||||
; RUN: mkdir -p %t/libs1 %t/libs2
|
||||
; RUN: rm -f %t/libs1/libshadow.a %t/libs2/libshadow.a
|
||||
; RUN: llvm-ar rc %t/libs1/libshadow.a %t/addFive.bc
|
||||
; RUN: llvm-ar rc %t/libs2/libshadow.a %t/unusedFunc.bc
|
||||
; RUN: clang-sycl-linker %t/foo.bc -L %t/libs2 -L %t/libs1 --whole-archive -l shadow --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-LIB-ORDER
|
||||
; CHECK-LIB-ORDER: define {{.*}}unusedFunc
|
||||
; CHECK-LIB-ORDER-NOT: define {{.*}}addFive
|
||||
;
|
||||
; Test that -u forces extraction of an otherwise-unreferenced archive member.
|
||||
; Without -u, unusedFunc is not extracted. With -u unusedFunc, it is pulled in.
|
||||
; RUN: clang-sycl-linker %t/bar.bc %t/libdevice.a --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-NO-FORCE-UNDEF
|
||||
; CHECK-NO-FORCE-UNDEF: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-NO-FORCE-UNDEF-NOT: define {{.*}}unusedFunc{{.*}}
|
||||
; CHECK-NO-FORCE-UNDEF-NOT: define {{.*}}addFive{{.*}}
|
||||
;
|
||||
; RUN: clang-sycl-linker %t/bar.bc %t/libdevice.a -u unusedFunc --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-FORCE-UNDEF
|
||||
; CHECK-FORCE-UNDEF: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-FORCE-UNDEF: define {{.*}}unusedFunc{{.*}}
|
||||
; CHECK-FORCE-UNDEF-NOT: define {{.*}}addFive{{.*}}
|
||||
;
|
||||
; Test that multiple -u flags work correctly and extract all specified members.
|
||||
; RUN: clang-sycl-linker %t/bar.bc %t/libdevice.a -u unusedFunc -u addFive --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-MULTI-UNDEF
|
||||
; CHECK-MULTI-UNDEF: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-MULTI-UNDEF: define {{.*}}addFive{{.*}}
|
||||
; CHECK-MULTI-UNDEF: define {{.*}}unusedFunc{{.*}}
|
||||
;
|
||||
; Test that -u works correctly with -l library syntax (not just positional archives).
|
||||
; RUN: clang-sycl-linker %t/bar.bc -l device -L %t -u unusedFunc --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-UNDEF-WITH-L
|
||||
; CHECK-UNDEF-WITH-L: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-UNDEF-WITH-L: define {{.*}}unusedFunc{{.*}}
|
||||
; CHECK-UNDEF-WITH-L-NOT: define {{.*}}addFive{{.*}}
|
||||
;
|
||||
; Test that -u combined with actual references works correctly (both should be extracted).
|
||||
; foo.bc references addFive, and -u forces unusedFunc.
|
||||
; RUN: clang-sycl-linker %t/foo.bc %t/bar.bc %t/libdevice.a -u unusedFunc --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-UNDEF-PLUS-REF
|
||||
; CHECK-UNDEF-PLUS-REF: define {{.*}}foo_func1{{.*}}
|
||||
; CHECK-UNDEF-PLUS-REF: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-UNDEF-PLUS-REF: define {{.*}}addFive{{.*}}
|
||||
; CHECK-UNDEF-PLUS-REF: define {{.*}}unusedFunc{{.*}}
|
||||
;
|
||||
; Regression test: -u symbol should remain undefined until resolved by archive member.
|
||||
; This test verifies the fix for the bug where forced-undefined entries were overwritten
|
||||
; before ResolvesReference was computed, making -u ineffective.
|
||||
; RUN: clang-sycl-linker %t/bar.bc -u addFive %t/libdevice.a --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-UNDEF-REMAINS
|
||||
; CHECK-UNDEF-REMAINS: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-UNDEF-REMAINS: define {{.*}}addFive{{.*}}
|
||||
; CHECK-UNDEF-REMAINS-NOT: define {{.*}}unusedFunc{{.*}}
|
||||
;
|
||||
; Test -u with archive processed BEFORE the symbol table has been populated by regular inputs.
|
||||
; This specifically tests that the forced-undefined placeholder survives initial processing.
|
||||
; RUN: clang-sycl-linker -u addFive %t/libdevice.a %t/bar.bc --dry-run -o a.spv --print-linked-module 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=CHECK-UNDEF-FIRST
|
||||
; CHECK-UNDEF-FIRST: define {{.*}}addFive{{.*}}
|
||||
; CHECK-UNDEF-FIRST: define {{.*}}bar_func1{{.*}}
|
||||
; CHECK-UNDEF-FIRST-NOT: define {{.*}}unusedFunc{{.*}}
|
||||
|
||||
;--- foo.ll
|
||||
target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n8:16:32:64-G1"
|
||||
@@ -93,3 +181,23 @@ entry:
|
||||
%res = mul nsw i32 %a, 5
|
||||
ret i32 %res
|
||||
}
|
||||
|
||||
;--- addFive.ll
|
||||
target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n8:16:32:64-G1"
|
||||
target triple = "spirv64"
|
||||
|
||||
define spir_func i32 @addFive(i32 %a) {
|
||||
entry:
|
||||
%res = add nsw i32 %a, 5
|
||||
ret i32 %res
|
||||
}
|
||||
|
||||
;--- unusedFunc.ll
|
||||
target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n8:16:32:64-G1"
|
||||
target triple = "spirv64"
|
||||
|
||||
define spir_func i32 @unusedFunc(i32 %a) {
|
||||
entry:
|
||||
%res = mul nsw i32 %a, 5
|
||||
ret i32 %res
|
||||
}
|
||||
@@ -12,7 +12,7 @@
|
||||
; Test the split mode ("none"): kernels from different TUs are not split into separate images.
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t.bc -o %t-none.out 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=SPLIT-NONE
|
||||
; SPLIT-NONE: link: inputs: {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
|
||||
; SPLIT-NONE: link: inputs: {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
|
||||
; SPLIT-NONE-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: {{.*}}_0.spv
|
||||
; SPLIT-NONE-NEXT: sycl-bundle: image kind: spv, triple: spirv64, arch: {{$}}
|
||||
; SPLIT-NONE-NOT: {{.+}}
|
||||
@@ -20,7 +20,7 @@
|
||||
; Test the split mode ("kernel"): each SPIR_KERNEL function produces its own device image.
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=kernel %t.bc -o %t-split-kernel.out 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=SPLIT-KERNEL
|
||||
; SPLIT-KERNEL: link: inputs: {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
|
||||
; SPLIT-KERNEL: link: inputs: {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
|
||||
; SPLIT-KERNEL-NEXT: sycl-module-split: input: [[LLVMLINKOUT]].bc, mode: kernel
|
||||
; SPLIT-KERNEL-NEXT: [[SPLIT0:.*]].bc [kernel_c ]
|
||||
; SPLIT-KERNEL-NEXT: [[SPLIT1:.*]].bc [kernel_b ]
|
||||
@@ -43,7 +43,7 @@
|
||||
; Test per-TU split ('source' explicitly provided)
|
||||
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=source %t.bc -o %t-src.out 2>&1 \
|
||||
; RUN: | FileCheck %s --check-prefix=SPLIT-SRC
|
||||
; SPLIT-SRC: link: inputs: {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
|
||||
; SPLIT-SRC: link: inputs: {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
|
||||
; SPLIT-SRC-NEXT: sycl-module-split: input: [[LLVMLINKOUT]].bc, mode: source
|
||||
; SPLIT-SRC-NEXT: [[S0:.*]].bc [kernel_b kernel_c ]
|
||||
; SPLIT-SRC-NEXT: [[S1:.*]].bc [kernel_a ]
|
||||
|
||||
@@ -63,6 +63,8 @@ define spir_kernel void @kernel_c() #0 {
|
||||
attributes #0 = { "sycl-module-id"="TU3.cpp" }
|
||||
|
||||
;--- no-triple.ll
|
||||
target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n8:16:32:64-G1"
|
||||
|
||||
define spir_kernel void @kernel_d() #0 {
|
||||
ret void
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ set(LLVM_LINK_COMPONENTS
|
||||
BitWriter
|
||||
Core
|
||||
BinaryFormat
|
||||
FrontendOffloading
|
||||
MC
|
||||
Target
|
||||
TransformUtils
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
#include "llvm/BinaryFormat/Magic.h"
|
||||
#include "llvm/Bitcode/BitcodeWriter.h"
|
||||
#include "llvm/CodeGen/CommandFlags.h"
|
||||
#include "llvm/Frontend/Offloading/ArchiveLinker.h"
|
||||
#include "llvm/IR/DiagnosticPrinter.h"
|
||||
#include "llvm/LTO/LTO.h"
|
||||
#include "llvm/Object/Archive.h"
|
||||
@@ -206,47 +207,6 @@ Expected<std::string> findProgram(const ArgList &Args, StringRef Name,
|
||||
return *Path;
|
||||
}
|
||||
|
||||
std::optional<std::string> findFile(StringRef Dir, StringRef Root,
|
||||
const Twine &Name) {
|
||||
SmallString<128> Path;
|
||||
if (Dir.starts_with("="))
|
||||
sys::path::append(Path, Root, Dir.substr(1), Name);
|
||||
else
|
||||
sys::path::append(Path, Dir, Name);
|
||||
|
||||
if (sys::fs::exists(Path))
|
||||
return static_cast<std::string>(Path);
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
std::optional<std::string>
|
||||
findFromSearchPaths(StringRef Name, StringRef Root,
|
||||
ArrayRef<StringRef> SearchPaths) {
|
||||
for (StringRef Dir : SearchPaths)
|
||||
if (std::optional<std::string> File = findFile(Dir, Root, Name))
|
||||
return File;
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
std::optional<std::string>
|
||||
searchLibraryBaseName(StringRef Name, StringRef Root,
|
||||
ArrayRef<StringRef> SearchPaths) {
|
||||
for (StringRef Dir : SearchPaths)
|
||||
if (std::optional<std::string> File =
|
||||
findFile(Dir, Root, "lib" + Name + ".a"))
|
||||
return File;
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
/// Search for static libraries in the linker's library path given input like
|
||||
/// `-lfoo` or `-l:libfoo.a`.
|
||||
std::optional<std::string> searchLibrary(StringRef Input, StringRef Root,
|
||||
ArrayRef<StringRef> SearchPaths) {
|
||||
if (Input.starts_with(":"))
|
||||
return findFromSearchPaths(Input.drop_front(), Root, SearchPaths);
|
||||
return searchLibraryBaseName(Input, Root, SearchPaths);
|
||||
}
|
||||
|
||||
void printCommands(ArrayRef<StringRef> CmdArgs) {
|
||||
if (CmdArgs.empty())
|
||||
return;
|
||||
@@ -255,49 +215,6 @@ void printCommands(ArrayRef<StringRef> CmdArgs) {
|
||||
errs() << join(std::next(CmdArgs.begin()), CmdArgs.end(), " ") << "\n";
|
||||
}
|
||||
|
||||
/// A minimum symbol interface that provides the necessary information to
|
||||
/// extract archive members and resolve LTO symbols.
|
||||
struct Symbol {
|
||||
enum Flags {
|
||||
None = 0,
|
||||
Undefined = 1 << 0,
|
||||
Weak = 1 << 1,
|
||||
};
|
||||
|
||||
Symbol() : File(), Flags(None), UsedInRegularObj(false) {}
|
||||
Symbol(Symbol::Flags Flags) : File(), Flags(Flags), UsedInRegularObj(true) {}
|
||||
|
||||
Symbol(MemoryBufferRef File, const irsymtab::Reader::SymbolRef Sym)
|
||||
: File(File), Flags(0), UsedInRegularObj(false) {
|
||||
if (Sym.isUndefined())
|
||||
Flags |= Undefined;
|
||||
if (Sym.isWeak())
|
||||
Flags |= Weak;
|
||||
}
|
||||
|
||||
Symbol(MemoryBufferRef File, const SymbolRef Sym)
|
||||
: File(File), Flags(0), UsedInRegularObj(false) {
|
||||
auto FlagsOrErr = Sym.getFlags();
|
||||
if (!FlagsOrErr)
|
||||
reportError(FlagsOrErr.takeError());
|
||||
if (*FlagsOrErr & SymbolRef::SF_Undefined)
|
||||
Flags |= Undefined;
|
||||
if (*FlagsOrErr & SymbolRef::SF_Weak)
|
||||
Flags |= Weak;
|
||||
|
||||
auto NameOrErr = Sym.getName();
|
||||
if (!NameOrErr)
|
||||
reportError(NameOrErr.takeError());
|
||||
}
|
||||
|
||||
bool isWeak() const { return Flags & Weak; }
|
||||
bool isUndefined() const { return Flags & Undefined; }
|
||||
|
||||
MemoryBufferRef File;
|
||||
uint32_t Flags;
|
||||
bool UsedInRegularObj;
|
||||
};
|
||||
|
||||
Expected<StringRef> runPTXAs(StringRef File, const ArgList &Args) {
|
||||
SmallVector<StringRef, 1> SearchPaths;
|
||||
if (Arg *A = Args.getLastArg(OPT_cuda_path_EQ))
|
||||
@@ -413,97 +330,10 @@ Expected<std::unique_ptr<lto::LTO>> createLTO(const ArgList &Args) {
|
||||
return std::make_unique<lto::LTO>(std::move(Conf), Backend, Partitions, Kind);
|
||||
}
|
||||
|
||||
Expected<bool> getSymbolsFromBitcode(MemoryBufferRef Buffer,
|
||||
StringMap<Symbol> &SymTab, bool IsLazy) {
|
||||
Expected<IRSymtabFile> IRSymtabOrErr = readIRSymtab(Buffer);
|
||||
if (!IRSymtabOrErr)
|
||||
return IRSymtabOrErr.takeError();
|
||||
bool Extracted = !IsLazy;
|
||||
StringMap<Symbol> PendingSymbols;
|
||||
for (unsigned I = 0; I != IRSymtabOrErr->Mods.size(); ++I) {
|
||||
for (const auto &IRSym : IRSymtabOrErr->TheReader.module_symbols(I)) {
|
||||
if (IRSym.isFormatSpecific() || !IRSym.isGlobal())
|
||||
continue;
|
||||
|
||||
Symbol &OldSym = !SymTab.count(IRSym.getName()) && IsLazy
|
||||
? PendingSymbols[IRSym.getName()]
|
||||
: SymTab[IRSym.getName()];
|
||||
Symbol Sym = Symbol(Buffer, IRSym);
|
||||
if (OldSym.File.getBuffer().empty())
|
||||
OldSym = Sym;
|
||||
|
||||
bool ResolvesReference =
|
||||
!Sym.isUndefined() &&
|
||||
(OldSym.isUndefined() || (OldSym.isWeak() && !Sym.isWeak())) &&
|
||||
!(OldSym.isWeak() && OldSym.isUndefined() && IsLazy);
|
||||
Extracted |= ResolvesReference;
|
||||
|
||||
Sym.UsedInRegularObj = OldSym.UsedInRegularObj;
|
||||
if (ResolvesReference)
|
||||
OldSym = Sym;
|
||||
}
|
||||
}
|
||||
if (Extracted)
|
||||
for (const auto &[Name, Symbol] : PendingSymbols)
|
||||
SymTab[Name] = Symbol;
|
||||
return Extracted;
|
||||
}
|
||||
|
||||
Expected<bool> getSymbolsFromObject(ObjectFile &ObjFile,
|
||||
StringMap<Symbol> &SymTab, bool IsLazy) {
|
||||
bool Extracted = !IsLazy;
|
||||
StringMap<Symbol> PendingSymbols;
|
||||
for (SymbolRef ObjSym : ObjFile.symbols()) {
|
||||
auto NameOrErr = ObjSym.getName();
|
||||
if (!NameOrErr)
|
||||
return NameOrErr.takeError();
|
||||
|
||||
Symbol &OldSym = !SymTab.count(*NameOrErr) && IsLazy
|
||||
? PendingSymbols[*NameOrErr]
|
||||
: SymTab[*NameOrErr];
|
||||
Symbol Sym = Symbol(ObjFile.getMemoryBufferRef(), ObjSym);
|
||||
if (OldSym.File.getBuffer().empty())
|
||||
OldSym = Sym;
|
||||
|
||||
bool ResolvesReference = OldSym.isUndefined() && !Sym.isUndefined() &&
|
||||
(!OldSym.isWeak() || !IsLazy);
|
||||
Extracted |= ResolvesReference;
|
||||
|
||||
if (ResolvesReference)
|
||||
OldSym = Sym;
|
||||
OldSym.UsedInRegularObj = true;
|
||||
}
|
||||
if (Extracted)
|
||||
for (const auto &[Name, Symbol] : PendingSymbols)
|
||||
SymTab[Name] = Symbol;
|
||||
return Extracted;
|
||||
}
|
||||
|
||||
Expected<bool> getSymbols(MemoryBufferRef Buffer, StringMap<Symbol> &SymTab,
|
||||
bool IsLazy) {
|
||||
switch (identify_magic(Buffer.getBuffer())) {
|
||||
case file_magic::bitcode: {
|
||||
return getSymbolsFromBitcode(Buffer, SymTab, IsLazy);
|
||||
}
|
||||
case file_magic::elf_relocatable: {
|
||||
Expected<std::unique_ptr<ObjectFile>> ObjFile =
|
||||
ObjectFile::createObjectFile(Buffer);
|
||||
if (!ObjFile)
|
||||
return ObjFile.takeError();
|
||||
return getSymbolsFromObject(**ObjFile, SymTab, IsLazy);
|
||||
}
|
||||
default:
|
||||
return createStringError("Unsupported file type");
|
||||
}
|
||||
}
|
||||
|
||||
Expected<SmallVector<StringRef>> getInput(const ArgList &Args) {
|
||||
SmallVector<StringRef> LibraryPaths;
|
||||
for (const opt::Arg *Arg : Args.filtered(OPT_library_path))
|
||||
LibraryPaths.push_back(Arg->getValue());
|
||||
|
||||
// Build input descriptors for the archive resolver
|
||||
SmallVector<offloading::InputDesc> InputDescs;
|
||||
bool WholeArchive = false;
|
||||
SmallVector<std::pair<std::unique_ptr<MemoryBuffer>, bool>> InputFiles;
|
||||
for (const opt::Arg *Arg : Args.filtered(
|
||||
OPT_INPUT, OPT_library, OPT_whole_archive, OPT_no_whole_archive)) {
|
||||
if (Arg->getOption().matches(OPT_whole_archive) ||
|
||||
@@ -512,84 +342,43 @@ Expected<SmallVector<StringRef>> getInput(const ArgList &Args) {
|
||||
continue;
|
||||
}
|
||||
|
||||
std::optional<std::string> Filename =
|
||||
Arg->getOption().matches(OPT_library)
|
||||
? searchLibrary(Arg->getValue(), /*Root=*/"", LibraryPaths)
|
||||
: std::string(Arg->getValue());
|
||||
|
||||
if (!Filename && Arg->getOption().matches(OPT_library))
|
||||
return createStringError("unable to find library -l%s", Arg->getValue());
|
||||
|
||||
if (!Filename || !sys::fs::exists(*Filename) ||
|
||||
sys::fs::is_directory(*Filename))
|
||||
continue;
|
||||
|
||||
ErrorOr<std::unique_ptr<MemoryBuffer>> BufferOrErr =
|
||||
MemoryBuffer::getFileOrSTDIN(*Filename);
|
||||
if (std::error_code EC = BufferOrErr.getError())
|
||||
return createFileError(*Filename, EC);
|
||||
|
||||
MemoryBufferRef Buffer = **BufferOrErr;
|
||||
switch (identify_magic(Buffer.getBuffer())) {
|
||||
case file_magic::bitcode:
|
||||
case file_magic::elf_relocatable:
|
||||
InputFiles.emplace_back(std::move(*BufferOrErr), /*IsLazy=*/false);
|
||||
break;
|
||||
case file_magic::archive: {
|
||||
Expected<std::unique_ptr<object::Archive>> LibFile =
|
||||
object::Archive::create(Buffer);
|
||||
if (!LibFile)
|
||||
return LibFile.takeError();
|
||||
Error Err = Error::success();
|
||||
for (auto Child : (*LibFile)->children(Err)) {
|
||||
auto ChildBufferOrErr = Child.getMemoryBufferRef();
|
||||
if (!ChildBufferOrErr)
|
||||
return ChildBufferOrErr.takeError();
|
||||
std::unique_ptr<MemoryBuffer> ChildBuffer =
|
||||
MemoryBuffer::getMemBufferCopy(
|
||||
ChildBufferOrErr->getBuffer(),
|
||||
ChildBufferOrErr->getBufferIdentifier());
|
||||
InputFiles.emplace_back(std::move(ChildBuffer), !WholeArchive);
|
||||
}
|
||||
if (Err)
|
||||
return Err;
|
||||
break;
|
||||
}
|
||||
default:
|
||||
return createStringError("Unsupported file type");
|
||||
}
|
||||
offloading::InputDesc Desc;
|
||||
Desc.Value = Arg->getValue();
|
||||
Desc.InputKind = Arg->getOption().matches(OPT_library)
|
||||
? offloading::InputDesc::Kind::Library
|
||||
: offloading::InputDesc::Kind::File;
|
||||
Desc.WholeArchive = WholeArchive;
|
||||
InputDescs.push_back(Desc);
|
||||
}
|
||||
|
||||
bool Extracted = true;
|
||||
StringMap<Symbol> SymTab;
|
||||
for (auto &Sym : Args.getAllArgValues(OPT_u))
|
||||
SymTab[Sym] = Symbol(Symbol::Undefined);
|
||||
SmallVector<std::unique_ptr<MemoryBuffer>> LinkerInput;
|
||||
while (Extracted) {
|
||||
Extracted = false;
|
||||
for (auto &[Input, IsLazy] : InputFiles) {
|
||||
if (!Input)
|
||||
continue;
|
||||
// Gather search paths and forced undefined symbols
|
||||
SmallVector<StringRef> LibraryPaths;
|
||||
for (const opt::Arg *Arg : Args.filtered(OPT_library_path))
|
||||
LibraryPaths.push_back(Arg->getValue());
|
||||
|
||||
if (hasFatBinary(Args, *Input)) {
|
||||
LinkerInput.emplace_back(std::move(Input));
|
||||
continue;
|
||||
}
|
||||
std::vector<std::string> ForcedUndefStorage = Args.getAllArgValues(OPT_u);
|
||||
SmallVector<StringRef> ForcedUndefs(ForcedUndefStorage.begin(),
|
||||
ForcedUndefStorage.end());
|
||||
|
||||
// Archive members only extract if they define needed symbols. We will
|
||||
// re-scan all the inputs if any files were extracted for the link job.
|
||||
Expected<bool> ExtractOrErr = getSymbols(*Input, SymTab, IsLazy);
|
||||
if (!ExtractOrErr)
|
||||
return ExtractOrErr.takeError();
|
||||
// The device code we are linking targets NVPTX. Any other ELF object is a
|
||||
// host "fat binary" that should be forwarded without symbol scanning. The
|
||||
// --assume-device-object flag (under --dry-run) overrides this and treats
|
||||
// every input as device code, so disable detection by passing no archs.
|
||||
SmallVector<Triple::ArchType> DeviceArchs;
|
||||
if (!(Args.hasArg(OPT_dry_run) && Args.hasArg(OPT_assume_device_object)))
|
||||
DeviceArchs = {Triple::nvptx, Triple::nvptx64};
|
||||
|
||||
Extracted |= *ExtractOrErr;
|
||||
if (!*ExtractOrErr)
|
||||
continue;
|
||||
// Resolve archive members.
|
||||
Expected<offloading::ResolvedInputs> ResolvedOrErr =
|
||||
offloading::resolveArchiveMembers(InputDescs, LibraryPaths, ForcedUndefs,
|
||||
"", DeviceArchs);
|
||||
if (!ResolvedOrErr)
|
||||
return ResolvedOrErr.takeError();
|
||||
|
||||
LinkerInput.emplace_back(std::move(Input));
|
||||
}
|
||||
}
|
||||
InputFiles.clear();
|
||||
offloading::ResolvedInputs &Resolved = *ResolvedOrErr;
|
||||
SmallVector<std::unique_ptr<MemoryBuffer>> LinkerInput =
|
||||
std::move(Resolved.Buffers);
|
||||
StringMap<offloading::Symbol> &SymTab = Resolved.SymTab;
|
||||
|
||||
// Extract any bitcode files to be passed to the LTO pipeline.
|
||||
SmallVector<std::unique_ptr<MemoryBuffer>> BitcodeFiles;
|
||||
@@ -616,7 +405,7 @@ Expected<SmallVector<StringRef>> getInput(const ArgList &Args) {
|
||||
size_t Idx = 0;
|
||||
for (auto &Sym : Symbols) {
|
||||
lto::SymbolResolution &Res = Resolutions[Idx++];
|
||||
Symbol ObjSym = SymTab[Sym.getName()];
|
||||
offloading::Symbol ObjSym = SymTab[Sym.getName()];
|
||||
// We will use this as the prevailing symbol in LTO if it is not
|
||||
// undefined and it is from the file that contained the canonical
|
||||
// definition.
|
||||
|
||||
@@ -18,11 +18,13 @@
|
||||
#include "clang/Basic/OffloadArch.h"
|
||||
#include "clang/Basic/Version.h"
|
||||
|
||||
#include "llvm/ADT/STLExtras.h"
|
||||
#include "llvm/ADT/StringExtras.h"
|
||||
#include "llvm/ADT/StringSwitch.h"
|
||||
#include "llvm/BinaryFormat/Magic.h"
|
||||
#include "llvm/Bitcode/BitcodeWriter.h"
|
||||
#include "llvm/CodeGen/CommandFlags.h"
|
||||
#include "llvm/Frontend/Offloading/ArchiveLinker.h"
|
||||
#include "llvm/Frontend/Offloading/Utility.h"
|
||||
#include "llvm/IR/DiagnosticPrinter.h"
|
||||
#include "llvm/IR/LLVMContext.h"
|
||||
@@ -187,84 +189,49 @@ static Error executeCommands(StringRef ExecutablePath,
|
||||
return Error::success();
|
||||
}
|
||||
|
||||
static Expected<SmallVector<std::string>> getInput(const ArgList &Args) {
|
||||
// Collect all input bitcode files to be passed to the linking stage.
|
||||
SmallVector<std::string> BitcodeFiles;
|
||||
auto Inputs = Args.filtered(OPT_INPUT);
|
||||
if (Inputs.empty())
|
||||
static Expected<SmallVector<std::unique_ptr<MemoryBuffer>>>
|
||||
getInput(const ArgList &Args) {
|
||||
// Build input descriptors for the shared archive resolver
|
||||
SmallVector<offloading::InputDesc> InputDescs;
|
||||
bool WholeArchive = false;
|
||||
for (const opt::Arg *Arg : Args.filtered(
|
||||
OPT_INPUT, OPT_library, OPT_whole_archive, OPT_no_whole_archive)) {
|
||||
if (Arg->getOption().matches(OPT_whole_archive) ||
|
||||
Arg->getOption().matches(OPT_no_whole_archive)) {
|
||||
WholeArchive = Arg->getOption().matches(OPT_whole_archive);
|
||||
continue;
|
||||
}
|
||||
|
||||
offloading::InputDesc Desc;
|
||||
Desc.Value = Arg->getValue();
|
||||
Desc.InputKind = Arg->getOption().matches(OPT_library)
|
||||
? offloading::InputDesc::Kind::Library
|
||||
: offloading::InputDesc::Kind::File;
|
||||
Desc.WholeArchive = WholeArchive;
|
||||
InputDescs.push_back(Desc);
|
||||
}
|
||||
|
||||
if (InputDescs.empty())
|
||||
return createStringError("No input files provided");
|
||||
for (const opt::Arg *Arg : Inputs) {
|
||||
StringRef Filename = Arg->getValue();
|
||||
if (!sys::fs::exists(Filename) || sys::fs::is_directory(Filename))
|
||||
return createStringError("Input file '" + Filename + "' does not exist");
|
||||
file_magic Magic;
|
||||
if (auto EC = identify_magic(Filename, Magic))
|
||||
return createStringError("Failed to open file " + Filename);
|
||||
// TODO: Current use case involves LLVM IR bitcode files as input.
|
||||
// This will be extended to support SPIR-V IR files.
|
||||
if (Magic != file_magic::bitcode)
|
||||
return createStringError("Unsupported file type for '" + Filename + "'");
|
||||
BitcodeFiles.push_back(std::string(Filename));
|
||||
}
|
||||
return BitcodeFiles;
|
||||
}
|
||||
|
||||
/// Handle cases where input file is a LLVM IR bitcode file.
|
||||
/// When clang-sycl-linker is called via clang-linker-wrapper tool, input files
|
||||
/// are LLVM IR bitcode files.
|
||||
// TODO: Support SPIR-V IR files.
|
||||
static Expected<std::unique_ptr<Module>> getBitcodeModule(StringRef File,
|
||||
LLVMContext &C) {
|
||||
SMDiagnostic Err;
|
||||
|
||||
auto M = getLazyIRFileModule(File, Err, C);
|
||||
if (M)
|
||||
return std::move(M);
|
||||
return createStringError(Err.getMessage());
|
||||
}
|
||||
|
||||
static std::optional<std::string> findFile(StringRef Dir, const Twine &Name) {
|
||||
SmallString<128> Path(Dir);
|
||||
llvm::sys::path::append(Path, Name);
|
||||
if (sys::fs::exists(Path) && !sys::fs::is_directory(Path))
|
||||
return std::string(Path);
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
static std::optional<std::string>
|
||||
searchLibrary(StringRef Name, ArrayRef<StringRef> SearchPaths) {
|
||||
// An absolute path is taken as-is; -L paths are only consulted for relative
|
||||
// names.
|
||||
if (sys::path::is_absolute(Name)) {
|
||||
if (sys::fs::exists(Name) && !sys::fs::is_directory(Name))
|
||||
return std::string(Name);
|
||||
return std::nullopt;
|
||||
}
|
||||
for (StringRef Dir : SearchPaths)
|
||||
if (std::optional<std::string> File = findFile(Dir, Name))
|
||||
return File;
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
/// Gather all library files. The list of files and its location are passed from
|
||||
/// driver.
|
||||
static Expected<SmallVector<std::string>>
|
||||
getBCLibraryNames(const ArgList &Args) {
|
||||
// Gather search paths and forced undefined symbols
|
||||
SmallVector<StringRef> LibraryPaths;
|
||||
for (const opt::Arg *Arg : Args.filtered(OPT_library_path))
|
||||
LibraryPaths.push_back(Arg->getValue());
|
||||
|
||||
SmallVector<std::string> LibraryFiles;
|
||||
for (const opt::Arg *Arg : Args.filtered(OPT_bc_library)) {
|
||||
std::optional<std::string> LibName =
|
||||
searchLibrary(Arg->getValue(), LibraryPaths);
|
||||
if (!LibName)
|
||||
return createStringError("'" + Twine(Arg->getValue()) +
|
||||
"' library file not found");
|
||||
LibraryFiles.push_back(std::move(*LibName));
|
||||
}
|
||||
std::vector<std::string> ForcedUndefStorage = Args.getAllArgValues(OPT_u);
|
||||
SmallVector<StringRef> ForcedUndefs(ForcedUndefStorage.begin(),
|
||||
ForcedUndefStorage.end());
|
||||
|
||||
return LibraryFiles;
|
||||
Expected<offloading::ResolvedInputs> ResolvedOrErr =
|
||||
offloading::resolveArchiveMembers(InputDescs, LibraryPaths, ForcedUndefs);
|
||||
if (!ResolvedOrErr)
|
||||
return ResolvedOrErr.takeError();
|
||||
|
||||
if (ResolvedOrErr->Buffers.empty())
|
||||
return createStringError("No input files could be resolved");
|
||||
|
||||
return std::move(ResolvedOrErr->Buffers);
|
||||
}
|
||||
|
||||
namespace {
|
||||
@@ -280,19 +247,12 @@ struct LinkResult {
|
||||
/// first input that supplies a triple as canonical. Issue an error if any
|
||||
/// triple inputs disagree.
|
||||
/// 2. Link all input bitcode images into one image using the linkInModule API.
|
||||
/// 3. Gather all library bitcode images.
|
||||
/// 4. Link all the images gathered in Step 3 with the output of Step 2 using
|
||||
/// linkInModule API. LinkOnlyNeeded flag is used.
|
||||
static Expected<LinkResult> linkInputs(ArrayRef<std::string> InputFiles,
|
||||
const ArgList &Args, LLVMContext &C) {
|
||||
static Expected<LinkResult>
|
||||
linkInputs(ArrayRef<std::unique_ptr<MemoryBuffer>> InputBuffers,
|
||||
const ArgList &Args, LLVMContext &C) {
|
||||
llvm::TimeTraceScope TimeScope("Link code");
|
||||
|
||||
assert(InputFiles.size() && "No inputs to link");
|
||||
|
||||
// Get all library files.
|
||||
Expected<SmallVector<std::string>> BCLibFiles = getBCLibraryNames(Args);
|
||||
if (!BCLibFiles)
|
||||
return BCLibFiles.takeError();
|
||||
assert(InputBuffers.size() && "No inputs to link");
|
||||
|
||||
// Create a new file to write the linked file to.
|
||||
auto BitcodeOutput =
|
||||
@@ -301,11 +261,12 @@ static Expected<LinkResult> linkInputs(ArrayRef<std::string> InputFiles,
|
||||
return BitcodeOutput.takeError();
|
||||
|
||||
if (Verbose) {
|
||||
std::string Inputs = llvm::join(InputFiles.begin(), InputFiles.end(), ", ");
|
||||
std::string LibInputs =
|
||||
llvm::join((*BCLibFiles).begin(), (*BCLibFiles).end(), ", ");
|
||||
errs() << formatv("link: inputs: {0} libfiles: {1} output: {2}\n", Inputs,
|
||||
LibInputs, *BitcodeOutput);
|
||||
std::string Inputs = llvm::join(
|
||||
llvm::map_range(InputBuffers,
|
||||
[](const auto &B) { return B->getBufferIdentifier(); }),
|
||||
", ");
|
||||
errs() << formatv("link: inputs: {0} output: {1}\n", Inputs,
|
||||
*BitcodeOutput);
|
||||
}
|
||||
|
||||
// Link input files. Resolve the target triple.
|
||||
@@ -314,8 +275,14 @@ static Expected<LinkResult> linkInputs(ArrayRef<std::string> InputFiles,
|
||||
auto LinkerOutput = std::make_unique<Module>("linker-output", C);
|
||||
Linker L(*LinkerOutput);
|
||||
|
||||
for (auto &File : InputFiles) {
|
||||
auto ModOrErr = getBitcodeModule(File, C);
|
||||
for (const auto &Buffer : InputBuffers) {
|
||||
// Check file type before attempting to parse as bitcode
|
||||
file_magic Magic = identify_magic(Buffer->getBuffer());
|
||||
if (Magic != file_magic::bitcode)
|
||||
return createStringError("Unsupported file type: '" +
|
||||
Buffer->getBufferIdentifier() + "'");
|
||||
|
||||
auto ModOrErr = parseBitcodeFile(Buffer->getMemBufferRef(), C);
|
||||
if (!ModOrErr)
|
||||
return ModOrErr.takeError();
|
||||
|
||||
@@ -323,11 +290,12 @@ static Expected<LinkResult> linkInputs(ArrayRef<std::string> InputFiles,
|
||||
if (!T.empty() && T != TargetTriple) {
|
||||
if (TargetTriple.empty()) {
|
||||
TargetTriple = T;
|
||||
TripleSource = File;
|
||||
TripleSource = Buffer->getBufferIdentifier();
|
||||
} else {
|
||||
return createStringError(
|
||||
"conflicting target triples: '" + TargetTriple.str() + "' (from " +
|
||||
TripleSource + ") vs '" + T.str() + "' (from " + File + ")");
|
||||
TripleSource + ") vs '" + T.str() + "' (from " +
|
||||
Buffer->getBufferIdentifier() + ")");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -339,18 +307,6 @@ static Expected<LinkResult> linkInputs(ArrayRef<std::string> InputFiles,
|
||||
return createStringError(
|
||||
"Target triple must be specified or inferable from inputs");
|
||||
|
||||
// Link in library files.
|
||||
for (auto &File : *BCLibFiles) {
|
||||
auto LibMod = getBitcodeModule(File, C);
|
||||
if (!LibMod)
|
||||
return LibMod.takeError();
|
||||
if ((*LibMod)->getTargetTriple() == TargetTriple) {
|
||||
unsigned Flags = Linker::Flags::LinkOnlyNeeded;
|
||||
if (L.linkInModule(std::move(*LibMod), Flags))
|
||||
return createStringError("Could not link IR");
|
||||
}
|
||||
}
|
||||
|
||||
// Dump linked output for testing.
|
||||
if (Args.hasArg(OPT_print_linked_module))
|
||||
outs() << *LinkerOutput;
|
||||
@@ -693,13 +649,14 @@ static bool canSkipModuleSplit(IRSplitMode Mode, const Module &M,
|
||||
/// 4. Optionally run AOT compilation when targeting an Intel HW arch.
|
||||
/// 5. Pack the resulting images into a single OffloadBinary written to the
|
||||
/// output file.
|
||||
static Error runSYCLLink(ArrayRef<std::string> Files, const ArgList &Args) {
|
||||
static Error runSYCLLink(ArrayRef<std::unique_ptr<MemoryBuffer>> Buffers,
|
||||
const ArgList &Args) {
|
||||
llvm::TimeTraceScope TimeScope("SYCL linking");
|
||||
|
||||
LLVMContext C;
|
||||
|
||||
// Link all input bitcode files and library files.
|
||||
Expected<LinkResult> LinkedOrErr = linkInputs(Files, Args, C);
|
||||
Expected<LinkResult> LinkedOrErr = linkInputs(Buffers, Args, C);
|
||||
if (!LinkedOrErr)
|
||||
return LinkedOrErr.takeError();
|
||||
LinkResult &Result = *LinkedOrErr;
|
||||
@@ -849,10 +806,10 @@ int main(int argc, char **argv) {
|
||||
reportError(createStringError("Output file must be specified"));
|
||||
OutputFile = Args.getLastArgValue(OPT_o);
|
||||
|
||||
// Get the input files to pass to the linking stage.
|
||||
auto FilesOrErr = getInput(Args);
|
||||
if (!FilesOrErr)
|
||||
reportError(FilesOrErr.takeError());
|
||||
// Get the input buffers to pass to the linking stage.
|
||||
auto BuffersOrErr = getInput(Args);
|
||||
if (!BuffersOrErr)
|
||||
reportError(BuffersOrErr.takeError());
|
||||
|
||||
if (auto *A = Args.getLastArg(OPT_spirv_dump_device_code_EQ)) {
|
||||
StringRef V = A->getValue();
|
||||
@@ -871,7 +828,7 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
|
||||
// Run SYCL linking process on the generated inputs.
|
||||
if (Error Err = runSYCLLink(*FilesOrErr, Args))
|
||||
if (Error Err = runSYCLLink(*BuffersOrErr, Args))
|
||||
reportError(std::move(Err));
|
||||
|
||||
// Remove the temporary files created.
|
||||
|
||||
@@ -24,12 +24,21 @@ def library_path_S : Separate<["--", "-"], "library-path">, Flags<[HelpHidden]>,
|
||||
def library_path_EQ : Joined<["--", "-"], "library-path=">, Flags<[HelpHidden]>,
|
||||
Alias<library_path>;
|
||||
|
||||
def bc_library : Separate<["--", "-"], "bc-library">, MetaVarName<"<name>">,
|
||||
HelpText<"Add LLVM bitcode library <name> (with extension) to the link. A "
|
||||
"relative <name> is resolved against -L paths; an absolute path is "
|
||||
"taken as-is.">;
|
||||
def bc_library_EQ : Joined<["--", "-"], "bc-library=">, Flags<[HelpHidden]>,
|
||||
Alias<bc_library>;
|
||||
def library : JoinedOrSeparate<["-"], "l">, MetaVarName<"<libname>">,
|
||||
HelpText<"Search for library <libname>">;
|
||||
def library_S : Separate<["--", "-"], "library">, Flags<[HelpHidden]>,
|
||||
Alias<library>;
|
||||
def library_EQ : Joined<["--", "-"], "library=">, Flags<[HelpHidden]>,
|
||||
Alias<library>;
|
||||
|
||||
def whole_archive : Flag<["--", "-"], "whole-archive">,
|
||||
HelpText<"Include all archive members in the link">;
|
||||
def no_whole_archive : Flag<["--", "-"], "no-whole-archive">,
|
||||
HelpText<"Only include archive members that resolve undefined symbols (default)">;
|
||||
|
||||
def u : JoinedOrSeparate<["-"], "u">, MetaVarName<"<symbol>">,
|
||||
HelpText<"Force undefined symbol during linking">;
|
||||
def undefined : JoinedOrSeparate<["--"], "undefined">, Alias<u>;
|
||||
|
||||
def arch_EQ : Joined<["--", "-"], "arch=">,
|
||||
Flags<[LinkerOnlyOption]>,
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
//===----------------------------------------------------------------------===//
|
||||
//
|
||||
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
||||
// See https://llvm.org/LICENSE.txt for license information.
|
||||
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
||||
//
|
||||
//===----------------------------------------------------------------------===//
|
||||
//
|
||||
// This file declares shared functionality for linking static libraries
|
||||
// (archives) in offloading tools. It provides a symbol-driven fixed-point
|
||||
// archive member selection algorithm used by both clang-nvlink-wrapper and
|
||||
// clang-sycl-linker.
|
||||
//
|
||||
//===----------------------------------------------------------------------===//
|
||||
|
||||
#ifndef LLVM_FRONTEND_OFFLOADING_ARCHIVELINKER_H
|
||||
#define LLVM_FRONTEND_OFFLOADING_ARCHIVELINKER_H
|
||||
|
||||
#include "llvm/ADT/ArrayRef.h"
|
||||
#include "llvm/ADT/SmallVector.h"
|
||||
#include "llvm/ADT/StringMap.h"
|
||||
#include "llvm/ADT/StringRef.h"
|
||||
#include "llvm/Object/IRSymtab.h"
|
||||
#include "llvm/Object/SymbolicFile.h"
|
||||
#include "llvm/Support/Error.h"
|
||||
#include "llvm/Support/MemoryBufferRef.h"
|
||||
#include "llvm/TargetParser/Triple.h"
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
|
||||
namespace llvm {
|
||||
class MemoryBuffer;
|
||||
|
||||
namespace object {
|
||||
class SymbolRef;
|
||||
} // namespace object
|
||||
|
||||
namespace offloading {
|
||||
|
||||
/// A minimum symbol interface that provides the necessary information to
|
||||
/// extract archive members and resolve LTO symbols.
|
||||
struct Symbol {
|
||||
enum Flags {
|
||||
None = 0,
|
||||
Undefined = 1 << 0,
|
||||
Weak = 1 << 1,
|
||||
};
|
||||
|
||||
Symbol() : File(), SymFlags(None), UsedInRegularObj(false) {}
|
||||
Symbol(Symbol::Flags F) : File(), SymFlags(F), UsedInRegularObj(true) {}
|
||||
|
||||
Symbol(MemoryBufferRef File, const irsymtab::Reader::SymbolRef Sym)
|
||||
: File(File), SymFlags(0), UsedInRegularObj(false) {
|
||||
if (Sym.isUndefined())
|
||||
SymFlags |= Undefined;
|
||||
if (Sym.isWeak())
|
||||
SymFlags |= Weak;
|
||||
}
|
||||
|
||||
/// Create a Symbol from an object file symbol reference.
|
||||
/// Returns an error if symbol flags cannot be retrieved.
|
||||
static Expected<Symbol> createFromObject(MemoryBufferRef File,
|
||||
const object::SymbolRef &Sym);
|
||||
|
||||
bool isWeak() const { return SymFlags & Weak; }
|
||||
bool isUndefined() const { return SymFlags & Undefined; }
|
||||
|
||||
MemoryBufferRef File;
|
||||
uint32_t SymFlags;
|
||||
bool UsedInRegularObj;
|
||||
};
|
||||
|
||||
/// Description of a single input (file or library).
|
||||
struct InputDesc {
|
||||
enum class Kind { File, Library };
|
||||
|
||||
StringRef Value; // File path, or library name for -l (the value after -l).
|
||||
Kind InputKind = Kind::File;
|
||||
bool WholeArchive = false; // --whole-archive state in effect at this input.
|
||||
};
|
||||
|
||||
/// Result of archive member resolution.
|
||||
struct ResolvedInputs {
|
||||
SmallVector<std::unique_ptr<MemoryBuffer>>
|
||||
Buffers; // Members to link, in order.
|
||||
StringMap<Symbol> SymTab; // Symbol table (for LTO resolution).
|
||||
};
|
||||
|
||||
/// Resolve archive members from the given inputs using a symbol-driven
|
||||
/// fixed-point algorithm. For each input:
|
||||
/// - If it's a Library, search for lib<name>.a or :<name> in SearchPaths
|
||||
/// - If it's a File, use the path directly
|
||||
/// - Archives are expanded and members are lazily extracted based on symbol
|
||||
/// references unless WholeArchive is true
|
||||
/// - Non-archive inputs (bitcode, ELF objects) are always included
|
||||
///
|
||||
/// Returns the buffers to link and the symbol table for LTO resolution.
|
||||
///
|
||||
/// \param Order Positional inputs + -l libraries in order.
|
||||
/// \param SearchPaths -L paths for library search.
|
||||
/// \param ForcedUndefs -u symbols (may be empty).
|
||||
/// \param Root Sysroot for "=" prefixed paths ("" if none).
|
||||
/// \param DeviceArchs Architectures of the device code being linked. When
|
||||
/// non-empty, any ELF input whose architecture is not in this list (or
|
||||
/// which cannot be parsed as an object) is treated as a "fat binary"
|
||||
/// and passed through without symbol scanning (e.g., nvlink's cubin
|
||||
/// detection). When empty, all inputs are scanned normally.
|
||||
Expected<ResolvedInputs> resolveArchiveMembers(
|
||||
ArrayRef<InputDesc> Order, ArrayRef<StringRef> SearchPaths,
|
||||
ArrayRef<StringRef> ForcedUndefs = {}, StringRef Root = "",
|
||||
ArrayRef<Triple::ArchType> DeviceArchs = {});
|
||||
|
||||
} // namespace offloading
|
||||
} // namespace llvm
|
||||
|
||||
#endif // LLVM_FRONTEND_OFFLOADING_ARCHIVELINKER_H
|
||||
@@ -0,0 +1,301 @@
|
||||
//===----------------------------------------------------------------------===//
|
||||
//
|
||||
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
||||
// See https://llvm.org/LICENSE.txt for license information.
|
||||
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
||||
//
|
||||
//===----------------------------------------------------------------------===//
|
||||
//
|
||||
// This file implements shared functionality for linking static libraries
|
||||
// (archives) in offloading tools.
|
||||
//
|
||||
//===----------------------------------------------------------------------===//
|
||||
|
||||
#include "llvm/Frontend/Offloading/ArchiveLinker.h"
|
||||
#include "llvm/ADT/STLExtras.h"
|
||||
#include "llvm/BinaryFormat/Magic.h"
|
||||
#include "llvm/Object/Archive.h"
|
||||
#include "llvm/Object/IRObjectFile.h"
|
||||
#include "llvm/Object/ObjectFile.h"
|
||||
#include "llvm/Support/Error.h"
|
||||
#include "llvm/Support/FileSystem.h"
|
||||
#include "llvm/Support/MemoryBuffer.h"
|
||||
#include "llvm/Support/Path.h"
|
||||
#include <optional>
|
||||
#include <string>
|
||||
|
||||
using namespace llvm;
|
||||
using namespace llvm::object;
|
||||
|
||||
namespace llvm {
|
||||
namespace offloading {
|
||||
|
||||
Expected<Symbol> Symbol::createFromObject(MemoryBufferRef File,
|
||||
const SymbolRef &Sym) {
|
||||
Symbol Result;
|
||||
Result.File = File;
|
||||
|
||||
auto FlagsOrErr = Sym.getFlags();
|
||||
if (!FlagsOrErr)
|
||||
return FlagsOrErr.takeError();
|
||||
|
||||
if (*FlagsOrErr & SymbolRef::SF_Undefined)
|
||||
Result.SymFlags |= Undefined;
|
||||
if (*FlagsOrErr & SymbolRef::SF_Weak)
|
||||
Result.SymFlags |= Weak;
|
||||
|
||||
return Result;
|
||||
}
|
||||
|
||||
static std::optional<std::string> findFile(StringRef Dir, StringRef Root,
|
||||
const Twine &Name) {
|
||||
SmallString<128> Path;
|
||||
if (Dir.starts_with("="))
|
||||
sys::path::append(Path, Root, Dir.substr(1), Name);
|
||||
else
|
||||
sys::path::append(Path, Dir, Name);
|
||||
|
||||
if (sys::fs::exists(Path))
|
||||
return static_cast<std::string>(Path);
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
static std::optional<std::string>
|
||||
findFromSearchPaths(StringRef Name, StringRef Root,
|
||||
ArrayRef<StringRef> SearchPaths) {
|
||||
for (StringRef Dir : SearchPaths)
|
||||
if (std::optional<std::string> File = findFile(Dir, Root, Name))
|
||||
return File;
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
/// Search for static libraries in the linker's library path given input like
|
||||
/// `-lfoo` or `-l:libfoo.a`.
|
||||
static std::optional<std::string>
|
||||
searchLibrary(StringRef Input, StringRef Root,
|
||||
ArrayRef<StringRef> SearchPaths) {
|
||||
if (Input.starts_with(":"))
|
||||
return findFromSearchPaths(Input.drop_front(), Root, SearchPaths);
|
||||
SmallString<128> LibName;
|
||||
("lib" + Input + ".a").toVector(LibName);
|
||||
return findFromSearchPaths(LibName, Root, SearchPaths);
|
||||
}
|
||||
|
||||
static Expected<bool> getSymbolsFromBitcode(MemoryBufferRef Buffer,
|
||||
StringMap<Symbol> &SymTab,
|
||||
bool IsLazy) {
|
||||
Expected<IRSymtabFile> IRSymtabOrErr = readIRSymtab(Buffer);
|
||||
if (!IRSymtabOrErr)
|
||||
return IRSymtabOrErr.takeError();
|
||||
bool Extracted = !IsLazy;
|
||||
StringMap<Symbol> PendingSymbols;
|
||||
for (unsigned I = 0; I != IRSymtabOrErr->Mods.size(); ++I) {
|
||||
for (const auto &IRSym : IRSymtabOrErr->TheReader.module_symbols(I)) {
|
||||
if (IRSym.isFormatSpecific() || !IRSym.isGlobal())
|
||||
continue;
|
||||
|
||||
StringMap<Symbol> &Target =
|
||||
(IsLazy && !SymTab.count(IRSym.getName())) ? PendingSymbols : SymTab;
|
||||
Symbol &OldSym = Target[IRSym.getName()];
|
||||
Symbol Sym = Symbol(Buffer, IRSym);
|
||||
if (OldSym.SymFlags == Symbol::None)
|
||||
OldSym = Sym;
|
||||
|
||||
bool ResolvesReference =
|
||||
!Sym.isUndefined() &&
|
||||
(OldSym.isUndefined() || (OldSym.isWeak() && !Sym.isWeak())) &&
|
||||
!(OldSym.isWeak() && OldSym.isUndefined() && IsLazy);
|
||||
Extracted |= ResolvesReference;
|
||||
|
||||
Sym.UsedInRegularObj = OldSym.UsedInRegularObj;
|
||||
if (ResolvesReference)
|
||||
OldSym = Sym;
|
||||
}
|
||||
}
|
||||
if (Extracted)
|
||||
for (const auto &[Name, Symbol] : PendingSymbols)
|
||||
SymTab[Name] = Symbol;
|
||||
return Extracted;
|
||||
}
|
||||
|
||||
static Expected<bool> getSymbolsFromObject(ObjectFile &ObjFile,
|
||||
StringMap<Symbol> &SymTab,
|
||||
bool IsLazy) {
|
||||
bool Extracted = !IsLazy;
|
||||
StringMap<Symbol> PendingSymbols;
|
||||
for (SymbolRef ObjSym : ObjFile.symbols()) {
|
||||
auto NameOrErr = ObjSym.getName();
|
||||
if (!NameOrErr)
|
||||
return NameOrErr.takeError();
|
||||
|
||||
StringMap<Symbol> &Target =
|
||||
(IsLazy && !SymTab.count(*NameOrErr)) ? PendingSymbols : SymTab;
|
||||
Symbol &OldSym = Target[*NameOrErr];
|
||||
|
||||
auto SymOrErr =
|
||||
Symbol::createFromObject(ObjFile.getMemoryBufferRef(), ObjSym);
|
||||
if (!SymOrErr)
|
||||
return SymOrErr.takeError();
|
||||
Symbol Sym = *SymOrErr;
|
||||
|
||||
if (OldSym.SymFlags == Symbol::None)
|
||||
OldSym = Sym;
|
||||
|
||||
bool ResolvesReference = OldSym.isUndefined() && !Sym.isUndefined() &&
|
||||
(!OldSym.isWeak() || !IsLazy);
|
||||
Extracted |= ResolvesReference;
|
||||
|
||||
if (ResolvesReference)
|
||||
OldSym = Sym;
|
||||
OldSym.UsedInRegularObj = true;
|
||||
}
|
||||
if (Extracted)
|
||||
for (const auto &[Name, Symbol] : PendingSymbols)
|
||||
SymTab[Name] = Symbol;
|
||||
return Extracted;
|
||||
}
|
||||
|
||||
/// Identify "fat binary" inputs that should be passed through to the linker
|
||||
/// without symbol-driven extraction. An input is a fat binary if \p DeviceArchs
|
||||
/// is non-empty and the input is an ELF object whose architecture is not one of
|
||||
/// the device architectures (or which fails to parse as an object file).
|
||||
static bool isFatBinary(MemoryBufferRef Buffer,
|
||||
ArrayRef<Triple::ArchType> DeviceArchs) {
|
||||
if (DeviceArchs.empty())
|
||||
return false;
|
||||
if (identify_magic(Buffer.getBuffer()) != file_magic::elf_relocatable)
|
||||
return false;
|
||||
Expected<std::unique_ptr<ObjectFile>> ObjFile =
|
||||
ObjectFile::createObjectFile(Buffer);
|
||||
if (!ObjFile) {
|
||||
// Assume fat binary if the object creation fails.
|
||||
consumeError(ObjFile.takeError());
|
||||
return true;
|
||||
}
|
||||
return !llvm::is_contained(DeviceArchs, (*ObjFile)->getArch());
|
||||
}
|
||||
|
||||
static Expected<bool> getSymbols(MemoryBufferRef Buffer,
|
||||
StringMap<Symbol> &SymTab, bool IsLazy) {
|
||||
switch (identify_magic(Buffer.getBuffer())) {
|
||||
case file_magic::bitcode: {
|
||||
return getSymbolsFromBitcode(Buffer, SymTab, IsLazy);
|
||||
}
|
||||
case file_magic::elf_relocatable: {
|
||||
Expected<std::unique_ptr<ObjectFile>> ObjFile =
|
||||
ObjectFile::createObjectFile(Buffer);
|
||||
if (!ObjFile)
|
||||
return ObjFile.takeError();
|
||||
return getSymbolsFromObject(**ObjFile, SymTab, IsLazy);
|
||||
}
|
||||
default:
|
||||
return createStringError("Unsupported file type: '" +
|
||||
Buffer.getBufferIdentifier() + "'");
|
||||
}
|
||||
}
|
||||
|
||||
Expected<ResolvedInputs>
|
||||
resolveArchiveMembers(ArrayRef<InputDesc> Order,
|
||||
ArrayRef<StringRef> SearchPaths,
|
||||
ArrayRef<StringRef> ForcedUndefs, StringRef Root,
|
||||
ArrayRef<Triple::ArchType> DeviceArchs) {
|
||||
ResolvedInputs Result;
|
||||
SmallVector<std::pair<std::unique_ptr<MemoryBuffer>, bool>> InputFiles;
|
||||
|
||||
// Process each input descriptor.
|
||||
for (const InputDesc &Desc : Order) {
|
||||
std::optional<std::string> Filename;
|
||||
|
||||
if (Desc.InputKind == InputDesc::Kind::Library) {
|
||||
Filename = searchLibrary(Desc.Value, Root, SearchPaths);
|
||||
if (!Filename)
|
||||
return createStringError("unable to find library -l%s",
|
||||
Desc.Value.str().c_str());
|
||||
if (sys::fs::is_directory(*Filename))
|
||||
return createStringError("'%s': Is a directory", Filename->c_str());
|
||||
} else {
|
||||
if (!sys::fs::exists(Desc.Value))
|
||||
return createStringError("input file not found: '" + Desc.Value + "'");
|
||||
if (sys::fs::is_directory(Desc.Value))
|
||||
return createStringError("'" + Desc.Value + "': Is a directory");
|
||||
Filename = Desc.Value.str();
|
||||
}
|
||||
|
||||
if (!Filename)
|
||||
continue;
|
||||
|
||||
auto BufferOrErr =
|
||||
errorOrToExpected(MemoryBuffer::getFileOrSTDIN(*Filename));
|
||||
if (!BufferOrErr)
|
||||
return createFileError(*Filename, BufferOrErr.takeError());
|
||||
|
||||
MemoryBufferRef Buffer = (*BufferOrErr)->getMemBufferRef();
|
||||
switch (identify_magic(Buffer.getBuffer())) {
|
||||
case file_magic::bitcode:
|
||||
case file_magic::elf_relocatable:
|
||||
InputFiles.emplace_back(std::move(*BufferOrErr), /*IsLazy=*/false);
|
||||
break;
|
||||
case file_magic::archive: {
|
||||
Expected<std::unique_ptr<object::Archive>> LibFile =
|
||||
object::Archive::create(Buffer);
|
||||
if (!LibFile)
|
||||
return LibFile.takeError();
|
||||
Error Err = Error::success();
|
||||
for (auto Child : (*LibFile)->children(Err)) {
|
||||
auto ChildBufferOrErr = Child.getMemoryBufferRef();
|
||||
if (!ChildBufferOrErr)
|
||||
return ChildBufferOrErr.takeError();
|
||||
// Include archive name in buffer identifier for better diagnostics.
|
||||
std::string BufferIdentifier =
|
||||
(*Filename + "(" + ChildBufferOrErr->getBufferIdentifier() + ")")
|
||||
.str();
|
||||
std::unique_ptr<MemoryBuffer> ChildBuffer =
|
||||
MemoryBuffer::getMemBufferCopy(ChildBufferOrErr->getBuffer(),
|
||||
BufferIdentifier);
|
||||
InputFiles.emplace_back(std::move(ChildBuffer), !Desc.WholeArchive);
|
||||
}
|
||||
if (Err)
|
||||
return Err;
|
||||
break;
|
||||
}
|
||||
default:
|
||||
return createStringError("Unsupported file type: '" + *Filename + "'");
|
||||
}
|
||||
}
|
||||
|
||||
// Seed symbol table with forced undefined symbols.
|
||||
for (StringRef Sym : ForcedUndefs)
|
||||
Result.SymTab[Sym] = Symbol(Symbol::Undefined);
|
||||
|
||||
// Fixed-point loop to extract archive members.
|
||||
bool Extracted = true;
|
||||
while (Extracted) {
|
||||
Extracted = false;
|
||||
for (auto &[Input, IsLazy] : InputFiles) {
|
||||
if (!Input)
|
||||
continue;
|
||||
|
||||
// Check if this is a fat binary that should be passed through.
|
||||
if (isFatBinary(*Input, DeviceArchs)) {
|
||||
Result.Buffers.emplace_back(std::move(Input));
|
||||
continue;
|
||||
}
|
||||
|
||||
// Archive members only extract if they define needed symbols.
|
||||
Expected<bool> ExtractOrErr = getSymbols(*Input, Result.SymTab, IsLazy);
|
||||
if (!ExtractOrErr)
|
||||
return ExtractOrErr.takeError();
|
||||
|
||||
Extracted |= *ExtractOrErr;
|
||||
if (!*ExtractOrErr)
|
||||
continue;
|
||||
|
||||
Result.Buffers.emplace_back(std::move(Input));
|
||||
}
|
||||
}
|
||||
|
||||
return Result;
|
||||
}
|
||||
|
||||
} // namespace offloading
|
||||
} // namespace llvm
|
||||
@@ -1,4 +1,5 @@
|
||||
add_llvm_component_library(LLVMFrontendOffloading
|
||||
ArchiveLinker.cpp
|
||||
Utility.cpp
|
||||
OffloadWrapper.cpp
|
||||
PropertySet.cpp
|
||||
|
||||
Reference in new issue
Block a user