Fix inline_memset_arm_low_end to use the splatted 32-bit fill value for
all byte stores. This ensures that all stores, including unaligned and
trailing byte fills, write the requested repeated byte value on
big-endian ARM.
The low-end ARM memset path already builds value32 as value 0x01010101U
for aligned word/block stores, but its byte-wise prefix and tail
handling still passed the raw 8-bit value * widened to uint32_t. Use
value32 for the byte-wise alignment prefix and final tail loop as well.
This makes the low-end path endian-safe and consistent with
inline_memset_arm_mid_end, fixing incorrect NUL/truncated output in
printf-family formatting and direct memset tests on big-endian ARM
no-unaligned-access targets.
- Add more fine-grained constexpr annotations.
- Apply fine-grained constexpr annotations to `src/__support`,
`src/__support/FPUtil` folder.
- Remove or update constexpr annotations for functions in
`src/__support/math` folder.
- Fix or ignore old gcc's warnings.
Assisted-by: Gemini 3.1 Flash with step 3.
Add a new CMake and C++ definition configuration option
`LIBC_CONF_USE_MEM_BUILTINS` to allow users to use compiler builtins for
memory utility functions (memcpy, memset, memmove, memcmp, and bcmp)
instead of LLVM libc's internal implementations. Main use-cases are:
- when users want to bring their own memory functions implementations
that are highly optimized for their targets
- improve portability by providing a fallback for targets for which LLVM
libc does not have memory utility implementations yet
- to be used for libc/shared functions and their testings, as we expect
libc/shared functions to provide their own memory functions.
This patch adds the `strnlen_s` function from Annex K.
In order to reduce duplication between `strnlen` and `strnlen_s`, the
common logic has been extracted to a new internal function which both
now call.
In addition to the function definition, the patch adds a unit test and a
fuzzing test.
This moves the LLVM_LIBC_IS_DEFINED macro to its own header is
__support/macros. Its implementation leverages cpp::string_view
instead of rolling its own strcmp; this necessitated fixing
several missing constexpr in the string_view implementation.
The new __support/macros/macro-utils.h is also broken out to hold
the stringification macro and can be used in future for token
pasting shenanigans and other such generic macro machinery.
Replace the explicit `if constexpr` branching for big and little
endianness with compile-time calculated shift constants `VAL_SHIFT` and
`NEXT_SHIFT`. This simplifies the logic and reduces code duplication,
relying on the compiler to constant-fold the zero shifts into no-ops.
The variadic template helper `load_aligned` performs a specific case of
an unaligned integer load, by loading a sequence of integers from memory
at addresses expected to be aligned, and glues the results back together
with shifts and ORs into an output.
The implementation works by performing the first load, recursing on a
shorter parameter type list for the rest, and recombining via
first | (rest << size_of_first) // if little-endian
(first << size_of_first) | rest // if big-endian
But the big-endian case is wrong: it should shift left by the size of
the _rest_ of the types, not the size of the first. In the case where
you load 8, 16 and 8 bits from an odd address, you want
(first_byte << 24) | (middle_halfword << 8) | (last_byte)
but in fact we were calculating
(first_byte << 8) | (middle_halfword << 16) | (last_byte)
leading to three out of every four bytes being permuted in a memcpy from
an odd to an even address.
According to section 7.24.1 of the C standard, character comparison in
string functions must be performed as if the characters had the type
`unsigned char`.
The previous implementations of `strcmp`, `strncmp`, and `strcoll` were
doing a direct subtraction of `char` values. On platforms where `char`
is signed, this resulted in incorrect negative values being returned
when characters exceeding 127 were being compared.
This patch fixes the comparison functions to explicitly cast the
character values to `unsigned char` prior to computing their difference.
It also adds regression tests to ensure the comparison behaves correctly
for ASCII values greater than 127.
This moves the libc-internal AllocChecker API out of
src/__support/CPP/new.h and updates CPP/README.md to state the
intent to keep src/__support/CPP and the LIBC_NAMESPACE::cpp
namespace a "pure" subset of standard C++ API polyfills.
[Previous commit had an incorrect default case when
FIND_FIRST_CHARACTER_WIDE_READ_IMPL was not specified in config.json.
This PR is identical to that one with one line fixed.]
As we implement more high-performance string-related functions, we have
found a need for better control over their selection than the big-hammer
LIBC_CONF_STRING_LENGTH_WIDE_READ. For example, I have a memchr
implementation coming, and unless I implement it in every variant, a
simple binary value doesn't work.
This PR makes gives finer-grained control over high-performance
functions than the generic LIBC_CONF_UNSAFE_WIDE_READ option. For any
function they like, the user can now select one of four implementations
at build time:
1. element, which reads byte-by-byte (or wchar by wchar)
2. wide, which reads by unsigned long
3. generic, which uses standard clang vector implemenations, if
available
4. arch, which uses an architecture-specific implemenation
(Reading the code carefully, you may note that a user can actually
specify any namespace they want, so we aren't technically limited to
those 4.)
We may also want to switch from command-line #defines as it is currently
done, to something more like
llvm-project/llvm/include/llvm/Config/llvm-config.h.cmake, and
complexity out of the command-line. But that's a future problem.
[This is more of a straw-proposal than a ready-for-merging PR. I got
started thinking about what this might look like, and ended up just
implementing something as a proof-of-concept. Totally open to other
methods an ideas.]
As we implement more high-performance string-related functions, we have
found a need for better control over their selection than the big-hammer
LIBC_CONF_STRING_LENGTH_WIDE_READ. For example, I have a memchr
implementation coming, and unless I implement it in every variant, a
simple binary value doesn't work.
This PR makes gives finer-grained control over high-performance
functions than the generic LIBC_CONF_UNSAFE_WIDE_READ option. For any
function they like, the user can now select one of four implementations
at build time:
1. element, which reads byte-by-byte (or wchar by wchar)
2. wide, which reads by unsigned long
3. generic, which uses standard clang vector implemenations, if
available
4. arch, which uses an architecture-specific implemenation
(Reading the code carefully, you may note that a user can actually
specify any namespace they want, so we aren't technically limited to
those 4.)
We may also want to switch from command-line #defines as it is currently
done, to something more like
llvm-project/llvm/include/llvm/Config/llvm-config.h.cmake, and
#including the resulting file, which would move quite a bit of
complexity out of the command-line. But that's a future problem.
On exit from the loop, char_ptr had not been updated to match block_ptr,
resulting in erroneous results. Moving all updates out of the loop fixes
that.
Adjust derefences to always be inside bounds checks.
Functions like isalpha / tolower can operate on chars internally. This
allows us to get rid of unnecessary casts and open a way to creating
wchar_t overloads with the same names (e.g. for isalpha), that would
simplify templated code for conversion functions (see
315dfe5865).
Add the int->char converstion to public entrypoints implementation
instead. We also need to introduce bounds check on the input argument
values - these functions' behavior is unspecified if the argument is
neither EOF nor fits in "unsigned char" range, but the tests we've had
verified that they always return false for small negative values. To
preserve this behavior, cover it explicitly.
This options sets a compile option when building sources inside the
string directory, and this option affects string_utils.h. But
string_utils.h is #included from more places than just the string
directory (such as from __support/CPP/string.h), leading to both
narrow-reads in those cases, but more seriously, ODR violations when the
two different string_length implementations are included int he same
program.
Having this option at the top level avoids this problem.
Reverts llvm/llvm-project#107540
This PR demonstrated improvements on micro-benchmarks but the gains did
not seem to materialize in production. We are reverting this change for
now to get more data. This PR might be reintegrated later once we're
more confident in its effects.
We can use UMAXV.4S to reduce the comparison result in a single
instruction. This improves performance by roughly 4% on Apple M1:
Summary
bin/libc.src.string.bcmp_benchmark3 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10 ran
1.01 ± 0.02 times faster than bin/libc.src.string.bcmp_benchmark3 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.01 ± 0.03 times faster than bin/libc.src.string.bcmp_benchmark3 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.01 ± 0.03 times faster than bin/libc.src.string.bcmp_benchmark3 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.01 ± 0.02 times faster than bin/libc.src.string.bcmp_benchmark2 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.02 ± 0.03 times faster than bin/libc.src.string.bcmp_benchmark2 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.03 ± 0.03 times faster than bin/libc.src.string.bcmp_benchmark2 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.03 ± 0.03 times faster than bin/libc.src.string.bcmp_benchmark2 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.05 ± 0.02 times faster than bin/libc.src.string.bcmp_benchmark1 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.05 ± 0.02 times faster than bin/libc.src.string.bcmp_benchmark1 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.05 ± 0.03 times faster than bin/libc.src.string.bcmp_benchmark1 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
1.05 ± 0.02 times faster than bin/libc.src.string.bcmp_benchmark1 --study-name="new bcmp" --sweep-mode --sweep-max-size=128 --output=/dev/null --num-trials=10
(1 = original, 2 = a variant of this patch that uses UMAXV.16B, 3 = this patch)
Reviewers: michaelrj-google, gchatelet, overmighty, SchrodingerZhu
Pull Request: https://github.com/llvm/llvm-project/pull/99260
Fast strlen implementations (naive wide-reads, SIMD-based, and
x86_64/aarch64-optimized versions) all may perform
technically-out-of-bound reads, which leads to reports under ASan,
HWASan (on ARM machines), and also TSan (which also has the capability
to detect heap out-of-bound reads). So, we need to explicitly disable
instrumentation in all three cases.
Tragically, Clang didn't support `[[gnu::no_sanitize]]` syntax until
recently, and since we're supporting both GCC and Clang, we have to
revert to `__attribute__` syntax.
Summary:
This unifies the interface to just be a bunch of `load` and `store`
functions that optionally accept a mask / indices for gathers and
scatters with masks.
I had to rename this from `load` and `store` because it conflicts with
the other version in `op_generic`. I might just work around that with a
trait instead.
Summary:
I landed a change in clang that allows integral vectors to implicitly
convert to boolean ones. This means I can simplify the interface and
remove the need to cast to bool on every use. Also do some other
cleanups of the traits.
Summary:
This PR introduces a new 'simd.h' header that implements an interface
similar to the proposed `stdx::simd` in C++. However, we instead wrap
around the LLVM internal type. This makes heavy use of the clang vector
extensions and boolean vectors, instead using primitive vector types
instead of a class (many benefits to this).
I use this interface to implement a generic strlen implementation, but
propse we use this for math. Right now this requires a feature only
introduced in clang-22.
These functions turned out to have the same bug that was in wcstok()
(fixed by 4fc9801), so add the missing tests and fix the code in a way
that matches wcstok().
Also fix incorrect test expectations in existing tests.
Also update the BUILD.bazel files to actually build the strsep() test.
Summary:
This patch changes the linux build to use the wide reads on the memory
operations by default. These memory functions will now potentially read
outside of the bounds explicitly allowed by the current function. While
technically undefined behavior in the standard, plenty of C library
implementations do this. it will not cause a segmentation fault on linux
as long as you do not cross a page boundary, and because we are only
*reading* memory it should not have atomic effects.
string_utils.h uses uintptr_t, and there seems to be no tracking of this
dependency. It seems upstream builds are unaffected but downstream this
is causing a lot of flaky builds.
The code for `memcpy` is the same as in #148204 but it fixes the build
bot error by using `static_assert(cpp::always_false<decltype(access)>)`
instead of `static_assert(false)` (older compilers fails on
`static_assert(false)` in `constexpr` `else` bodies).
The code for `memset` is new and vastly improves performance over the
current byte per byte implementation.
Both `memset` and `memcpy` implementations use prefetching for sizes >=
64. This lowers a bit the performance for sizes between 64 and 256 but
improves throughput for greater sizes.
This implementation has been compiled with the [pigweed toolchain](https://pigweed.dev/toolchain.html) and tested on:
- Raspberry Pi Pico 2 with the following options\
`--target=armv8m.main-none-eabi`
`-march=armv8m.main+fp+dsp`
`-mcpu=cortex-m33`
- Raspberry Pi Pico with the following options\
`--target=armv6m-none-eabi`
`-march=armv6m`
`-mcpu=cortex-m0+`
They both compile down to a little bit more than 200 bytes and are between 2 and 10 times faster than byte per byte copies.
For best performance the following options can be set in the `libc/config/baremetal/arm/config.json`
```
{
"codegen": {
"LIBC_CONF_KEEP_FRAME_POINTER": {
"value": false
}
},
"general": {
"LIBC_ADD_NULL_CHECKS": {
"value": false
}
}
}
```