This finishes the removal of ConstString from FileSpec's interface. Note
that FileSpec is still _backed_ by ConstStrings, that will be changed in
a follow-up.
A non-goal for this change is to rewrite business logic, so I kept the
type of some variables as ConstString where a refactor would necessary
to change that. I converted printf-style formatting to formatv-style
formatting where required, and otherwise tried to leave callsites alone
to the extent that was possible.
Module::GetSectionList built the section list (m_sections_up) without a
lock, so parallel module loading (e.g. crashlog.py's thread pool) could
race two builders on the unique_ptr and the SectionList vector, crashing
in AppleObjCRuntime::GetObjCVersion.
Build through ObjectFile::GetSectionList instead of locking the module
mutex and calling CreateSections directly: that path locks the object
file's section mutex before the module mutex, and the build can re-enter
it, so holding the module mutex across the build would invert the order
and risk a deadlock. The Module-level counterpart to a0176fd9df.
rdar://180308581
Today we can run `target modules dump separate-debug-info --json` to get
a json blob of all the separate debug info, but it has a few
shortcomings when developing some scripting against it. Namely, the
caller has to know the structure of the JSON per architecture that will
be returned.
I've been working on a Minidump packing utility where we enumerate
symbols and source and put them in a package so we can debug with
symbols portably, and it's been difficult to maintain multiple
architectures due to the above shortcomings. To address this for myself,
I've exposed a simple iterator for the SBModule to get all the
separate-debug-info as list of filespecs with no need for the caller to
have context on what kind of data it is.
I also extened the swig interfaces to make writing my test easier and as
a nice to have.
This change is to make DataFileCache symbol table caching work with
memory-read binary modules.
When we read a Module out of memory, we keep the address of the module
in Module's m_object_name field as a string. This is normally the name
of a file in a ranlib/static library/.a archive like the "main.o" in
"foo.a(main.o)". The address is most often seen in the "image list"
output, and is the only easy way to distinguish in that output which
binaries were read out of memory, versus found on local disk. The "name"
of the Module ends up being the combination of the FileSpec plus this
m_object_name.
Reading a binary out of memory is expensive, primarily because of
reading the symbol table. The DataFileCache feature that Greg introduced
five years ago can cache the Symbol Table for a binary locally, and when
we see the same binary loaded again in a future debug session/lldb
session, we can skip parsing the symbol table (or in the case of Memory
Module, reading it from memory).
Unfortunately because the "name" of the Module is the combination of the
FileSpec PLUS the address, if this is a system where binaries can load
at different addresses, we may not be able to use the DataFileCache for
a memory module. The address should not be part of the name for this.
In Module, I created a new optional ivar to track the memory module
address, if this is one, instead of putting the string representation in
m_object_name. In all places (except the DataFileCache name
construction), I append the address to the name just like we do with
m_object_name.
There were a few logging points that were manually constructing the name
when a method would do it for them already; I changed them to call the
method instead.
Probably the least clean part is in CommandObjectTarget (the image list
output), we add the ObjectName to the FileSpec here as well. For now,
instead of trying to leverage one of the methods in Module that combine
them already, I changed it to use the same change.
I used a std::optional for this address instead of the
LLDB_INVALID_ADDRESS convention because it was a lot cleaner to read, I
thought, even if it is different than most often used in the codebase.
Open to using the older coding style if other people think consistency
is more important.
Follow-up to #198941, which introduced Locked<T> and SharedLocked<T>.
Add GetObjectFileLocked, GetSymbolFileLocked, GetSymtabLocked, and
GetSectionListLocked alongside the existing accessors.
The locked variants cover two things:
1. They prevent the pointer from being swapped out from under the
caller. The old getters take m_mutex only during lazy initialization and
release it before returning. The unique_ptr or shared_ptr that owns the
pointee can therefore be reassigned by another thread while the caller
still holds the raw value. LockedPtr keeps the Module mutex held
alongside the borrowed pointer, pinning the binding for the lifetime of
the handle.
2. They serialize access to the pointee itself. This is not new, the
classes in question were already relying on the Module mutex for
synchronization.
Migrate the four call sites in Module where the existing patter maps to
a single LockedPtr.
The legacy raw-pointer getters remain so call sites can migrate
incrementally.
LLDB invokes xcrun to find SDKs on disk. This is usually very fast, but
sometimes (after an Xcode update, or when the searched SDK does not
exist) it can take very long (10s or more). The progress event provides
user feedback to explain the hang.
Previously this functionality was in HostInfo, but it may not depend on
Core, so this patch is instrumenting the call sites instead.
cf: https://github.com/llvm/llvm-project/pull/198931
This patch makes the default parameter to ConstString::AsCString
explicit. Currently it defaults to `nullptr`. However, there a bunch of
callsites that do `printf("%s", foo.AsCString())`, which is UB (I have
at least 1 crash on me where this was the root cause).
Alternatively we could change the default to `""` instead of `nullptr`,
but code that did `if (foo.AsCString())` would then change meaning.
Command I ran to generate this patch:
```
rg "\.AsCString\(\)" lldb -l | xargs sed -i '' 's/\.AsCString()/.AsCString(nullptr)/g'
```
There's also a commonly used `Status::AsCString` with a default
parameter. So I added a temporary:
```
const char *AsCString(std::nullptr_t) const = delete;
```
overload to `Status`, and fixed all the compile errors.
In a follow-up PR I intend to change the callsites that do `printf("%s",
foo.AsCString(nullptr)` to pass `""` as the fallback parameter. Another
dangrous pattern is `std::string(error.AsCString(nullptr))`
rdar://171219173
This patch is motivated by
https://github.com/llvm/llvm-project/pull/189943, where we would like to
print the "these module scripts weren't loaded" warning for *all*
modules batched together. I.e., we want to print the warning *after* all
the script loading attempts, not from within each attempt.
To do so we want to hoist the `ReportWarning` calls in
`Module::LoadScriptingResourceInTarget` out into the callsites. But if
we do that, the callers have to remember to print the warnings. To avoid
this, we redirect all callsites to use
`ModuleList::LoadScriptingResourceInTarget`, which will be responsible
for printing the warnings.
To avoid future accidental uses of
`Module::LoadScriptingResourceInTarget` I moved the API into
`ModuleList` and made it `private`.
This patch changes the `Platform::LocateXXX` to return a map from
`FileSpec` to `LoadScriptFromSymFile` enum.
This is needed for https://github.com/llvm/llvm-project/pull/188722,
where I intend to set `LoadScriptFromSymFile` per-module.
By default the `Platform::LocateXXX` set the value to whatever the
target's current `target.load-script-from-symbol-file` is set to. In
https://github.com/llvm/llvm-project/pull/188722 we'll allow overriding
this per-target setting on a per-module basis.
Drive-by:
* Added logging when we fail to load a script.
We already log the import attempt. This patch logs the failure to load
the script since otherwise the log may be misleading. Didn't think it
was worth adding a test-case for this.
Depends on:
* https://github.com/llvm/llvm-project/pull/187229
(only second commit and onwards are relevant)
This patch implements the base infrastructure described in this [RFC re.
Moving libc++ data-formatters out of
LLDB](https://discourse.llvm.org/t/rfc-lldb-moving-libc-data-formatters-out-of-lldb/89591)
The intention is to provide vendors with a way to pre-configure a set of
paths that LLDB can automatically ingest formatter scripts from.
Three main changes:
1. Adds a CMake variable `LLDB_SAFE_AUTO_LOAD_PATHS` which is a
semi-colon separated list of paths. This is intended to be set by
vendors when building LLDB for distribution.
2. Adds a setting that only exists in asserts mode called
`testing.safe-auto-load-paths` which allows setting the paths without a
CMake configuration. Used for local development and, more crucially, the
shell and unit tests
3. Adds a `LocateScriptingResourcesFromSafePaths` which
`Platform::LocateExecutableScriptingResources` calls by default (and
hence used by all platforms). I add a
`LocateExecutableScriptingResourcesImpl` that platforms can override if
they have platform-specific resource script locations (e.g., dSYMs on
Darwin).
Whenever we load an image, we check the safe path (starting with the
last appended path) for a directory called
`/safe/path/to/<module>/<module>.py`. If such script exists, we import
it as a Python module. If not, we move on to the next safe path.
Currently the default for `LLDB_SAFE_AUTO_LOAD_PATHS` is empty.
Eventually the plan is to make those point to the libc++ installation
(where the formatters will live) depending on platform/vendor. For macOS
we'll add a special `$SDK_ROOT` that can be used in the path variable,
which `LocateScriptingResourcesFromSafePaths` will resolve to an actual
SDK path.
*AI Usage*:
* Claude assisted with the CMake machinery which I then reviewed and
cleaned up. I'm not sure this is the most idiomatic way of letting a
user provide lists of paths, but I couldn't find a better way.
* Wrote the basic auto-load Shell tests myself and asked claude to stamp
out a bunch more for different scenarios. Reviewed and cleaned those up
myself.
lldb had three preprocessor defines for logging,
LLDB_LOG - formatv style argument
LLDB_LOGF - printf style argument
LLDB_LOGV - formatv style argument, only when verbose enabled
If you weren't looking at Log.h and the definition of these three, and
wanted to log something with formatv, it was easy to use LLDB_LOGV by
accident. We just had a situation where an important log statement
wasn't logging and it turned out to be this. This is fragile if you
aren't looking at the header directly, so I'd like to make this more
explicit. My proposal:
LLDB_LOG - formatv style argument
LLDB_LOG_VERBOSE - formatv style argument, only when verbose enabled
LLDB_LOGF - printf style argument
LLDB_LOGF_VERBOSE - printf style argument, only when verbose enabled
The new fouth one is to remove several places where we do `if (log &&
log->GetVerbose()) LLDB_LOGF (...)` in the sources today, and make both
styles consistent.
This PR implements that change, mechanically changing all LLDB_LOGV's to
LLDB_LOG_VERBOSE.
It also updates many of the `if (log && log->GetVerbose()) LLDB_LOGF`'s.
Some uses of this conditional expression do extra calculations in
addition to logging, and so those were left as-is so we're not doing
throwaway work when running without verbose logging.
There were many instances throughout lldb where callers are still doing
`if (log) LLDB_LOG*(...)`, a remnant of when all calls were to the `Log`
object's `Printf()` method, and you had to check if your local Log*
pointer was non-nullptr before calling the method. I removed those,
again keeping ones where work for logging is done in the block of code.
The code changes are all mechanical and uninteresting, but the question
of whether this naming change is widely agreed on is maybe worth
discussing.
Fix and improve #187145 for following issues:
* Fix unhandled error.
* Align the log type with the file where it contains.
* The added test doesn't work on windows host for remote debugging, add
decorator to skip when host and target do not match.
Add a LoadCompilationPrefixMap() helper in SymbolFile::FindPlugin that
walks up from the symbol file's directory looking for a
compilation-prefix-map.json file. When found, each key→value entry is
applied to the module's source path mapping list, allowing LLDB to
resolve source file paths that were rewritten by -fdebug-prefix-map at
build time without requiring manual `settings set target.source-map`.
The JSON file format maps fake paths (as written into debug info) back
to their real on-disk counterparts:
{ "/fake/srcdir": "/real/srcdir" }
Directory results are cached so the filesystem is walked at most once
per unique directory across all modules loaded in a session.
Also apply the module's source path remappings in
SymbolFileDWARFDebugMap::ParseCompileUnitAtIndex when constructing
compile units from N_SO stabs. This mirrors what MakeAbsoluteAndRemap
does for the dSYM case so that fake paths baked into the debug map are
transparently resolved to real paths.
rdar://84824567
Assisted-By: Claude
I'm in the process of moving `SanitizedScriptingModuleName` into
`ScriptInterpreter` as a `virtual` API. The nullptr check inside the
constructor made that more difficult because it implied we may not have
a `ScriptInterpreter` available to call the sanitization API on. Really
the `nullptr` check is redundant because even if we succesfully sanitize
and then locate some scripts, `Module::LoadScriptingResourceInTarget`
bails out if we don't have a `ScriptInterpreter`.
This patch moves the early exit in `LoadScriptingResourceInTarget` to
before we make the call to `LocateExecutableScriptingResources`. That
way we ensure we never get to `SanitizedScriptingModuleName` without a
valid `ScriptInterpreter`.
I'm in the process of making `LoadScriptingResources` interactively ask
a user whether to load a script. I'd like to turn the existing warning
into the prompt. The simplest way to achieve this is to not print into a
`feedback_stream` parameter, and instead create a prompt right there.
This patch removes the `feedback_stream` parameter and emits a
`ReportWarning` instead. If we get around to adding the prompt instead
of the warning, those changes will be simpler to review. But even if we
don't end up replacing the warning with a prompt, moving away from
output parameters and towards more structured error reporting is a
nice-to-have (e.g., the `warning` prefix is now colored, IDEs have more
flexibility on how to present the warning, etc.).
For a command-line user nothing should change with this patch (apart
from `warning:` being highlighted).
Makes it obvious what the warning will look like (with the indenentation
etc.). Also adds a test since we had no coverage for the warning before
(as far as I'm aware).
Planning on adding more to this function/loop soon. Making it
early-return style (as suggested by the LLVM style guide) makes those
changes easier to reason about.
Drive-by:
* Reduced the indentation of the loop by doing an early-continue if the
`FileSpec` is invalid or doesn't exist
I was reading through ObjectContainerBSDArchive and came across some
dead method decls, a less-than-completely-clear `shared_ptr` typedef in
`ObjectContainerBSDArchive::Archive` for a shared_ptr<Archive> which was
a little unclear when reading a decl like `shared_ptr archive_sp;` for a
local variable.
d7fb086668 changed some calls from SymbolContextList::Append to
SymbolContextList::AppendIfUnique. This has unfortunately caused a huge
slow down in operations involving a large amount of symbol contexts (for
example, trying to autocomplete from an empty input "b <TAB>" will add
every function to the list), since AppendIfUnique scans the entire
symbol context list. Speed this up by adding a hash set to quickly
answer whether a symbol context is on the list or not.
This takes the time from running "b <TAB>" when debugging yaml2obj on my
machine from 600 seconds down to 13, which is about the same as before
d7fb086668.
Note that AppendIfUnique has a logic error, which has been present since
its introduction. This has to do with the behavior controlled by
"merge_symbol_into_function", which will try to merge symbols with
symbol context containing the equivalent function to that symbol.
The previous patch tried to correct this by adding
CompareConsideringPossiblyNullSymbol(), which is not quite correct.
With CompareConsideringPossiblyNullSymbol(), if symbols are added in
this order:
- Symbol context without symbol.
- Equivalent symbol context with symbol.
The list will have only one symbol context WITHOUT the symbol.
If we stop using CompareConsideringPossiblyNullSymbol() and instead go
back to the == operator which d7fb086668 introduced, with symbols
added in this order, the following will happen:
- Symbol context without symbol.
- Equivalent symbol context with symbol.
- The bare symbol, with "merge_symbol_into_function = true", the list
will have the same symbol twice.
This patch does not attempt to solve this, and instead focuses on the
performance issue d7fb086668 introduced.
rdar://170477680
This revert #181334 and its follow-up PRs (including #181488, #181492,
#181493, #181494 and #181498) as well as Ismail's documentation changes
(#181594, #181717). The original commit causes a test failure in CI
(https://github.com/llvm/llvm-project/issues/181938) but the more I look
at the patch, the more I'm convinced it was not ready to land. It will
be easier to iterate on the feedback by re-landing this than by using
post-commit review.
## Summary
Based on discussion from
[RFC](https://discourse.llvm.org/t/rfc-python-callback-for-source-file-resolution/83545),
this PR adds a new `SymbolLocatorScripted` plugin that allows Python
scripts to implement custom symbol and source file resolution logic.
This enables downstream users to build custom symbol servers, source
file remapping, and build artifact resolution entirely in Python.
### Changes
- Adds `LocateSourceFile()` to the SymbolLocator plugin interface,
called during source path resolution with a fully loaded `ModuleSP`, so
the plugin has access to the module's UUID, file paths, and symbols.
- Adds `SymbolLocatorScripted` plugin that delegates all four
SymbolLocator methods (`LocateExecutableObjectFile`,
`LocateExecutableSymbolFile`, `DownloadObjectAndSymbolFile`,
`LocateSourceFile`) to a user-provided Python class.
- Adds `ScriptedSymbolLocatorPythonInterface` to bridge C++ calls to
Python, with proper GIL management and error handling.
- Results for `LocateSourceFile` are cached per (module UUID, source
file) pair.
- The Python class is configured via: `settings set
plugin.symbol-locator.scripted.script-class module.ClassName`
### Python class interface
```python
class MyLocator:
def __init__(self, exe_ctx, args): ...
def locate_source_file(self, module, original_source_file):
...
def locate_executable_object_file(self, module_spec): ...
def locate_executable_symbol_file(self, module_spec,
default_search_paths): ...
def download_object_and_symbol_file(self, module_spec,
force_lookup, copy_executable): ...
```
### Test plan
```
Added TestScriptedSymbolLocator.py with 3 test cases:
- test_locate_source_file — verifies the locator resolves source
files, receives a valid SBModule with UUID, and remaps paths correctly
- test_locate_source_file_none_fallthrough — verifies returning
None falls through to default LLDB resolution, and that having no script
class set works normally
- test_invalid_script_class — verifies graceful handling of
invalid class names without crashing
```
Co-authored-by: Rahul Reddy Chamala <rachamal@fb.com>
In a PR last month I changed the ObjectFile CreateInstance etc methods
to accept an optional DataExtractorSP instead of a DataBufferSP, and
retain the extractor in a shared pointer internally in all of the
ObjectFile subclasses. This is laying the groundwork for using a
VirtualDataExtractor for some Mach-O binaries on macOS, where the
segments of the binary are out-of-order in actual memory, and we add a
lookup table to make it appear that the TEXT segment is at offset 0 in
the Extractor, etc. Working on the actual implementation, I realized we
were still using DataBufferSP's in ModuleSpec and Module, as well as in
ObjectFile::GetModuleSpecifications.
I originally was making a much larger NFC change where I had all
ObjectFile subclasses operating on DataExtractors throughout their
implementation, as well as in the DWARF parser. It was a very large
patchset. Many subclasses start with their DataExtractor, then create
smaller DataExtractors for parts of the binary image - the string table,
the symbol table, etc., for processing.
After consideration and discussion with Jonas, we agreed that a
segment/section of a binary will never require a lookup table to access
the bytes within it, so I changed
VirtualDataExtractor::GetSubsetExtractorSP to (1) require that the
Subset be contained within a single lookup table entry, and (2) return a
simple DataExtractor bounded on that byte range. By doing this, I was
able to remove all of my very-invasive changes to the ObjectFile
subclass internals; it's only when they are operating on the entire
binary image that care is needed.
One pattern that subclasses like ObjectFileBreakpad use is to take an
ArrayRef of the DataBuffer for a binary, then create a StringRef of
that, then look for strings in it. With a VirtualDataExtractor and
out-of-order binary segments, with gaps between them, this allows us to
search the entire buffer looking for a string, and segfault when it gets
to an unmapped region of the buffer. I added a
VirtualDataExtractor::GetSubsetExtractorSP(0) which gets the largest
contiguous memory region starting at offset 0 for this use case, and I
added a comment about what was being done there because I know it is not
obvious, and people not working on macOS wouldn't be familiar with the
requirement. (when we have a ModuleSpec with a DataExtractor, any of the
ObjectFile subclasses get a shot at Creating, so they all have to be
able to iterate on these)
rdar://148939795
The current behavior will pick the first symbol that contains the
address, this causes LLDB to pick the wrong symbol when looking for
swift reflection metadata on Linux, as in that case it is valid for a
symbol to completely encompass another one.
Instead, this function should prefer the symbol which is an exact, if it
exists. As a bonus, this should also be faster in the vast majority of
the cases, as we probably query symbols by their exact address most of
the time.
rdar://166344740
The ObjectFile plugin interface accepts an optional DataBufferSP
argument. If the caller has the contents of the binary, it can provide
this in that DataBufferSP. The ObjectFile subclasses in their
CreateInstance methods will fill in the DataBufferSP with the actual
binary contents if it is not set.
ObjectFile base class creates an ivar DataExtractor from the
DataBufferSP passed in.
My next patch will be a caller that creates a VirtualDataExtractor with
the binary data, and needs to pass that in to the ObjectFile plugin,
instead of the bag-of-bytes DataBufferSP. It builds on the previous
patch changing ObjectFile's ivar from DataExtractor to DataExtractorSP
so I could pass in a subclass in the shared ptr. And it will be using
the VirtualDataExtractor that Jonas added in
https://github.com/llvm/llvm-project/pull/168802
No behavior is changed by the patch; we're simply moving the creation of
the DataExtractor to the caller, instead of a DataBuffer that is
immediately used to set up the ObjectFile DataExtractor. The patch is a
bit complicated because all of the ObjectFile subclasses have to
initialize their DataExtractor to pass in to the base class.
I ran the testsuite on macOS and on AArch64 Ubutnu. (btw David, I ran it
under qemu on my M4 mac with SME-no-SVE again, Ubuntu 25.10, checked
lshw(1) cpu capabilities, and qemu doesn't seem to be virtualizing the
SME, that explains why the testsuite passes)
rdar://148939795
---------
Co-authored-by: Jonas Devlieghere <jonas@devlieghere.com>
Currently if we are debugging an app that was compiled against an SDK
that we don't know about on the host, then every time we evaluate an
expression we get following spam on the console:
```
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
error: Error while searching for Xcode SDK: Unrecognized SDK type: <some SDK>
```
It is not a fatal error but the way we spam it pretty much ruins the
debugging experience.
This patch makes it so we only log this error once per debugger session.
Not sure how to best test it since we'd need to build a program with a
particular SDK and then make it unrecognized by LLDB. Confirmed manually
that the error only gets reported once after this patch.
Inside the LLVM codebase, const vector& should just be ArrayRef, as this
more general API works both with vectors, SmallVectors and
SmallVectorImpl, as well as with single elements.
This commit replaces two uses introduced in
https://github.com/llvm/llvm-project/pull/168797 .
Some months ago, the LookupInfo constructor logic was refactored to not
depend on language specific logic, and use languages plugins instead. In
this refactor, when the language type is unknown, a single LookupInfo
object will handle multiple languages. This doesn't work well, as
multiple languages might want to configure the LookupInfo object in
different ways. For example, different languages might want to set the
m_lookup_name differently from each other, but the previous
implementation would pick the first name a language provided, and
effectively ignored every other language. Other fields of the LookupInfo
object are also configured in incompatible ways.
This approach doesn't seem to be a problem upstream, since only the
C++/Objective-C language plugins are available, but it broke downstream
on the Swift fork, as adding Swift to the list of default languages when
the language type is unknown breaks C++ tests.
This patch makes it so instead of building a single LookupInfo object
for multiple languages, one LookupInfo object is built per language
instead.
rdar://159531216
LLDB currently attaches `AsmLabel`s to `FunctionDecl`s such that that
the `IRExecutionUnit` can determine which mangled name to call (we can't
rely on Clang deriving the correct mangled name to call because the
debug-info AST doesn't contain all the info that would be encoded in the
DWARF linkage names). However, we don't attach `AsmLabel`s for structors
because they have multiple variants and thus it's not clear which
mangled name to use. In the [RFC on fixing expression evaluation of
abi-tagged
structors](https://discourse.llvm.org/t/rfc-lldb-handling-abi-tagged-constructors-destructors-in-expression-evaluator/82816)
we discussed encoding the structor variant into the `AsmLabel`s.
Specifically in [this
thread](https://discourse.llvm.org/t/rfc-lldb-handling-abi-tagged-constructors-destructors-in-expression-evaluator/82816/7)
we discussed that the contents of the `AsmLabel` are completely under
LLDB's control and we could make use of it to uniquely identify a
function by encoding the exact module and DIE that the function is
associated with (mangled names need not be enough since two identical
mangled symbols may live in different modules). So if we already have a
custom `AsmLabel` format, we can encode the structor variant in a
follow-up (the current idea is to append the structor variant as a
suffix to our custom `AsmLabel` when Clang emits the mangled name into
the JITted IR). Then we would just have to teach the `IRExecutionUnit`
to pick the correct structor variant DIE during symbol resolution. The
draft of this is available
[here](https://github.com/llvm/llvm-project/pull/149827)
This patch sets up the infrastructure for the custom `AsmLabel` format
by encoding the module id, DIE id and mangled name in it.
**Implementation**
The flow is as follows:
1. Create the label in `DWARFASTParserClang`. The format is:
`$__lldb_func:module_id:die_id:mangled_name`
2. When resolving external symbols in `IRExecutionUnit`, we parse this
label and then do a lookup by DIE ID (or mangled name into the module if
the encoded DIE is a declaration).
Depends on https://github.com/llvm/llvm-project/pull/151355
Refactor code revolving source objects such that most logics will be
reused.
The main change is to expose a single `CreateSource(addr, target)` that
can return either a normal or an assembly source object, and call
`ShouldDisplayAssemblySource()` only from this function instead of
multiple places across the code.
Other functions can use `source.IsAssemblySource()` in order to check
which type the source is.
Fix a [test
failure](https://github.com/llvm/llvm-project/pull/136236#issuecomment-2819772879)
in #136236, apply a minor renaming of statistics, and remerge. See
details below.
# Changes in #136236
Currently, `DebuggerStats::ReportStatistics()` calls
`Module::GetSymtab(/*can_create=*/false)`, but then the latter calls
`SymbolFile::GetSymtab()`. This will load symbols if haven't yet. See
stacktrace below.
The problem is that `DebuggerStats::ReportStatistics` should be
read-only. This is especially important because it reports stats for
symtab parsing/indexing time, which could be affected by the reporting
itself if it's not read-only.
This patch fixes this problem by adding an optional parameter
`SymbolFile::GetSymtab(bool can_create = true)` and receiving the
`false` value passed down from `Module::GetSymtab(/*can_create=*/false)`
when the call is initiated from `DebuggerStats::ReportStatistics()`.
---
Notes about the following stacktrace:
1. This can be reproduced. Create a helloworld program on **macOS** with
dSYM, add `settings set target.preload-symbols false` to `~/.lldbinit`,
do `lldb a.out`, then `statistics dump`.
2. `ObjectFile::GetSymtab` has `llvm::call_once`. So the fact that it
called into `ObjectFileMachO::ParseSymtab` means that the symbol table
is actually being parsed.
```
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
frame #0: 0x0000000124c4d5a0 LLDB`ObjectFileMachO::ParseSymtab(this=0x0000000111504e40, symtab=0x0000600000a05e00) at ObjectFileMachO.cpp:2259:44
* frame #1: 0x0000000124fc50a0 LLDB`lldb_private::ObjectFile::GetSymtab()::$_0::operator()(this=0x000000016d35c858) const at ObjectFile.cpp:761:9
frame #5: 0x0000000124fc4e68 LLDB`void std::__1::__call_once_proxy[abi:v160006]<std::__1::tuple<lldb_private::ObjectFile::GetSymtab()::$_0&&>>(__vp=0x000000016d35c7f0) at mutex:652:5
frame #6: 0x0000000198afb99c libc++.1.dylib`std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*)) + 196
frame #7: 0x0000000124fc4dd0 LLDB`void std::__1::call_once[abi:v160006]<lldb_private::ObjectFile::GetSymtab()::$_0>(__flag=0x0000600003920080, __func=0x000000016d35c858) at mutex:670:9
frame #8: 0x0000000124fc3cb0 LLDB`void llvm::call_once<lldb_private::ObjectFile::GetSymtab()::$_0>(flag=0x0000600003920080, F=0x000000016d35c858) at Threading.h:88:5
frame #9: 0x0000000124fc2bc4 LLDB`lldb_private::ObjectFile::GetSymtab(this=0x0000000111504e40) at ObjectFile.cpp:755:5
frame #10: 0x0000000124fe0a28 LLDB`lldb_private::SymbolFileCommon::GetSymtab(this=0x0000000104865200) at SymbolFile.cpp:158:39
frame #11: 0x0000000124d8fedc LLDB`lldb_private::Module::GetSymtab(this=0x00000001113041a8, can_create=false) at Module.cpp:1027:21
frame #12: 0x0000000125125bdc LLDB`lldb_private::DebuggerStats::ReportStatistics(debugger=0x000000014284d400, target=0x0000000115808200, options=0x000000014195d6d1) at Statistics.cpp:329:30
frame #13: 0x0000000125672978 LLDB`CommandObjectStatsDump::DoExecute(this=0x000000014195d540, command=0x000000016d35d820, result=0x000000016d35e150) at CommandObjectStats.cpp:144:18
frame #14: 0x0000000124f29b40 LLDB`lldb_private::CommandObjectParsed::Execute(this=0x000000014195d540, args_string="", result=0x000000016d35e150) at CommandObject.cpp:832:9
frame #15: 0x0000000124efbd70 LLDB`lldb_private::CommandInterpreter::HandleCommand(this=0x0000000141b22f30, command_line="statistics dump", lazy_add_to_history=eLazyBoolCalculate, result=0x000000016d35e150, force_repeat_command=false) at CommandInterpreter.cpp:2134:14
frame #16: 0x0000000124f007f4 LLDB`lldb_private::CommandInterpreter::IOHandlerInputComplete(this=0x0000000141b22f30, io_handler=0x00000001419b2aa8, line="statistics dump") at CommandInterpreter.cpp:3251:3
frame #17: 0x0000000124d7b5ec LLDB`lldb_private::IOHandlerEditline::Run(this=0x00000001419b2aa8) at IOHandler.cpp:588:22
frame #18: 0x0000000124d1e8fc LLDB`lldb_private::Debugger::RunIOHandlers(this=0x000000014284d400) at Debugger.cpp:1225:16
frame #19: 0x0000000124f01f74 LLDB`lldb_private::CommandInterpreter::RunCommandInterpreter(this=0x0000000141b22f30, options=0x000000016d35e63c) at CommandInterpreter.cpp:3543:16
frame #20: 0x0000000122840294 LLDB`lldb::SBDebugger::RunCommandInterpreter(this=0x000000016d35ebd8, auto_handle_events=true, spawn_thread=false) at SBDebugger.cpp:1212:42
frame #21: 0x0000000102aa6d28 lldb`Driver::MainLoop(this=0x000000016d35ebb8) at Driver.cpp:621:18
frame #22: 0x0000000102aa75b0 lldb`main(argc=1, argv=0x000000016d35f548) at Driver.cpp:829:26
frame #23: 0x0000000198858274 dyld`start + 2840
```
# Changes in this PR top of the above
Fix a [test
failure](https://github.com/llvm/llvm-project/pull/136236#issuecomment-2819772879)
in `TestStats.py`. The original version of the added test checks that
all modules have symbol count zero when `target.preload-symbols ==
false`. The test failed on macOS. Due to various reasons, on macOS,
symbols can be loaded for dylibs even with that setting, but not for the
main module. For now, the fix of the test is to limit the assertion to
only the main module. The test now passes on macOS. In the future, when
we have a way to control a specific list of plug-ins to be loaded, there
may be a configuration that this test can use to assert that all modules
have symbol count zero.
Apply a minor renaming of statistics, per the
[suggestion](https://github.com/llvm/llvm-project/pull/136226#issuecomment-2825080275)
in #136226 after merge.
This reverts commit d5b40c71f6.
This change broke greendragon lldb test:
lldb-api :: commands/statistics/basic/TestStats.py
And is therefore being reverted.
Currently, `DebuggerStats::ReportStatistics()` calls
`Module::GetSymtab(/*can_create=*/false)`, but then the latter calls
`SymbolFile::GetSymtab()`. This will load symbols if haven't yet. See
stacktrace below.
The problem is that `DebuggerStats::ReportStatistics` should be
read-only. This is especially important because it reports stats for
symtab parsing/indexing time, which could be affected by the reporting
itself if it's not read-only.
This patch fixes this problem by adding an optional parameter
`SymbolFile::GetSymtab(bool can_create = true)` and receive the `false`
value passed down from `Module::GetSymtab(/*can_create=*/false)` when
the call was initiated from `DebuggerStats::ReportStatistics()`.
Reverts llvm/llvm-project#132274
Broke a test on LLDB Widows on Arm:
https://lab.llvm.org/buildbot/#/builders/141/builds/7726
```
FAIL: test_dwarf (lldbsuite.test.lldbtest.TestExternCSymbols.test_dwarf)
<...>
self.assertTrue(self.res.Succeeded(), msg + output)
AssertionError: False is not true : Command 'expression -- foo()' did not return successfully
Error output:
error: Couldn't look up symbols:
int foo(void)
Hint: The expression tried to call a function that is not present in the target, perhaps because it was optimized out by the compiler.
```
So the dSYM can be told what target it has been loaded into.
When lldb is loading modules, while creating a target, it will run
"command script import" on any Python modules in Resources/Python in the
dSYM. However, this happens WHILE the target is being created, so it is
not yet in the target list. That means that these scripts can't act on
the target that they a part of when they get loaded.
This patch adds a new python API that lldb will call:
__lldb_module_added_to_target
if it is defined in the module, passing in the Target the module was
being added to, so that code in these dSYM's don't have to guess.
This commit modifies the `DebuggerStats::ReportStatistics`
implementation to avoid loading symbol files for unloaded symbols. We
collect stats on debugger shutdown and without this change it can cause
the debugger to hang for a long while on shutdown if they symbols were
not previously loaded (e.g. `settings set target.preload-symbols
false`).
The implementation is done by adding an optional parameter to
`Module::GetSymtab` to control if the corresponding symbol file will be
loaded in the same way that can control it for `Module::GetSymbolFile`.
The original code resulted in a misfire in the symtab vs. debug info
deduplication code, which caused us to return the same function twice
when searching via a regex (for functions whose entry point is also not
the lowest address).
"statistics dump" currently report the statistics of all targets in
debugger instead of current target. This is wrong because there is a
"statistics dump --all-targets" option that supposed to include
everything.
This PR fixes the issue by only report statistics for current target
instead of all. It also includes the change to reset statistics debug
info/symbol table parsing/indexing time during debugger destroy. This is
required so that we report current statistics if we plan to reuse
lldb/lldb-dap across debug sessions
---------
Co-authored-by: jeffreytan81 <jeffreytan@fb.com>
I've been getting complaints from users being spammed by -gmodules
missing file warnings going out of control because each object file
depends on an entire DAG of PCM files that usually are all missing at
once. To reduce this problem, this patch does two things:
1. Module now maintains a DenseMap<hash, once> that is used to display
each warning only once, based on its actual text.
2. The PCM warning itself is reworded to include less details, such as
the DIE offset, which is only useful to LLDB developers, who can get
this from the dwarf log if they need it. Because the detail is omitted
the hashing from (1) deduplicates the warnings.
rdar://138144624
As specified in the docs,
1) raw_string_ostream is always unbuffered and
2) the underlying buffer may be used directly
( 65b13610a5 for further reference )
* Don't call raw_string_ostream::flush(), which is essentially a no-op.
* Avoid unneeded calls to raw_string_ostream::str(), to avoid excess
indirection.