I did some investigation into libdw performance hotspots, and came up
with a few tweaks that in total trim nearly 1/3 the running time. I'm
running on Mark's recent empty-location patch, and I primarily used
"tests/varlocs -k >/dev/null" as a moderately long-running benchmark.
I'm using gcc-4.8.2-1.fc20.x86_64 and kernel-3.11.9-300.fc20.x86_64,
running on an i7-2600.
The perf profile initially looked like this:
Samples: 343K of event 'cycles', Event count (approx.): 318932712301
33.69% varlocs libdw.so [.] __libdw_find_attr
16.47% varlocs libdw.so [.] lookup.isra.0
15.63% varlocs libdw.so [.] __libdw_form_val_len
13.11% varlocs libdw.so [.] dwarf_siblingof
4.84% varlocs libdw.so [.] dwarf_tag
4.62% varlocs libdw.so [.] walk_children.4364
2.35% varlocs libdw.so [.] __libdw_findabbrev
2.32% varlocs libdw.so [.] Dwarf_Abbrev_Hash_find
1.26% varlocs libdw.so [.] dwarf_child
Patch 1 addresses form_val_len with an inlined fast path for forms with
constant length. Patch 2 is a rework of get_uleb128 and get_sleb128,
which are significant in find_attr and elsewhere. Patch 3 addresses the
hash lookup which is called often to find DIE abbreviations.
The perf profile now looks like this:
Samples: 229K of event 'cycles', Event count (approx.): 213925592727
44.63% varlocs libdw.so [.] __libdw_find_attr
22.28% varlocs libdw.so [.] dwarf_siblingof
7.64% varlocs libdw.so [.] walk_children.4388
7.07% varlocs libdw.so [.] dwarf_tag
5.18% varlocs libdw.so [.] __libdw_findabbrev
2.88% varlocs libdw.so [.] __libdw_form_val_compute_len
2.11% varlocs libdw.so [.] dwarf_child
1.44% varlocs libdw.so [.] __libdw_formref
1.12% varlocs libdw.so [.] scope_visitor
The remaining busy work is simply walking through attributes, from DIE
to DIE. I believe optimizing this further will be hard without keeping
track of DIE lengths somewhere, which is a lot to cache. Putting the
length in Dwarf_Die itself is not feasible, because those are short-
lived and frequently recreated.
Here's some summary information for how these patches change varloc -k:
libdw varlocs varlocs
text time maxres
Base: 243072 84.42s 242360k
P1: 243296 74.91s 242356k
P2: 243184 70.61s 242360k
P3: 243600 56.75s 243588k
My timings are not statistically rigorous measurements, but it still
seems a clear win across the board. Other benchmarks I've tried, like
tests/allfcts and stap -l syscall.*, show similar improvement.
Feedback is always appreciated.
I maintain a database which extracts symbol information from ELF objects
(among other things). I would like to enrich that with DWARF producer
data, and perhaps additional DWARF information in the future.
I'd really like to avoid importing the ELF symbol information twice,
once from the real object file, and once from the separate debuginfo.
The database performs content-based deduplication, this means I do not
have path name information during extraction. This mean I cannot use
file system paths to disambiguate the real thing and its debugging
information. Both files are loaded separately and not necessarily at
the same time. I don't want to change that if possible because this
would result in a scalability issue eventually. I don't want to assume
that *all* debuginfo data has been separated, either.
Based on the previous discussion around program interpreter reporting in
readelf, there is no easy way to detect separate debuginfo to trigger
special processing for it (e.g., do not extract symbols, only
One thing that would help me as well if there is a way to get the exact
same set of exported symbols from the real file and its separate
debuginfo. The I could deduplicate based on that, and processing both
files would not matter anymore. eu-readelf shows quite different output
for the two files, so I'm not sure how to achieve that.
I don't actually use eu-readelf output (but my extraction code is
derived from it), and I'm open to suggestions to look at particular
sections/headers to get matching output. I'm mainly interested in
public symbols and undefined symbols. Internal symbols from debugging
information could be ignored for the time being.
Florian Weimer / Red Hat Product Security Team
as discussed in
[PATCH] stack: Add new '-n MAXFRAMES' option. Resolve addresses after unwind.
I have found I forgot to implement what GDB has as
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
I find it easier in the case of elfutils as all supported unwinding archs have
grows-down kind of stack.
It breaks support for -fsplit-stack (GDB gdb.base/morestack.exp) but that is
IMO an add-on extension which just coincidentally worked with current
checked-in elfutils unwinder (I did not test it).
Hi Mark, Jan,
Here is fwded the posting of the patches for libdw integration in perf
The mjw/aarch64-unwind branch of elfultis is used and it works ok! One
caveat though: the compat mode is not working, I am investigating this
---------- Forwarded message ----------
From: Jean Pihet <jean.pihet(a)linaro.org>
Date: 19 March 2014 10:42
Subject: [PATCH 0/3] perf tools: Add libdw DWARF post unwind support for ARM64
To: "linux-kernel(a)vger.kernel.org" <linux-kernel(a)vger.kernel.org>,
<acme(a)ghostprotocols.net>, Ingo Molnar <mingo(a)kernel.org>, Jiri Olsa
<jolsa(a)redhat.com>, Steve Capper <steve.capper(a)linaro.org>, Mark
Cc: patches(a)linaro.org, Jean Pihet <jean.pihet(a)linaro.org>
Adding libdw DWARF post unwind support, which is part
of elfutils-devel/libdw-dev package from version 0.158.
Also includes the test suite for dwarf unwinding, by adding the
arch specific test code and the perf_regs_load function.
This series depends on the following kernel patches series:
- AARCH64 unwinding support ,
- ARM libdw integration ,
and on the changes from the branch for:
- libdw AARCH64 unwinding support .
Jean Pihet (3):
perf tests: Introduce perf_regs_load function on ARM64
perf tests: Add dwarf unwind test on ARM64
perf tools: Add libdw DWARF post unwind support for ARM64
tools/perf/Makefile.perf | 2 +-
tools/perf/arch/arm64/Makefile | 7 ++++
tools/perf/arch/arm64/include/perf_regs.h | 5 +++
tools/perf/arch/arm64/tests/dwarf-unwind.c | 59 ++++++++++++++++++++++++++++++
tools/perf/arch/arm64/tests/regs_load.S | 39 ++++++++++++++++++++
tools/perf/arch/arm64/util/unwind-libdw.c | 53 +++++++++++++++++++++++++++
tools/perf/tests/builtin-test.c | 3 +-
tools/perf/tests/tests.h | 3 +-
8 files changed, 168 insertions(+), 3 deletions(-)
create mode 100644 tools/perf/arch/arm64/tests/dwarf-unwind.c
create mode 100644 tools/perf/arch/arm64/tests/regs_load.S
create mode 100644 tools/perf/arch/arm64/util/unwind-libdw.c
- Rebased on latest acme/perf/core git tree,
- Tested on the ARMv8 Foundation emulator.
This is tested now in a native aarch64 build by tests/run-elflint-self.sh
since we added some .o files to the self tests.
Signed-off-by: Mark Wielaard <mjw(a)redhat.com>
backends/ChangeLog | 5 +++++
backends/aarch64_reloc.def | 6 +++---
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/backends/ChangeLog b/backends/ChangeLog
index 2863c00..958b34c 100644
@@ -1,3 +1,8 @@
+2014-03-19 Mark Wielaard <mjw(a)redhat.com>
+ * aarch64_reloc.def: AARCH64_ABS32 and AARCH64_ABS64 are also valid
+ in ET_REL.
2014-01-30 Petr Machata <pmachata(a)redhat.com>
* aarch64_regs.c (aarch64_register_info.regtype): Make this
diff --git a/backends/aarch64_reloc.def b/backends/aarch64_reloc.def
index f633157..36d29e6 100644
@@ -1,5 +1,5 @@
/* List the relocation types for AArch64. -*- C -*-
- Copyright (C) 2013 Red Hat, Inc.
+ Copyright (C) 2013, 2014 Red Hat, Inc.
This file is part of elfutils.
This file is free software; you can redistribute it and/or modify
@@ -28,8 +28,8 @@
/* NAME, REL|EXEC|DYN */
-RELOC_TYPE (ABS64, EXEC|DYN)
-RELOC_TYPE (ABS32, EXEC|DYN)
+RELOC_TYPE (ABS64, REL|EXEC|DYN)
+RELOC_TYPE (ABS32, REL|EXEC|DYN)
RELOC_TYPE (COPY, EXEC)
RELOC_TYPE (GLOB_DAT, EXEC|DYN)
RELOC_TYPE (JUMP_SLOT, EXEC|DYN)
this is a different approach to:
[patch] Support live PIDs with deleted files
The patch above required /proc/PID/map_files/ which is currently accessibly
only by root. Why it is so in Linux kernel:
An security breach countercase upstream is ftruncate(2) for mmap(2)ed and
close(2)ed file which was not possible without map_files/ before. This is why
map_files/ was restricted to CAP_SYS_ADMIN (=root) in the end:
As their concern there is about R/W mapping and modifications of the file they
also claim it is only an example, that the security implications may be wider.
My original plan was to make the mapping accessible for non-root in R/O mode
as elfutils is enough with R/O access only.
But now I think about a countercase where one has only a part of file
mmap(2)ped while the file is unlinked or located in inaccessible directory.
Then R/O map_files/ access will make the whole file accessible to the user.
This makes even R/O map_files/ non-root access losening of security rules.
Fortunately for backtracing of deleted executables/libraries one does not need
to access any unmapped memory (assuming here Fedora executables built with
-fasynchronous-unwind-tables). ELF header, program headers and PT_GNU_EH_FRAME
are all mapped in memory and accessible by PTRACE_PEEKTEXT (and /proc/PID/mem).
Therefore the whole solution can be done in elfutils emulating all mapped
parts of ELF from remote memory. Moreover elfutils already provides support
for ELFs in the middle of a file - as elfutils supports .a archives.
This patchset (/dev/PID/mem solution) works only on recent Linux kernels
(tested kernel-3.13.3-201.fc20.x86_64) as older kernels (RHEL-6) required
PTRACE_ATTACHed PID to read its /dev/PID/mem. This patch does not try to
workaround older kernels by doing PTRACE_ATTACH.
$ sleep 1h&p=$!;sleep 0.1;dd if=/proc/$p/mem bs=1 skip=$[0x400000] count=4|(sleep 0.1;od -c)
4+0 records in
4+0 records out
4 bytes (4 B) copied, 7.0551e-05 s, 56.7 kB/s
0000000 177 E L F
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000110785 s, 0.0 kB/s
On older kernels the testcase will:
The former patch using /proc/PID/map_files/ still may be useful so that
elfutils can access .debug_frame or .symtab embedded in the same file
(=not using /usr/lib/debug separate debuginfo).
/proc/PID/exe also makes sense.
/proc/PID/map_files/ - all files but only for root tracers
/proc/PID/exe - only main executable but even for non-root tracers
There is currently a bug in symbol resolving (eu-stack without -q) of deleted
$ cp /bin/sleep /tmp/deleteme;/tmp/deleteme 100h&p =$!;rm /tmp/deleteme;./src/stack -p $p
#0 0x000000320e6bc970 __nanosleep
#1 0x00000000004040cf x2realloc
#2 0x0000000000403f30 emit_bug_reporting_address
#3 0x00000000004018ed _init
#4 0x000000320e621d65 __libc_start_main
#5 0x00000000004019f9 _init
vs. the right one - unprelinked:
$ cp /bin/sleep /tmp/deleteme;prelink -u /tmp/deleteme;/tmp/deleteme 100h&p=$!;rm /tmp/deleteme;./src/stack -p $p
#0 0x000000320e6bc970 __nanosleep
#1 0x00000000004040cf rpl_nanosleep
#2 0x0000000000403f30 xnanosleep
#3 0x00000000004018ed main
#4 0x000000320e621d65 __libc_start_main
#5 0x00000000004019f9 _start
This is because this libdwfl/dwfl_module_getdwarf.c comment is no longer true
after this patch
If there are no section headers at all (full stripping), then
the end of the first segment is a valid synchronization address.
This cannot happen in a prelinked file, since prelink itself
relies on section headers for prelinking and for undoing it.
(If you do full stripping on a prelinked file, then you get what
you deserve--you can neither undo the prelinking, nor expect to
line it up with a debug file separated before prelinking.)
because /proc/PID/mem does not have to have section headers mapped.
Primarily I am not aware how to fix it. Besides that I find it an unrelated
bug to this patch. And also I am not completely sure it has a strong need of
a fix. Numerical backtrace (ABRT) is always right. The symbols are wrong
only for prelinked executables but in reality prelink has become or becoming
more rarely used tool.
When trying to add libdw support for Powerpc in the Linux perf tool,
I am getting the DWARF error NO_UNWIND.
Following is the rough sequence of calls to libdw are from the perf
dwfl = dwfl_begin()
dwfl_attach_state(dwfl, EM_NONE, thread_id, &callbacks, arg);
dwfl_getthread_frames(dwfl, thread_id, frame_callback, arg);
The call to dwfl_getthread_frames() results in the NO_UNWIND error
(Unwinding not supported for this architecture), although the calls
before it seem to succeed.
Looking through elfutils.git source I see that this might be due
to frames_nregs being 0, but I alos see that ppc64_init() sets
frames_nregs to 145 in backends/ppc64_init.c.
I am running Fedora 20 and built the libdw library from sources
with following elfutils.git commit as HEAD:
Author: Mark Wielaard <mjw(a)redhat.com>
Date: Wed Feb 26 17:17:07 2014 +0100
Appreciate any input on how to resolve the NO_UNWIND error on
Please see: https://sourceware.org/bugzilla/show_bug.cgi?id=16676#c2
I'm using elfutils-0.158-1.fc20.x86_64, but I don't think git libdwfl
has changed in any relevant way since then.
The gist is that I have an ET_EXEC binary, /usr/bin/ls. When I call
dwfl_module_getsymtab -> find_symtab -> find_aux_sym -> open_elf, the
following heuristic is triggered:
134│ mod->e_type = ehdr->e_type;
136│ /* Relocatable Linux kernels are ET_EXEC but act like ET_DYN. */
137│ if (mod->e_type == ET_EXEC && file->vaddr != mod->low_addr)
138├> mod->e_type = ET_DYN;
Here file->vaddr = 0x400020 and mod->low_addr = 0x400000, but this
certainly is not a kernel, and should not be treated as ET_DYN.
Maybe this only needs to add a check that file == &mod->main, but I
wonder if this heuristic is even needed at all, because report_kernel()
forcibly sets ET_DYN itself.