I did some investigation into libdw performance hotspots, and came up
with a few tweaks that in total trim nearly 1/3 the running time. I'm
running on Mark's recent empty-location patch, and I primarily used
"tests/varlocs -k >/dev/null" as a moderately long-running benchmark.
I'm using gcc-4.8.2-1.fc20.x86_64 and kernel-3.11.9-300.fc20.x86_64,
running on an i7-2600.
The perf profile initially looked like this:
Samples: 343K of event 'cycles', Event count (approx.): 318932712301
33.69% varlocs libdw.so [.] __libdw_find_attr
16.47% varlocs libdw.so [.] lookup.isra.0
15.63% varlocs libdw.so [.] __libdw_form_val_len
13.11% varlocs libdw.so [.] dwarf_siblingof
4.84% varlocs libdw.so [.] dwarf_tag
4.62% varlocs libdw.so [.] walk_children.4364
2.35% varlocs libdw.so [.] __libdw_findabbrev
2.32% varlocs libdw.so [.] Dwarf_Abbrev_Hash_find
1.26% varlocs libdw.so [.] dwarf_child
Patch 1 addresses form_val_len with an inlined fast path for forms with
constant length. Patch 2 is a rework of get_uleb128 and get_sleb128,
which are significant in find_attr and elsewhere. Patch 3 addresses the
hash lookup which is called often to find DIE abbreviations.
The perf profile now looks like this:
Samples: 229K of event 'cycles', Event count (approx.): 213925592727
44.63% varlocs libdw.so [.] __libdw_find_attr
22.28% varlocs libdw.so [.] dwarf_siblingof
7.64% varlocs libdw.so [.] walk_children.4388
7.07% varlocs libdw.so [.] dwarf_tag
5.18% varlocs libdw.so [.] __libdw_findabbrev
2.88% varlocs libdw.so [.] __libdw_form_val_compute_len
2.11% varlocs libdw.so [.] dwarf_child
1.44% varlocs libdw.so [.] __libdw_formref
1.12% varlocs libdw.so [.] scope_visitor
The remaining busy work is simply walking through attributes, from DIE
to DIE. I believe optimizing this further will be hard without keeping
track of DIE lengths somewhere, which is a lot to cache. Putting the
length in Dwarf_Die itself is not feasible, because those are short-
lived and frequently recreated.
Here's some summary information for how these patches change varloc -k:
libdw varlocs varlocs
text time maxres
Base: 243072 84.42s 242360k
P1: 243296 74.91s 242356k
P2: 243184 70.61s 242360k
P3: 243600 56.75s 243588k
My timings are not statistically rigorous measurements, but it still
seems a clear win across the board. Other benchmarks I've tried, like
tests/allfcts and stap -l syscall.*, show similar improvement.
Feedback is always appreciated.
as discussed in
[PATCH] stack: Add new '-n MAXFRAMES' option. Resolve addresses after unwind.
I have found I forgot to implement what GDB has as
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
I find it easier in the case of elfutils as all supported unwinding archs have
grows-down kind of stack.
It breaks support for -fsplit-stack (GDB gdb.base/morestack.exp) but that is
IMO an add-on extension which just coincidentally worked with current
checked-in elfutils unwinder (I did not test it).
this is a different approach to:
[patch] Support live PIDs with deleted files
The patch above required /proc/PID/map_files/ which is currently accessibly
only by root. Why it is so in Linux kernel:
An security breach countercase upstream is ftruncate(2) for mmap(2)ed and
close(2)ed file which was not possible without map_files/ before. This is why
map_files/ was restricted to CAP_SYS_ADMIN (=root) in the end:
As their concern there is about R/W mapping and modifications of the file they
also claim it is only an example, that the security implications may be wider.
My original plan was to make the mapping accessible for non-root in R/O mode
as elfutils is enough with R/O access only.
But now I think about a countercase where one has only a part of file
mmap(2)ped while the file is unlinked or located in inaccessible directory.
Then R/O map_files/ access will make the whole file accessible to the user.
This makes even R/O map_files/ non-root access losening of security rules.
Fortunately for backtracing of deleted executables/libraries one does not need
to access any unmapped memory (assuming here Fedora executables built with
-fasynchronous-unwind-tables). ELF header, program headers and PT_GNU_EH_FRAME
are all mapped in memory and accessible by PTRACE_PEEKTEXT (and /proc/PID/mem).
Therefore the whole solution can be done in elfutils emulating all mapped
parts of ELF from remote memory. Moreover elfutils already provides support
for ELFs in the middle of a file - as elfutils supports .a archives.
This patchset (/dev/PID/mem solution) works only on recent Linux kernels
(tested kernel-3.13.3-201.fc20.x86_64) as older kernels (RHEL-6) required
PTRACE_ATTACHed PID to read its /dev/PID/mem. This patch does not try to
workaround older kernels by doing PTRACE_ATTACH.
$ sleep 1h&p=$!;sleep 0.1;dd if=/proc/$p/mem bs=1 skip=$[0x400000] count=4|(sleep 0.1;od -c)
4+0 records in
4+0 records out
4 bytes (4 B) copied, 7.0551e-05 s, 56.7 kB/s
0000000 177 E L F
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000110785 s, 0.0 kB/s
On older kernels the testcase will:
The former patch using /proc/PID/map_files/ still may be useful so that
elfutils can access .debug_frame or .symtab embedded in the same file
(=not using /usr/lib/debug separate debuginfo).
/proc/PID/exe also makes sense.
/proc/PID/map_files/ - all files but only for root tracers
/proc/PID/exe - only main executable but even for non-root tracers
There is currently a bug in symbol resolving (eu-stack without -q) of deleted
$ cp /bin/sleep /tmp/deleteme;/tmp/deleteme 100h&p =$!;rm /tmp/deleteme;./src/stack -p $p
#0 0x000000320e6bc970 __nanosleep
#1 0x00000000004040cf x2realloc
#2 0x0000000000403f30 emit_bug_reporting_address
#3 0x00000000004018ed _init
#4 0x000000320e621d65 __libc_start_main
#5 0x00000000004019f9 _init
vs. the right one - unprelinked:
$ cp /bin/sleep /tmp/deleteme;prelink -u /tmp/deleteme;/tmp/deleteme 100h&p=$!;rm /tmp/deleteme;./src/stack -p $p
#0 0x000000320e6bc970 __nanosleep
#1 0x00000000004040cf rpl_nanosleep
#2 0x0000000000403f30 xnanosleep
#3 0x00000000004018ed main
#4 0x000000320e621d65 __libc_start_main
#5 0x00000000004019f9 _start
This is because this libdwfl/dwfl_module_getdwarf.c comment is no longer true
after this patch
If there are no section headers at all (full stripping), then
the end of the first segment is a valid synchronization address.
This cannot happen in a prelinked file, since prelink itself
relies on section headers for prelinking and for undoing it.
(If you do full stripping on a prelinked file, then you get what
you deserve--you can neither undo the prelinking, nor expect to
line it up with a debug file separated before prelinking.)
because /proc/PID/mem does not have to have section headers mapped.
Primarily I am not aware how to fix it. Besides that I find it an unrelated
bug to this patch. And also I am not completely sure it has a strong need of
a fix. Numerical backtrace (ABRT) is always right. The symbols are wrong
only for prelinked executables but in reality prelink has become or becoming
more rarely used tool.
On 02/12/2014 01:36 PM, Mark Wielaard wrote:
> On Wed, 2014-02-12 at 13:14 +0100, Florian Weimer wrote:
>> Here's the original issue I ran into: Separate debugging information,
>> for example the file /usr/lib/debug/usr/lib64/libdl-2.18.so.debug in
>> package glibc-debuginfo-2.18-12.fc20.x86_64, often contains garbage as
>> the program interpreter:
>> $ eu-readelf -a /usr/lib/debug/usr/lib64/libdl-2.18.so.debug | grep
>> Requesting | xxd
>> 0000000: 095b 5265 7175 6573 7469 6e67 2070 726f .[Requesting pro
>> 0000010: 6772 616d 2069 6e74 6572 7072 6574 6572 gram interpreter
>> 0000020: 3a20 0608 bf14 5d0a : ....].
>> Is this a known issue with debuginfo separation? Should I report this
> Aha. Yeah, the issue is that the phdr doesn't really make sense for a
> separate debuginfo file. It isn't really a loadable ELF file. Only the
> shdrs make sense, since the debuginfo sections are accessed through. In
> theory [eu]-strip could just remove the phdrs, but some tools do expect
> them to be there and we do sometimes use them to make sure the original
> (stripped) ELF file and the debug file match (eu-unstrip will only
> reconstruct the original ELF file, putting the debuginfo back in, if the
> phdrs match up for example).
> If you look with eu-readelf -l you'll see the mapping form phdr segments
> (which would be how an ELF file is mapped into memory) to the shdr
> sections. e.g.
> Program Headers:
> Type Offset VirtAddr PhysAddr FileSiz
> MemSiz Flg Align
> PHDR 0x000040 0x00000033c8e00040 0x00000033c8e00040 0x0001f8
> 0x0001f8 R E 0x8
> INTERP 0x001a50 0x00000033c8e01a50 0x00000033c8e01a50 0x00001c
> 0x00001c R 0x10
> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
> Section to Segment mapping:
> Segment Sections...
> 01 [RO: .interp]
> The problem in the .debug file is that the sections on disk that map to
> loadable segments have been replaced with NOBITS. eu-readelf -S on the
> debug file will give:
>  .interp NOBITS 0000000000001a50 00000280
> 0000001c 0 A 0 0 16
> But when eu-readelf tries to display the content of the PT_INTERP
> segment it doesn't know that, so it tries to print the content anyway.
> Separate debuginfo files stretch the ELF concept a little. I'll have to
> think about how to handle this nicely and/or how we can detect that the
> phdrs aren't to be trusted in this case.
Is there a reliable way to recognize separate debugging information?
Then I could avoid extracting data from phdrs.
Florian Weimer / Red Hat Product Security Team
If ino == last_ino && dmajor == last_dmajor && dminor == last_dminor then
we expect the file names to be the same as well. Which is reasonable if
the input came from the /proc file system. But there could be bad user
input if the file was supplied through dwfl_linux_proc_maps_report.
Instead of asserting on the bad input, just signal a bad_report.
Signed-off-by: Mark Wielaard <mjw(a)redhat.com>
libdwfl/ChangeLog | 4 ++++
libdwfl/linux-proc-maps.c | 3 ++-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/libdwfl/ChangeLog b/libdwfl/ChangeLog
index e55b03b..7d25330 100644
@@ -1,3 +1,7 @@
+2014-02-26 Mark Wielaard <mjw(a)redhat.com>
+ * linux-proc-maps.c (proc_maps_report): Don't assert on bad input.
2014-01-17 Petr Machata <pmachata(a)redhat.com>
* relocate.c (relocate_section): Use gelf_fsize instead of relying
diff --git a/libdwfl/linux-proc-maps.c b/libdwfl/linux-proc-maps.c
index cdb6959..a52920c 100644
@@ -241,7 +241,8 @@ proc_maps_report (Dwfl *dwfl, FILE *f, GElf_Addr sysinfo_ehdr, pid_t pid)
&& ino == last_ino && dmajor == last_dmajor && dminor == last_dminor)
/* This is another portion of the same file's mapping. */
- assert (!strcmp (last_file, file));
+ if (strcmp (last_file, file) != 0)
+ goto bad_report;
high = end;
Hi Jan, Mark,
I have a simple question which I cannot find the solution for.
The goal is to add the libdw support for perf on ARM (and later AARCH64).
I am trying to compile libdw for ARM, both natively and on x86_64. I
am getting build errors, it seems that the build is always targeted to
The question is: how to compile for non-x86, including the backends?
make: Leaving directory `/home/def/work/linaro/elfutils/libdw'
Making all in libcpu
make: Entering directory `/home/def/work/linaro/elfutils/libcpu'
m4 -Di386 -DDISASSEMBLER defs/i386 > i386_defsT
m4 -Dx86_64 -DDISASSEMBLER defs/i386 > x86_64_defsT
*** missing i386_dis.h; configure with --enable-maintainer-mode
*** missing x86_64_dis.h; configure with --enable-maintainer-mode
make: *** [i386_dis.h] Error 1
If I disable libcpu in Makefile I get an error in the backends dir:
make: *** No rule to make target `../libcpu/libcpu_i386.a', needed
by `libebl_i386.so'. Stop.
make: Leaving directory `/home/def/work/linaro/elfutils/backends'
make: *** [all-recursive] Error 1