relocate branch status

Roland McGrath roland at redhat.com
Thu Jul 1 23:55:22 UTC 2010


Aside from pushing out 0.148, I mostly worked more on the relocate branch
in the last week.

I got side-tracked for a while fiddling with CFI.  But then I set that
aside (on roland/relocate-cfi-tmp).

I redid the .debug_line relocatable handling.  This was with the
realization that relocs for .debug_line will always be relative to defined
symbols, since they come with the code.  Unlike the general case, we don't
really need to leave reloc resolution per se to the latest possible point.
We can always resolve the reloc to a "final" section-relative address and
section index in the defining file.  So dwarf_getsrclines now does this.
This has a few trade-offs:

We do more work up front, looking up all the relocs, addends, and symbols
for the whole line table (i.e. whole CU--perhaps one day for a merged line
table across CUs).  That is less lazy than before.

When resolving a Dwarf_Line, we don't have to do the reloc lookup any more.
Instead we've just saved the symtab index and a section-relative address.
So instead of binary search every time there, we skip straight to the
constant-time symtab lookup.  (We paid the binary search cost up front,
but only once per reloc.)

It's meaningful to sort a table, since we know what the actual
section-relative addresses are.  So now we keep the line table sorted for
the relocatable case as we always did for the final/absolute case.  It's
sorted first by the section index, and second by the section offset.

This makes it possible and efficient to add dwarf_getsrc_relocatable, which
I did.  You pass it a Dwarf_Relocatable, e.g. from some attribute.  It
resolves that to a section index and section-relative address, and finds a
match in the line table.

The same sort of plan will apply to CFI, where we also have an
address-keyed search table for FDEs to populate.  Then having
dwarf_addrframe_relocatable will make sense too.  (I'm still not
sure I'll work on that soon, but thinking about it is what got
me to redo the line stuff.)

I've also added dwarf_getlocation_relocatable_addr (type that ten times
fast!).  This is a cousin of dwarf_getlocation_implicit_value, as I
mentioned before.  You are supposed to pass it the Dwarf_Op pointer that is
part of an array you got back from dwarf_getlocation_relocatable, and is a
DW_OP_addr operation.  It gives you the relocatable form of that address.
(You only use it with the results of dwarf_getlocation_relocatable, because
a dwarf_getlocation or dwarf_getlocation_addr call will try to resolve each
DW_OP_addr immediately and thus give an error for non-dwfl cases that needed
any relocation.)

Because of the resolve-to-symndx hacking I did for the line table stuff,
this is actually simpler than I thought.  It doesn't actually store away
anything keyed on that Dwarf_Op pointer, as the implementation for
dwarf_getlocation_implicit_value does.  But it's still got the some API
contract that pointer (and the Dwarf_Attribute) match ones you could from a
previous dwarf_getlocation_relocatable call.  Since the symndx and
section-relative address are all we really need, and op->number was already
that natural place for that address (i.e. the absolute address is already
there if not relocatable), I just put the symndx in op->number2.  That reads
false if you look at the Dwarf_Op and call it "second operand", but you're
not supposed to be looking directly at a DW_OP_addr that might need
relocation anyway.  The rest of what you need to fill Dwarf_Relocatable is
just the CU and the IDX_debug_* value, which we can deduce from the
attribute.  So you can in fact pass a bogus Dwarf_Op to the new call and
just get a bogus Dwarf_Relocatable as result (rather than a clean error for
the pointer not being found in the table), but that's all you really deserve.

Note that DW_OP_addr can also come from CFI expressions (unlike
DW_OP_implicit_value, which is invalid in CFI).  The new API takes a
Dwarf_Attribute and refers to that CU, like dwarf_getlocation_implicit_value.
So we'll need a parallel API call for CFI cases, where you only pass it
a Dwarf_CFI pointer and a Dwarf_Op pointer.

So, modulo CFI, that covers the essential new relocatable interfaces (and
then a couple).  All those APIs remain utterly untested.

Next, I worked on libdwfl.  The relocate-on-demand hooks in libdw now call
directly into libdwfl, with the "needs relocation" errors for any Dwarf not
created by libdwfl.  The libdwfl relocation beforehand is disabled.
Unfortunately, I could not rip out the main libdwfl/relocate.c code
entirely, because we still have that godforsaken partial relocation of text
for dwfl_module_address_section and the even more dismal semantics that
dwfl_module_getelf does it on every section.  That's now every non-debugging
section, which I hope will not break any old users.

I'm not even sure if or for what stap actually still needs that relocation
logic.  It was originally for the markers sections in kernel modules, but
that has disappeared from kernels anyway.  I hope we might be able to
deprecate and get rid of that stuff at some point.

Along the way it's becoming clear we'll want to clean up libdwfl some more.
The spaghetti between libdw and libdwfl is now somewhat bizarre.  It's much
less like there is a pure libdw layer with libdwfl on top of it, and we are
not really benefitting from maintaining that fiction.  I think we'll make
them less separate over time.  The easy starting place that comes to mind
because of the ugliness of some of the relocation hook code now is to merge
dwfl_err* with dwarf_err*.  I think we'll probably do at least that before
any stable release with the new stuff now on the relocate branch, and maybe
some more.  I'd always figured that the main clean-up shake-out for libdwfl
would come when we work out good C++ versions of the dwflized interface.
But it's not at all clear when working on that would be a sensible priority.
Having it all good and clean in C++ APIs will be a boon to people working on
code like the stap translator, preventing many of the pit-falls they've had
heretofore working with the C APIs.  But a big conversion to that may be a
long way off.  The libdwfl part of the C++ picture is not necessary for
most, probably all, of the DWARF compression work.

Anyway, it seems to work again now.  I ran the systemtap-1.2 test suite
against the relocate-branch libdw and it had no regressions.  

I did some rough timings of trunk vs relocate branch with libdwfl looking at
one to many kernel modules.  I was far from rigorous enough to say much
about the performance, though it did help me isolate some bugs and big
braino-level performance problems in my recent hacking.  In theory, there
should be a big potential benefit primarily in VM load, and perhaps so in
user CPU time too, depending on the access pattern.  It is in part just an
eager/lazy tradeoff: a little new up-front cost to prepare to be lazy plus
some incremental on-demand cost, replacing a big up-front cost that prepares
to be optimal later on--so you would expect it to win at some threshold of
the ratio of how big the data is to how little of it your access pattern
actually looks at.  The other part is the VM/filesystem load of it.  We mmap
every debug file.  Previously we'd then read most of the pages of each one,
waiting to get it eagerly from the disk at startup, and then touch most of
those pages, causing a COW fault for each page.  Now we don't read very many
pages at startup time at all, and we never touch any--so the number of
file-backed page faults of any kind is strictly less, and the overall memory
consumption should be less too.

One test is "tests/line2addr -K getroot.c:54".  That will call
dwarf_getsrclines on every CU of every kernel module on the disk.
But it doesn't really look at the DIE tree at all, nor anything else.
So it will resolve all the relocs in .debug_line, and will have digested the
.debug_info relocs and resolved a few just to look at the CUs, but not
looked at the bulk of the data.  As expected it, it does a fair bit better
there.  It saves nearly as high a proportion of the system time spent (due
to fewer read and COW faults) as the proportion of the user time spent it
saves (due to less relocation work), and has noticeably smaller counts of
page faults and maximum RSS.

The obvious but misleading test is just "eu-unstrip -nK", of course.
But actually unstrip calls dwfl_module_getelf (not for any really
good reason, but I'm not messing with it this week).  So that spends
time doing the partial relocation of non-debugging sections, which is
especially useless.

I've only done warm-cache tests since flushing the file caches is too
annoying.  For kernel modules, a large portion of the overall cost is just
listing the directories and opening all the thousands of little files.  So
cold-cache runs will suffer quite badly in either branch.  (There's no
comparison to something like using eu-make-debug-archive or another scheme
to get it all in one or two files.)

A contrary sort of test is: stap -l 'module("*").function("*")'.
I was just happy enough that worked correctly, really.  But it was
marginally slower than the trunk, and that's not surprising.  This
will descend the DIE tree of each CU and resolve numerous relocs along
the way just to notice all the functions.  And I'm not at all sure
how much extra DWARF inspection stap may do that it doesn't really need to.
If you eventually look at nearly everything that gets relocated, then
laziness doesn't help at all and you're better off doing eager relocation.
But it's at least reassuring that it's not substantially slower.

The access pattern that should be helped most is something that loads all
the modules into the Dwfl, but then only dips into the DIE tree in narrow
direct ways, and little or not at all in some modules.  I'm not really
sure how to make stap behave much like that.


So, what next?  Unless I get distracted by CFI again (we loves teh CFI!),
it might be about time to shelve the relocate branch for a little while.
I might do a little libdwfl cleanup, I'll have to look around there.  It
would certainly be good for anyone interested to look at the new APIs
and/or new code and kibitz, or for more testing to be done on the branch.
But, though the API details are still quite subject to change, I've now
thought through pretty well what first-class relocatable support looks
like.  We can keep that in mind on the C++ side as we clean up and fill
out things there, but we don't actually need to get relocatable support
into that work until after a lot more meat is much more finished.  We
don't really need to get the relocate branch stuff firmed up until we are
ready to look at merging it into the dwarf branch.


Thanks,
Roland


More information about the elfutils-devel mailing list