0.138 release
by Roland McGrath
The roland/0.138 branch has the roland/bug-compat changes plus release prep
and some nits:
* make .tar.bz2 file now, not .tar.gz (1.4M vs 1.7M)
* rewrote awk crapola for better-looking %changelog
If there is no objection, I'd like to merge this to master and call it
0.138 in the next few days.
For a release tag with git, the procedure is:
git tag -s -m 'elfutils 0.138 release' elfutils-0.138
(That runs gpg to get your signature.) Then be sure to do autoreconf -f -i
before 'make distcheck'. Or I'm happy to do all this myself if you approve.
Thanks,
Roland
14 years, 9 months
dwarflint code review
by Roland McGrath
Overall it looks fine, a great start.
There are some cosmetic nits: { should always be on its own line;
comment lines don't use leading "*"; we don't use StudlyCaps in new
internal code, it's only for public type names; no need for __extension__
in our code, only for public headers.
Use xmalloc, xcalloc, xrealloc, rather than checking for failure.
Use assert rather than conditional calls to abort.
Use macros rather than 'static const size_t' for constants.
In read_ctx_read_var and similar cases, I'd use a switch instead of 'if's.
Use bool for storing 'has_children'. Just check its validity at the
decoding site.
There is no structural validity check for tag values, except that they are
not zero and not greater than INT_MAX. Likewise for attr names. These are
not "structural" elements--we can navigate the tree just fine without
knowing them, and they will not confuse libdw or its interfaces.
Don't say "non-NULL abbrev" and "NULL abbrev" in messages. "NULL" in caps
is a special term for pointers in C and C++. If "null" makes sense in
context, use it as a normal word. Here, I think you mean "zero" or "empty"
or something else like that.
Each warning/error message should give the section name and offset where
the suspicious data is found. (Many do, but some don't.)
The DW_FORM_ref_addr encoding depends on the DWARF version number field in
the CU header. For version==2, it's address_size. For 3, it's offset_size.
There is no validity check you need to do at the structural level for
DW_FORM_addr. We'll consider semantic attribute value checks in the higher
layer checks later on. The checks we have so far are only to handle the
structural level of stuff that could confuse libdw or that libdw would fail
to notice.
Likewise, string/strp don't need a structural check beyond being properly
terminated and not running off the end of the section (and the .debug_str
coverage checks). The contents of the string are for a higher-level value
check.
DW_FORM_flag could use a validity check for a value not 0 or 1.
This is not "structural" per se, but is information that libdw
loses (in dwarf_formflag), so we can't check it at higher level.
Thanks,
Roland
14 years, 9 months
exposing ebl from dwfl
by Petr Machata
Hi,
I have a small exploratory project that uses elfutils. I'm using
libdwfl and libasm.
libasm needs Ebl handle to do disassembly. So if my binary wants to use
libasm, I need to link in libebl.a. libebl opens the right backend via
dlopen, so in addition to that, I also need to set up the right RUNPATH
on my binary.
That's fine, but I noticed that libdwfl already obtains Ebl for certain
operations, and already has the right RUNPATH set up. So would it make
a sense to expose the Ebl handle via some libdwfl API function?
Thanks,
PM
14 years, 9 months
stap libdwfl bug-compatibility requirements
by Roland McGrath
We are preparing to release version 0.138 of elfutils. I gather that
systemtap might be expecting some buggy old behavior that we've fixed.
I'd like to coordinate on what any such issues are and address them so
users don't experience any regressions.
I'm concerned with two kinds of compatibility.
1. Existing systemtap binaries (including distro packages) linked
against system elfutils libs do not experience regressions when
newer elfutils libs are installed.
2. Systemtap binaries built freshly next week from new sources either
become #1 if built against elfutils-devel < 0.138, or are built so
they correctly require elfutils-libs >= 0.138 installed to run if
built against elfutils-devel >= 0.138.
I don't propose to worry about preexisting systemtap sources being
freshly rebuilt against elfutils-devel >= 0.138.
The one bug-compatibility issue I'm aware of where I think systemtap
might be relying on buggy library behavior is dwfl_module_build_id.
In older elfutils, this yields the wrong vaddr: one that (in practice)
points to the byte past the end of the build ID bytes. In 0.138, it
behaves as decribed and intended: it yields the vaddr of the first
byte of the build ID.
The other case I'm aware of is dwfl_module_relocate_address for prelinked
ET_DYN files with separate debuginfo. Here AFAIK systemtap has never
relied on the bad behavior. However, it now has (or shortly will have) a
workaround for the bad behavior that will be unnecessary with 0.138.
Issues in class #1 can be addressed via symbol versioning in libdwfl.
The dwfl_module_build_id function that fresh linking gets you has been
fixed, and it moves into the ELFUTILS_0.138 version set. libdw.so
also defines a dwfl_module_build_id symbol in the old version set
(ELFUTILS_0.130), which is a bug-compatible wrapper that behaves like
the old library did.
For #2, I propose adding an installed header <elfutils/version.h> that
will define a macro for the elfutils release that these headers and
libraries (i.e. elfutils-devel) come from. This allows users like
systemtap to test what library behavior they are getting. You can
either use this in a configure check that does an AC_COMPILE_CHECK on:
#include <elfutils/version.h>
#if !_ELFUTILS_PREREQ (0, 138)
# error "must build with elfutils >= 0.138"
#endif
or you can use it in the code to make it cope with either:
#if !_ELFUTILS_PREREQ (0, 138)
// work around dwfl_module_build_id bug
id_vaddr -= id_len;
#endif
I have implemented both of these on the roland/bug-compat branch at
git://git.fedorahosted.org/git/elfutils.git (which see), browse at:
http://git.fedorahosted.org/git/?p=elfutils.git;a=shortlog;h=refs/heads/r...
I don't propose to implement any bug-compatible wrapper for
dwfl_module_relocate_address. (You can use _ELFUTILS_PREREQ to optimize
out your workaround when it's not required, but the workaround for that is
correct for new or old, unlike the workaround for dwfl_module_build_id.)
Systemtap folks, please tell me if this covers your issues.
Elfutils folks, please tell me if these changes seem OK.
Thanks,
Roland
14 years, 9 months
roland/pending review
by Roland McGrath
Please review the libelf changes on roland/pending. This fixes a case
where a suspicious input file makes eu-strip crash.
The libelf bug is that a dubious use of the library can make elf_update
crash in weirdness. The use is quite questionable, but does not involve
any bad pointers or sizes (just overlap in sh_offset), and so should not
cause wild memory clobberation. The alternative would be to make
elf_update barf with graceful error return for this use, I think.
The other fixes I didn't think you might mind I pushed onto master.
I think this is everything we'll need for an 0.138 release RSN.
In git, the propagate can be done with:
git checkout master
git pull origin roland/pending
git push
commit 534ad315b34c1fcce8f51ca8762fed1044dbaa47
Author: Roland McGrath <roland(a)redhat.com>
Date: Thu Dec 11 21:23:44 2008 -0800
Comment fix in last change.
commit 75b07c00481fa85152fed79a5c5132b09da49d32
Author: Roland McGrath <roland(a)redhat.com>
Date: Thu Dec 11 21:15:09 2008 -0800
Don't crash in fill code when sh_offset layout is out of order.
Thanks,
Roland
14 years, 9 months
Status 2008-12-15
by Petr Machata
= Work done last week:
* I spent around 80% of time on elfutils work this week.
* Read roadmap and a write-up about dwarflint.
* Merge monotone trunk into my sharing & threads branches. Fix what
got broken, commit.
* Initial work on dwarflint. Current status is as follows:
- It loads .debug_abbrev, doing checks for validity of various
fields along the way. LEB128s are checked for overflow. All reads are
checked that they don't pass outside the section data.
- It uses this abbrev table to load .debug_info. Again, if a range
of allowed values is given, fields are validated. Invalid and unused
abbrev codes are detected. DIE references are checked to point to
actual DIE beginnings, as opposed to a bunch of bytes that happens to be
interpretable as a DIE. DIEs whose abbreviation has_children are
checked to actually have non-zero children.
- String references are marked and holes of unused bytes in
.dwarf_str are reported.
- Most of that can't be considered tested. I tried a couple
debuginfo files (emacs, libX11, libsc680lx from OpenOffice). debuginfo
files often seem to contain unused .debug_str bytes. Non-zero DIEs
followed by zero DIEs are reported for libsc680lx, which is suspicious
and likely a bug in lint. False positives of this type will have to be
fixed as the producer development proceeds.
- The implementation should be reasonably snappy for normal-sized
files. I tried to lint OpenOffice's libsc680lx, a 130MB debuginfo file,
and it takes a bit under six seconds on my 2G core2 machine; so there
may be a room for improvement, but I don't think it's pressing.
- The code sits on pmachata/dwarflint branch. My thinking is that
when the producer branch is created, this branch gets merged in and
deleted, and further development will take place on producer branch.
= Work scheduled for next week:
* My plan for this week doesn't leave room for any serious elfutils
work at all. If something changes, I'll work on:
- dwarflint fixes
- correct DW_AT_sibling attribute value
- checks of other sections (aranges, ranges, locs)
= Holidays:
* My plan is to have the last day this Friday (19th), and come back
5th January. But odds are good that I'll grow bored and work on
something anyway.
PM
14 years, 9 months
elfutils status 2008-12-15
by Roland McGrath
Sorry this report is late (breaking my own rules!). I didn't make as much
progress as I'd planned since last Thursday or so because I've gotten sick.
Last week:
Actual time on elfutils last week: 40%
* git migration
-> fedorahosted magic switched; commits now on git (mtn still accessible)
* libelf crash fixes
** some on master
** some on roland/pending
* libdwfl fix for stap: dwfl_module_relocate_address
* created F11 feature page
-> https://fedoraproject.org/wiki/Features/DebugInfoRevamp
unfinished, not filed as official F11 candidate feature
* more RoadMap content, new wiki pages:
** DwarfProducer
** DwarfCmp
** DwarfInterObject
** DwarfRelocs
** DebugInfo
** RpmDebugInfo
This week:
Expected time on elfutils this week: 60%
* script fiddling for unpacking all rawhide debuginfo rpms
* review dwarflint code
* 0.138 release
** coordinate bug compat issues with stap
* more RoadMap content
** C++ interface outline
** DwarfProducer interface outline
** task/milestone list
** testing plan
* more content on F11 feature page
* start on C++ interface
14 years, 9 months
.debug files for testing
by Roland McGrath
My plan for testing of various things along the way is that we will run
tests on all the data from .debug files in the Fedora rawhide distribution.
Pretty soon we will have dwarflint enough there that we can test it
exhaustively on the data. (As well as crashes or false positives, it will
be interesting to see if it finds any true positives and look into those.)
I have a local mirror of rawhide binaries on hand, and enough spare disk
space. So I've unpacked all the debuginfo files. It takes up about 32G
for x86_64 alone, and about 123G for x86_64,i386,ppc,ppc64 all together.
Today that's 16460 rpms containing 120458 .debug files we get to test on.
Below is the script I used to extract the files. I used it like:
$ cd /net/pa/mirror/fc/development/ppc64/debug
$ ls | xargs sh extract-rpm /home/tmp
where I cd'd into the debug/ subdir of the distro tree,
and /home/tmp is where I have lots of free space. It gets
populated with directory trees like:
0xFFFF-debuginfo-0.3.9-2.fc9.ppc64/usr/lib/debug/usr/bin/0xFFFF.debug
0xFFFF-debuginfo-0.3.9-2.fc9.ppc64/usr/lib/debug/usr/bin/logotool.debug
etc.
If you have a local mirror and the space to spare, do this locally to have
the files on hand to try out easily. I'll keep my unpacked copy around too
and whenever we hit a milestone worthy of thorough testing, we can throw a
test machine at chugging through all the files.
As it develops, we should probably cull some wisdom for wiki pages
about testing.
Thanks,
Roland
#!/bin/sh
dest=$1; shift
for rpm; do
name="${rpm%.rpm}"
if [ "$name" = "$rpm" ]; then
echo >&2 "not .rpm: $rpm"
continue
fi
name="${name##*/}"
if [ -d "$dest/$name" ] && ! rmdir "$dest/$name" 2> /dev/null; then
echo >&2 "$name already there"
continue
fi
mkdir -p "$dest/$name" || exit
rpm2cpio "$rpm" |
(cd "$dest/$name"; cpio --quiet --extract --no-abs --make-dir ||
echo >&2 "FAILED: $name")
done
14 years, 9 months
live git
by Roland McGrath
Since I haven't heard a peep about it, I've gone ahead with the git migration.
The fedorahosted magic all points to git now.
gitweb URL
http://git.fedorahosted.org/git/?p=elfutils.git;a=summary
r/o git URL
git://git.fedorahosted.org/elfutils.git
http://git.fedorahosted.org/git/elfutils.git
r/w git URL
ssh://git.fedorahosted.org/git/elfutils.git
The monotone access is still there until further notice, but let me know as
soon as you no longer care. The mtnelfutils group was renamed gitelfutils,
so everyone who had ssh-mtn access before has ssh-git access now (also).
Since there weren't any trunk commits since I did the test migration,
I just used that same git repo to start with. On top of that I've done one
git commit, adding a .gitignore file.
Please ACK that the git repo looks OK to you. If we need to rerun the
migration, we can still do it now. But soon we'll have fresh commits
waiting to come in and should be thoroughly done and fully committed to
using the new repo (no pun intended).
I just now noticed that tailor didn't migrate the named tags. But I
applied a little brute force and shell scripting, and made the right
correlations and pushed those tags. Note that migrated-from-mtn tags are
now "unsigned" tags, i.e. just refs/tags/* names for the revs. For future
release tags, we should use gpg-signed tags (git tag -s).
I also noticed that for branch mtn propagates that had to do merging, it
has produced a single git commit for the merge to trunk. For ones where it
just re-cert'd the rev, it seems to have the individual commit history.
So, for many of my .roland.pending merges the trunk hadn't changed and so
there is complete history. But for others, and for the .pmachata.threads
merges, there is just one trunk commit as if all the work merged from the
branch were applied as one big patch.
That all sucks a bit, but I have no interest in fighting more with tailor.
It's verified by the way I did my brute-force tag-migrating script that the
release tag revs had identical contents to their mtn tag counterparts. For
the detailed history in between, I trust tailor about as far as I can throw
a third of it and have the rest splatter around on the ground, or something.
But, feh. If you don't care real real hard, then I don't care at all.
The old mtn db's are always around somewhere. Let's get on with hacking
the code instead of hacking the repo setup.
If you have a venerable old checkout with miscellanea in it like I do,
then you can do:
git clone ssh://git.fedorahosted.org/git/elfutils.git eu-git
(cd eu-git; git reset --hard 12af5a28) # matches mtn rev 08bfa56b
mv _MTN _MTN~
mv eu-git/.git .
git pull
In my .git/config I have:
[remote "origin"]
url = ssh://git.fedorahosted.org/git/elfutils.git
fetch = +refs/heads/*:refs/remotes/origin/*
push = +refs/heads/*:refs/heads/*
push = +refs/tags/*:refs/tags/*
The "push" lines make it push everything I define locally up
when I use "git push origin" (or plain "git push"). If you use
that, be sure not to litter your local repo with heads or tags
you didn't mean to publish.
I have migrated my two live branches using the mtn diff | git apply --index -p0
method. That is:
git checkout -b portable origin/master
mtn --db ~/.monotone/db/elfutils.mtn \
diff -r h:com.redhat.elfutils \
-r h:com.redhat.elfutils.roland.pending |
git apply --index -p0
That "checkout" line makes it so that on the branch just created, plain
"git pull" merges from origin/master after fetching it via ssh. All it
does is put some lines in .git/config, which you can just edit by hand
to set other branches to implicitly merge some branch.
The convention I've used is that my branches are roland/name (both on the
server and in my local git repo). I figure we should all use this
convention (fedora account name), and for any shared branch names outside
the username/ space we post here and get consensus on the use of the branch.
I don't really expect anyone else to touch the "portable" branch, but I
made it shared. It's what produces elfutils-portability.patch in the
Fedora rpms and is used to build on RHEL[34] or similarly old systems.
Thanks,
Roland
14 years, 9 months
dwarf plans
by Roland McGrath
I've added a fair bit of brain dump to the wiki, hung off
https://fedorahosted.org/elfutils/wiki/RoadMap
Please look at the DwarfProducer stuff especially.
The relocs stuff and the interobject stuff is all for later on.
The interobject/archive stuff is especially unbaked, and won't
need to be a coherent plan for a while yet.
The concrete details of the producer interfaces are not there at all, but I
think will be a moderately obvious follow-on to the C++ interfaces on the
reading side. The least-determined part of that is the typing plan for
form (attr value) interfaces, which will also need to tie into reloc
support. I will write up more about those interface details, probably next
week.
But please make sure the overall design stuff I wrote about the producer
makes sense (and let me know if you disagree with anything).
Along with that, I will start turning this into specific tasks and
milestones with testing criteria.
Thanks,
Roland
14 years, 9 months