Status 2008-12-08
by Petr Machata
Ok so let's try how these status reports work...
Last week:
- wrap up thread safety work, dump mind-state so that I can bootstrap
myself when I pick up the work in the future:
- the mail that I sent on Thursday
- libdw/README.threads that I used to dump a bit that I realized at
the last moment. I'll use that file to write about any sudden
thread-related aha-moments, if they happen.
Next week:
- prepare for git migration. Sync my work, propagate trunk to sharing
and threads branches, so that the huge-patch-against-HEAD approach of
branch migration actually works.
- start with writing dwarflint
PM
15 years, 5 months
dwarflint
by Roland McGrath
The foundation of the DWARF compression plan is new libdw code to write
DWARF from scratch. It will be essential to have a tool that checks the
format for validity to go along with the development of the writing code.
As elflint is to ELF writing, so dwarflint will be to DWARF writing.
See https://fedorahosted.org/elfutils/wiki/DwarfLint
I think it makes sense to get started on dwarflint immediately. The
"structural" level checks do not have any prerequisites for new library
code or anything else. Coding those is also an excellent way to bone up on
the lowest-level format structures in the DWARF spec, which we need to be
completely immersed in to build the writer.
I think the basic program with the essential structural checks on
.debug_abbrev and .debug_info encoding is just a couple days of work.
I'm thinking Petr can do it this week. Petr?
I'll keep adding more detail to the wiki page, but I think the first draft
there now covers the basic lowest level .debug_info/.debug_abbrev checks
that should be done first.
Thanks,
Roland
15 years, 5 months
regular maintenance tasks
by Roland McGrath
On the RoadMap wiki page I listed a "regular maintenance" section.
These are not things we will schedule long in advance, but are always
there if you want something random to do in idle moments, or something
bite-sized to take a break from bigger hairy work.
There's a link to a bugzilla query for open Fedora elfutils bugs.
Not all merit immediate attention or need anything done. But it's
always a place to look. The feature-request items there should get
fleshed into plans and find their way onto the wiki, either in
RoadMap or on new pages hung off it.
The test coverage can always be better. If we get new contributors
who are timid about tackling big development items, maybe they can
cut their teeth with new tests.
The 0.138 release should probably be made soon. There are several
fixes on the trunk already. It may not need any more changes, but
it merits checking the open bugs. I'm thinking we can do it next
week, more or less first thing after the source control migration.
Thanks,
Roland
15 years, 5 months
development roadmap
by Roland McGrath
It's time to get this show on the road! We are going to start right now
on ramping up elfutils development work. I intend to carry on all the
planning and development discussions on this mailing list, and keep
track of things on https://fedorahosted.org/elfutils/wiki/.
I've started fleshing out immediate work priorities at:
https://fedorahosted.org/elfutils/wiki/RoadMap
I will be adding much more to that over the next 24 hours.
We have the challenging goal of delivering a polished solution for DWARF
size reduction and integrating it into the rpm build system for the
Fedora 11 release. The tentative deadline for when this would ideally
be all well and done is the beginning of March. With a holiday break
coming soon, that is not much more than two months from now.
Currently I think for this we can bank on a >50% time commitment from me
and >50% from Petr Machata. I hope we can get each of those numbers
closer to 80% for a solid push of work in January through February. I
also hope we'll be able to get at least a big chunk of one other person.
But we'll have to see about that, and we can't afford to wait to get started.
We will have to be extremely aggressive to have any chance of making
that delivery target. This will require staying much more closely
organized and schedule-oriented certainly than I've been managing
heretofore. To that end, I'll try to be quite demanding both of myself
and of the active contributors, to communicate frequently and to adjust
quickly to keep focus on the fastest progress where we need it most.
I don't see a need for scheduled meetings or anything, rather than just
email (or IRC when people like to). But I would like to have a
mandatory status update from each active contributor that is both
regular and frequent. (I mean this to apply to those trying to work to
this schedule, to start with just me and Petr. If Ulrich wants to give
us regular updates that would be great, but we don't have his time to
allocate to this roadmap's scheduled items.)
I propose that each active contributor post a required status update to
this list, either at the end of the work week or on Monday.
Specifically, each must post by 23:59 UTC each Monday.
The update should include:
* portion of time planned for elfutils work this week
* last's week's planned portion and actual portion spent
* items to do this week, rough time estimates
* each of last week's status items with percent complete, elaboration
Status postings don't need a lot of detail. They need to be made
consistently every week, and be realistic about where the time went.
The focus is to ensure we stay on track for realistic schedules of the
work. If non-elfutils work is taking more time than expected for
someone this week, we need to notice quickly and see what can be done.
Try to be both realistic and ambitious about what you can do in each
week. Don't worry about over-committing--there's no black mark in
planning something for this week and not getting to it. But don't just
say everything you can think of each week (note to self ;-), we need to
stay sensible about what will actually get finished when.
Ok?
We can start this week, today. I'll post my personal status by 23:59 UTC.
These items should be on everyone's list to do this week:
* prepare for git migration
(merge up personal branches in mtn; look at test git repo)
* use the wiki
* post things on your mind to elfutils-devel
Thanks,
Roland
15 years, 5 months
pmachata.threads branch status
by Petr Machata
Yesterday I've committed a patch that addresses some issues with locking
around callbacks and visitors. Currently it's done like this:
- __libdw_visit_scopes doesn't do any lock handling itself. It needs
at least a read lock, but the caller needs to use the right locking
level with respect to the visitor that is called. It's assumed that
previsit and postvisit may relock.
- When handing the control over to external callback (for example
dwarf_func_inline_instances), the visitor may need to unlock, so that
the callback can use official (locking) elfutils API, and later lock
again. My plan is[1] to use this unlock-callback-relock approach in all
places where callbacks are used.
- The lock that is taken after callback returns is rdlock. That makes
sense, because write lock is typically needed only to init caches and
similar, and that is relatively infrequent operation. However that also
means that the lock level can actually be downgraded. When taking
wrlock in advance ("we are going to need wrlock anyway, so take it right
away"), the care has to be taken not to call functions that can
downgrade the lock this way.
- I've been rich with comments that reason why we don't mind that this
or that function may relock. This is to mark calls where the relocking
analysis has been done. These marks, however, may become invalid as the
code evolves. I don't really know what to do about that, apart from
stripping these marks when the branch stabilizes, and insisting that any
future developers analyse the code again to see if relocking takes
place, and is a problem with respect to their patch.
[1] The "let's not work on thread safety right now" mail came in the
mean time, so actually that /was/ my plan. I'll try to wrap up callback
work before the end of the week. Failing that, I'll use the above notes
to bootstrap myself once I get back to work on this.
PM
15 years, 5 months
pmachata.threads branch status
by Petr Machata
Hi,
I started doing some work on making libdw thread-safe, based on rough
outline I've gotten back in June. Overall approach is to have a lock
per Dwarf, and rdlock/wrlock that as appropriate if work is done with
data structure "descended" from that given Dwarf. [E.g. lock would be
taken like this: rwlock_rdlock (attr->cu->dbg->lock);]
My changes are on pmachata.threads (commits from 8aacf0 on).
PM
15 years, 5 months
migration to git
by Roland McGrath
We're going to migrate the elfutils project sources from mtn to git.
We liked Monotone, but its development has stagnated, it is not
well-supported for project hosting sites, and it is not very widely
used. GIT is well-supported and is familiar to more potential contributors.
I've done a test migration, though we are not switching over quite yet.
Please look at the history and see if it seems to be tolerably intact.
I haven't really checked at all except that the tip checkout is complete.
You can get the test repository with:
git clone git://fedorapeople.org/~roland/eu.git/
Or browse it at:
http://fedorapeople.org/gitweb?p=roland/public_git/eu.git;a=summary
DO NOT base anything off this git repository! I will probably re-run
the migration from scratch again for the real migration. This migration
is up to com.redhat.elfutils rev 08bfa56bdb29649dc9359ffcc6bc498384bebe5b.
Below is the script I used. If someone can do a better migration,
please do. You can fetch a whole monotone database file to start with
from http://roland.fedorapeople.org/tmp/elfutils.mtn if you don't have it.
I fought with tailor a good bit a while ago (and a little bit more last
night), and I've given up on doing an ideal migration with all the
branches. The only thing I figured out how to get working apparently
well is just to track the main branch (com.redhat.elfutils), which folds
in the history of branches that were propagated onto the main branch.
Myself, I don't really care much about migrating the intermediate
history of my side branches. Since tailor seems to suck at it, I say
"oh well". For any unmerged branch I want to keep working on, what I'll
do is get it current with the trunk:
mtn propagate com.redhat.elfutils{,.roland.unwind}
then just save off the patch for the tip of the branch:
mtn diff -r h:com.redhat.elfutils \
-r h:com.redhat.elfutils.roland.unwind > unwind.patch
and then go manually turn that one big patch into one big git commit
for a git branch forking from the git trunk.
Even when we do formally switch the trunk and shared branches over to
git, I'll try to keep the ssh-mtn access to the old db for a while
(indefinitely) so you can get at it if you need (and of course if you
pulled lately you already have your own full copy).
Thanks,
Roland
==============
#!/bin/sh -x
# yum install tailor monotone git
# Copy our real mtn db, don't touch it.
cp -f ~/.monotone/db/elfutils.mtn .
rm -rf work com.redhat.elfutils*
mkdir work && (cd work && git init)
topdir=`pwd`
cat > tailor-eu.cf <<EOF
#!
"""
[DEFAULT]
encoding = "UTF8"
[com.redhat.elfutils]
target = git:com.redhat.elfutils
start-revision = INITIAL
root-directory = $topdir
state-file = com.redhat.elfutils-tailor.state
source = monotone:com.redhat.elfutils
patch-name-format = ""
subdir = work
before-commit = remap_author
[monotone:com.redhat.elfutils]
repository = $topdir/elfutils.mtn
module = com.redhat.elfutils
[git:com.redhat.elfutils]
repository = $topdir/.git
branch = master
overwrite-tags = True
"""
def remap_author(context, changeset):
if changeset.author == 'drepper(a)redhat.com':
changeset.author = 'Ulrich Drepper <drepper(a)redhat.com>'
elif changeset.author == 'roland(a)redhat.com':
changeset.author = 'Roland McGrath <roland(a)redhat.com>'
return True
EOF
tailor -c tailor-eu.cf
15 years, 5 months
Systemtap loading huge kernel file
by Petr Machata
This is an excerpt from the chat that I had with Frank Eigler the other day:
[...]
<fche> machatap, so that's one scenario. another one is all the work
elfutils appears to do when just opening a big fat vmlinux file
<fche> (one built with "allyesconfig" for example)
<fche> it would be nice to find out whether some of that prep work --
applying all those relocations -- is really necessary that early
<fche> because as it is, we page in a huge file, and proceed to dirty
lots of it, so we eat RAM
Before I start digging in elfutils to see what's even going on when
elfutils processes kernel files, does anyone have any thoughts or tips?
Thanks,
PM
15 years, 5 months
pmachata.threads branch status
by Petr Machata
Hi,
I found a couple cases in libelf that look like they need some extra
locking. I assumed that the client has control over the lifetime of Elf
structures, which is to say that I didn't try to fix the race between
releasing memory allocated for struct Elf in one thread, and trying to
lock the same Elf in other thread.
The changes are on pmachata.threads branch.
PM
15 years, 5 months
com.redhat.elfutils.roland.pending
by Roland McGrath
Please review and propagate rev ae00eca3779934fb4228077034dd4c7c313cca61.
Minor stuff.
backends/
* x86_corenote.c: New file.
* Makefile.am (noinst_HEADERS): Add it.
* i386_corenote.c: Include it, use EXTRA_NOTES_IOPERM in EXTRA_NOTES.
* x86_64_corenote.c: Likewise.
libdwfl/
2008-10-07 Roland McGrath <roland(a)redhat.com>
* dwfl_module_build_id.c (check_notes): Fix typo in vaddr calculation.
libebl/
2008-08-01 Roland McGrath <roland(a)redhat.com>
* eblcorenotetypename.c: Handle NT_386_IOPERM.
libelf/
2008-10-06 Roland McGrath <roland(a)redhat.com>
* elf_getarhdr.c (elf_getarhdr): Return NULL when passed NULL.
tests/
2008-09-10 Roland McGrath <roland(a)redhat.com>
* test-subr.sh (LC_ALL): Export it set to "C".
* run-dwfl-addr-sect.sh: Don't do it here.
* run-strings-test.sh: Likewise.
15 years, 5 months