Hi Roland,
there is a proven build-id 3c3ec6b5aee374a2944f91e07d57baba05f37c57 duplicity
for openoffice.org-ure-3.3.0-20.2.fc14.i686 and some unknown build/version of
the same library/package. IIRC you have found such bug but I cannot find
a fix, could you give a pointer to the fix?
Thanks,
Jan
On Wed, 16 Feb 2011 10:10:08 +0100, Michal Toman wrote:
So far I was troubleshooting just ooo-calc, therefore:
sha256sum
022e897c38fecd089b88dcb7f74c7d4d592feb03ba00583deb4dce189b7745d1 ooo-calc/coredump
The problem #1 is that during
gdb -c ./coredump
it prints
Missing separate debuginfo for the main executable file
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c
which is described in
[Bug 641377] hardlinks for new /usr/lib/rpm/find-debuginfo.sh
https://bugzilla.redhat.com/show_bug.cgi?id=641377
and trying any backtraces without resolving this problem makes no sense.
We will see how fast gets resolved the issue with hardlinks above, besides
/usr/lib/rpm/find-debuginfo.sh fix deployment it will require also rebuild of
the target packages and a GDB extensions for it. So far we can workaround it
for further testing by:
# mv /usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c-orig
# mv /usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c.debug
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c.debug-orig
# ln -s ../../../openoffice.org3/program/scalc.bin
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c
# ln -s ../../usr/lib/openoffice.org3/program/scalc.bin.debug
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c.debug
# ls -l /usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c*
lrwxrwxrwx 1 root root 42 Feb 19 11:15
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c ->
../../../openoffice.org3/program/scalc.bin*
lrwxrwxrwx 1 root root 40 Feb 16 12:40
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c-orig ->
../../../broffice.org3/program/scalc.bin
lrwxrwxrwx 1 root root 53 Feb 19 11:16
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c.debug ->
../../usr/lib/openoffice.org3/program/scalc.bin.debug
lrwxrwxrwx 1 root root 51 Feb 16 12:40
/usr/lib/debug/.build-id/6c/615ac474cbd79ca02f58b440ac3bfc3d648d0c.debug-orig ->
../../usr/lib/broffice.org3/program/scalc.bin.debug
The problem #2 is that still even afterwards one gets with `gdb -c core' lines
like:
warning: .dynamic section for "/usr/lib/openoffice.org/ure/lib/libuno_sal.so.3"
is not at the expected address (wrong library or version mismatch?)
The part "wrong library or version mismatch?" is important there (there exists
also a different but similar GDB message).
This proves the on-disk library does not match the library used during runtime
as its .dynamic section / PT_DYNAMIC segment addresses modulo PAGE_SIZE do not
match (are not congruent).
Just both this core file and my local libuno_sal.so.3 from
openoffice.org-ure-3.3.0-20.2.fc14.i686 have the same build-id:
3c3ec6b5aee374a2944f91e07d57baba05f37c57
/usr/lib/debug/.build-id/3c/3ec6b5aee374a2944f91e07d57baba05f37c57 ->
../../../openoffice.org/ure/lib/libuno_sal.so.3*
= /usr/lib/openoffice.org/ure/lib/libuno_sal.so.3
/usr/lib/debug/.build-id/3c/3ec6b5aee374a2944f91e07d57baba05f37c57.debug ->
../../usr/lib/openoffice.org/ure/lib/libuno_sal.so.3.debug
= /usr/lib/debug/usr/lib/openoffice.org/ure/lib/libuno_sal.so.3.debug
eu-readelf -n /usr/lib/openoffice.org/ure/lib/libuno_sal.so.3
Build ID: 3c3ec6b5aee374a2944f91e07d57baba05f37c57
eu-readelf -n /usr/lib/debug/usr/lib/openoffice.org/ure/lib/libuno_sal.so.3.debug
Build ID: 3c3ec6b5aee374a2944f91e07d57baba05f37c57
dd 2>/dev/null if=ooo-calc/coredump bs=1 skip=$[0xf104] count=20|od -An -txC|tr -d
'\n ';echo
3c3ec6b5aee374a2944f91e07d57baba05f37c57
dd 2>/dev/null if=ooo-calc/coredump bs=1 skip=$[0xf000] count=$[0x1000] >foo
readelf -l foo
DYNAMIC 0x1ca568 0x0039e568 0x0039e568 0x00118 0x00118 RW 0x4
readelf -l /usr/lib/openoffice.org/ure/lib/libuno_sal.so.3
DYNAMIC 0x1c5cb8 0x001c6cb8 0x001c6cb8 0x00118 0x00118 RW 0x4
This proves openoffice.org-ure-3.3.0-20.2.fc14.i686 was not in use for
ooo-calc/coredump and 3c3ec6b5aee374a2944f91e07d57baba05f37c57 exists also in
some other unknown build of /usr/lib/openoffice.org/ure/lib/libuno_sal.so.3 .
repoquery -qf /usr/lib/debug/.build-id/3c/3ec6b5aee374a2944f91e07d57baba05f37c57{,.debug}
openoffice.org-debuginfo-1:3.3.0-20.2.fc14.i686
But there is no different build in the current Fedora repositories - Michal,
could you look it up in your local copies?
BTW resolving the problem #1 is important to use `gdb -c core' instead
of `gdb executable core':
Denys provided for local backtracing
[Crash-catcher] [PATCH] save binary image along with coredump
https://fedorahosted.org/pipermail/crash-catcher/2010-June/000407.html
(a) this is no longer needed for RetraceServer, it has available all the needed
and matching versions.
(b) due to it ABRT is calling GDB with the explicit main executable filename.
ABRT is using
gdb ... -ex 'file /usr/bin/evolution' -ex 'core-file .../coredump'
...
instead of just
gdb ... -ex 'core-file .../coredump' ...
This way it does not use the default `set build-id-core-loads 1'.
The side-effect of build-id-core-loads is that it never loads a binary
with matching filename but non-matching build-id. This would safety check
a non-matching library does not get accidentally loaded.
Apparently it is all broken now when there are build-id duplicates out there.
As the problem #1 fix needs a rebuild of the target package I am not sure if
we will not need a workaround in the meantime. ABRT could send to the server
the executable filename (such as `/usr/bin/evolution' above) read from
/proc/PID/exe at the core dumping time. In such case the
`build-id-core-loads' build-id safety check will not be active even
server-side although it may not be required as the server-side packages should
always be provided in their right versions for the core file.
Apparently it is all broken now when there are build-id duplicates out there.
Thanks,
Jan