I browsed a few of the internals, and it looks like the ppc variant includes a complete source tree in /usr/src/debug

It's normal to have every source file referenced by any binary's DWARF info.

for x in */debug/kernel-debuginfo-2*.rpm; do for y in $x `echo $x | sed s,debuginfo,debuginfo-common,`; do printf "%-75s" $y; rpm -qlp $y | grep '^/usr/src/' | wc -l; done;done

i386/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.i586.rpm 0 i386/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.i586.rpm 9162 i386/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.i686.rpm 0 i386/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.i686.rpm 9165 ppc64/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.ppc64.rpm 0 ppc64/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.ppc64.rpm 8809 ppc/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.ppc.rpm 0 ppc/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.ppc.rpm 8574 x86_64/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.x86_64.rpm 0 x86_64/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.x86_64.rpm 8862

Any reason for ppc being different?

It's not a very different number of source files. Let's wonder about the two biggest files in each rpm.

for x in */debug/kernel-debuginfo-2*.rpm; do for y in $x `echo $x | sed s,debuginfo,debuginfo-common,`; do echo $y:; rpm -qlpv $y | sort -n -k 5,5 | tail -2; done;done

i386/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.i586.rpm: -rwxr--r-- 1 root root 10014436 Jan 21 16:14 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/kernel/fs/xfs/xfs.ko.debug -rwxr-xr-x 1 root root 77065021 Jan 21 16:11 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/vmlinux i386/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.i586.rpm: -rw-r--r-- 1 root root 802044 Oct 9 13:31 /usr/src/debug/kernel-2.6.23/linux-2.6.23.i586/drivers/usb/misc/emi62_fw_s.h -rw-r--r-- 1 root root 875265 Jan 21 15:55 /usr/src/debug/kernel-2.6.23/linux-2.6.23.i586/fs/nls/nls_cp949.c i386/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.i686.rpm: -rwxr--r-- 1 root root 10013800 Jan 21 18:44 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/kernel/fs/xfs/xfs.ko.debug -rwxr-xr-x 1 root root 77048944 Jan 21 18:37 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/vmlinux i386/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.i686.rpm: -rw-r--r-- 1 root root 802044 Oct 9 13:31 /usr/src/debug/kernel-2.6.23/linux-2.6.23.i686/drivers/usb/misc/emi62_fw_s.h -rw-r--r-- 1 root root 875265 Jan 21 15:57 /usr/src/debug/kernel-2.6.23/linux-2.6.23.i686/fs/nls/nls_cp949.c ppc64/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.ppc64.rpm: -rwxr--r-- 1 root root 13748248 Jan 21 17:09 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/kernel/fs/xfs/xfs.ko.debug -rwxr-xr-x 1 root root 67126229 Jan 21 16:30 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/vmlinux ppc64/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.ppc64.rpm: -rwxr-xr-x 1 root root 59098072 Jan 21 17:06 /usr/lib/debug/boot/vmlinuz-2.6.24-0.164.rc8.git4.fc9.debug -rwxr-xr-x 1 root root 59182336 Jan 21 17:06 /usr/lib/debug/boot/vmlinux-2.6.24-0.164.rc8.git4.fc9kdump.debug ppc/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.ppc.rpm: -rwxr--r-- 1 root root 9462912 Jan 21 16:29 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/kernel/fs/xfs/xfs.ko.debug -rwxr-xr-x 1 root root 46764240 Jan 21 16:10 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/vmlinux ppc/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.ppc.rpm: -rwxr-xr-x 1 root root 42107088 Jan 21 16:26 /usr/lib/debug/boot/vmlinuz-2.6.24-0.164.rc8.git4.fc9.debug -rwxr-xr-x 1 root root 43539424 Jan 21 16:26 /usr/lib/debug/boot/vmlinuz-2.6.24-0.164.rc8.git4.fc9smp.debug x86_64/debug/kernel-debuginfo-2.6.24-0.164.rc8.git4.fc9.x86_64.rpm: -rwxr--r-- 1 root root 13447688 Jan 21 16:06 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/kernel/fs/xfs/xfs.ko.debug -rwxr-xr-x 1 root root 57877978 Jan 21 16:04 /usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/vmlinux x86_64/debug/kernel-debuginfo-common-2.6.24-0.164.rc8.git4.fc9.x86_64.rpm: -rw-r--r-- 1 root root 802044 Oct 9 13:31 /usr/src/debug/kernel-2.6.23/linux-2.6.23.x86_64/drivers/usb/misc/emi62_fw_s.h -rw-r--r-- 1 root root 875265 Jan 21 15:54 /usr/src/debug/kernel-2.6.23/linux-2.6.23.x86_64/fs/nls/nls_cp949.c

So the funny bit is that kernel-debuginfo-common on ppc{,64} got the .debug files for two /boot/vmlinu[xz] files. Those ought to be in the kernel-debuginfo and kernel-kdump-debuginfo rpms if they're anywhere.

The placement of those .debug files is controlled via this gem snarfed from the build log:

+ /usr/lib/rpm/find-debuginfo.sh --strict-build-id -p '/.*/2.6.24-0.164.rc8.git4.fc9(-ppc64)?/.*|/.*2.6.24-0.164.rc8.git4.fc9' -o debuginfo.list -p '/.*/2.6.24-0.164.rc8.git4.fc9-?smp(-ppc64)?/.*|/.*2.6.24-0.164.rc8.git4.fc9smp' -o debuginfosmp.list -p '/.*/2.6.24-0.164.rc8.git4.fc9-?PAE(-ppc64)?/.*|/.*2.6.24-0.164.rc8.git4.fc9PAE' -o debuginfoPAE.list -p '/.*/2.6.24-0.164.rc8.git4.fc9-?PAEdebug(-ppc64)?/.*|/.*2.6.24-0.164.rc8.git4.fc9PAEdebug' -o debuginfoPAEdebug.list -p '/.*/2.6.24-0.164.rc8.git4.fc9-?debug(-ppc64)?/.*|/.*2.6.24-0.164.rc8.git4.fc9debug' -o debuginfodebug.list -p '/.*/2.6.24-0.164.rc8.git4.fc9-?xen(-ppc64)?/.*|/.*2.6.24-0.164.rc8.git4.fc9xen' -o debuginfoxen.list -p '/.*/2.6.24-0.164.rc8.git4.fc9-?kdump(-ppc64)?/.*|/.*2.6.24-0.164.rc8.git4.fc9kdump' -o debuginfokdump.list /builddir/build/BUILD/kernel-2.6.23

(The variants that never exist are all represented too through the limited joys of rpm macro magic. On ppc64 only the debuginfo.list and debuginfokdump.list output files are ever used.)

magilla 1528 % echo /boot/vmlinux-2.6.24-0.164.rc8.git4.fc9kdump | egrep -x -e '/.*/2.6.24-0.164.rc8.git4.fc9-?kdump(-ppc64)?/.*|/.*2.6.24-0.164.rc8.git4.fc9kdump' /boot/vmlinux-2.6.24-0.164.rc8.git4.fc9kdump magilla 1529 %

So go figure. I guess I'll have to look into that.

So that's half the problem: why kernel-debuginfo-common has those two files. If that worked the way I expect it to, then vmlinuz-blah.debug would be in kernel-debuginfo and vmlinux-blahkdump.debug would be in kernel-kdump-debuginfo.

Then you'd be noticing that kernel-debuginfo.ppc64 is about twice as big as it should be (and likewise for kernel-debuginfo-kdump). That's because we actually install the unstripped vmlinux in /boot as well as copying it into /usr/lib/debug/lib/modules/. The debuginfo extraction machinery does its thing and so we have both:

/usr/lib/debug/lib/modules/2.6.24-0.164.rc8.git4.fc9/vmlinux /usr/lib/debug/boot/vmlinuz-2.6.24-0.164.rc8.git4.fc9.debug

On ppc, the file in /boot is just the original vmlinux ELF executable, though it's quaintly called vmlinuz. The debugging tools actually only look for /boot/vmlinux-`uname -r`, not vmlinuz. So vmlinuz.debug is never going to be found. On x86, the /boot file is not a usable ELF file at all. So we copy vmlinux into /usr/lib/debug/lib/modules/ for debugging, though on ppc we already have a copy the debugger could use.

It just seems wrong for the upstream debugger tools to look for an ELF file called "vmlinuz", so I don't think that's going to happen. We can't just rename ppc's vmlinuz to /boot/vmlinux-* because of anaconda and that whole can of worms (snakes?). Of course, the kdump kernel is on all machines installed as /boot/vmlinux-* and so the debugger tools are already happy with it, but I guess that's different since anaconda et al don't need to know about it.

One solution is the (untested) patch below. This skips the extra cp of vmlinux when vmlinux is what's being installed in /boot (kdump and ppc). It makes a symlink vmlinux-blah -> vmlinuz-blah in /boot, which through the find-debuginfo magic will produce in kernel-debuginfo a symlink vmlinuz-blah.debug -> vmlinuz-blah.debug in /usr/lib/debug/boot/. /usr/lib/debug/boot/vmlinux-blah.debug is a place the debugger tools will look (even if /boot/vmlinux-blah is not there).

It is probably a better solution not to have a vmlinux.debug but instead keep the simple cp, and avoid the vmlinu[xz].debug files by explicitly stripping the /boot copy of vmlinux.

I will experiment some after I figure out what is going on with the failing separation between debuginfo/debuginfo-common.

Thanks, Roland

--- kernel.spec 21 Jan 2008 21:12:17 -0800 1.374 +++ kernel.spec 23 Jan 2008 22:44:39 -0800 @@ -1317,14 +1317,32 @@ BuildKernel() { install -m 644 .config $RPM_BUILD_ROOT/boot/config-$KernelVer install -m 644 System.map $RPM_BUILD_ROOT/boot/System.map-$KernelVer touch $RPM_BUILD_ROOT/boot/initrd-$KernelVer.img - cp $KernelImage $RPM_BUILD_ROOT/%{image_install_path}/vmlinuz-$KernelVer + if [ -f arch/$Arch/boot/zImage.stub ]; then cp arch/$Arch/boot/zImage.stub $RPM_BUILD_ROOT/%{image_install_path}/zImage.stub-$KernelVer || : fi

if [ "$Flavour" == "kdump" ]; then cp vmlinux $RPM_BUILD_ROOT/%{image_install_path}/vmlinux-$KernelVer - rm -f $RPM_BUILD_ROOT/%{image_install_path}/vmlinuz-$KernelVer + else + cp $KernelImage $RPM_BUILD_ROOT/%{image_install_path}/vmlinuz-$KernelVer +%if "%{kernel_image}" == "vmlinux" + # The /boot image is actually just vmlinux, though called vmlinuz + # for hysterical raisins. The normal debuginfo extraction will + # make a /usr/lib/debug/boot/*.debug file for it, and that preserves + # everything. However, the name that debugging tools know to look + # for is /boot/vmlinux-`uname -r`, not vmlinuz. So make a link. + # This will produce a parallel .debug symlink too. + ln -s vmlinuz-$KernelVer \ + $RPM_BUILD_ROOT/%{image_install_path}/vmlinux-$KernelVer +%else + # The /boot image is not the vmlinux ELF file that a debugger can use. + # So save the unstripped vmlinux file for kernel debugging. +%if %{with_debuginfo} + mkdir -p $RPM_BUILD_ROOT%{debuginfodir}/lib/modules/$KernelVer + cp vmlinux $RPM_BUILD_ROOT%{debuginfodir}/lib/modules/$KernelVer +%endif +%endif fi

mkdir -p $RPM_BUILD_ROOT/lib/modules/$KernelVer @@ -1394,14 +1412,6 @@ BuildKernel() { cp $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/.config $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include/config/auto.conf cd ..

- # - # save the vmlinux file for kernel debugging into the kernel-debuginfo rpm - # -%if %{with_debuginfo} - mkdir -p $RPM_BUILD_ROOT%{debuginfodir}/lib/modules/$KernelVer - cp vmlinux $RPM_BUILD_ROOT%{debuginfodir}/lib/modules/$KernelVer -%endif - find $RPM_BUILD_ROOT/lib/modules/$KernelVer -name "*.ko" -type f >modnames

# mark modules executable so that strip-to-file can strip them