I took a break from tearing my hair out in dwarf_output::copier hell to hack on dwarfcmp a bit. I'm glad I did.
I fixed some bugs in the tracker and comparator logic along the way. The main thing I've done is add a new mode of operation in dwarfcmp: with four or more arguments instead of two, you get to choose particular DIE pairs (by hex offset) and compare just those. This is especially useful for comparing two entries you'd think should be consolidated as duplicates and seeing exactly why they won't be.
I've also souped it up to give more information at the end for all the "reference mismatch" cases, not just the "context mismatch" subset. That's key so that when you have a hairy chain of structs pointing to other structs, you see the eventual non-reference difference that breaks the matching of the whole chain. e.g.:
41417 vs 749b7: type=[0x3b27e] vs type=[0x6f485] (<structure_type name="Dwarf_Die"> with reference mismatches) reference 3b27e vs 6f485: 3b298 vs 6f49f: type=[0x3b24f] vs type=[0x6f456] (<pointer_type/> with reference mismatches) reference 3b24f vs 6f456: 3b24f vs 6f456: type=[0x3b18d] vs type=[0x6f450] (<structure_type name="Dwarf_CU"> != <structure_type name="Dwarf_CU"/>) reference 3b18d vs 6f450: 3b18d vs 6f450: byte_size=0x60 vs declaration=1
That shows how everything is the same about 'struct Dwarf_Die' itself, but its member of type 'struct Dwarf_CU *' leads to "struct Dwarf_CU;" in one case and the real "struct Dwarf_CU { ... };" in the other.
I've just pushed all those fixes and changes so you can use that feature. There is a crash in double-free at final destructor time in some cases, which must be some sort of ref-count braino in sharing_stack though I have failed to figure it out at all. The error (valgrind catches it as use-after-free, otherwise a ref-count assert hits) goes away with:
- typedef subr::sharing_stack<die> die_path; + typedef subr::stackish<die> die_path;
in dwarf_tracker. But sharing_stack exists because it vastly reduces the memory consumption of the tracker, so we don't really want that change. We should figure out this crash, but I'm leaving it there for the moment (the crash, not the workaround).
The new feature helped me a lot. Using the dwarflint binary as a test case, I was demoralized about my dwarf_output::copier failing to consolidate duplicate 'class dwarf' entries as the first big example. But it turns out that it was actually correct about all those, after I used the new dwarfcmp feature to investigate them. (I'm still fairly demoralized about how shockingly slow it is, but I haven't really looked into that much yet.)
Many failures to match are due to one of a couple of GCC bugs I discovered. One only applies when using ccache (really it's using -fpreprocessed vs internal cpp in cc1{,plus}), so it gets out of the way in practice if you just don't use ccache.
The wrong decl coordinates bug (523810) bites many more, and exists all over in existing DWARF data. I'm still seeing it because the F-11 compiler still has the bug and that's what I'm using, but Jakub has already fixed it in the F-12 compiler. It's also something we could probably work around fairly easily with some fuzziness options in for comparator and copier/collector.
I've created this tracker bug in Fedora to list all the GCC issues I've come across that will foul up our attempts to do maximal compression: https://bugzilla.redhat.com/show_bug.cgi?id=526977
If anything else like that comes up, be sure to add its Fedora gcc bug # to 526977's blockers list, and also post here about the details.
Thanks, Roland
elfutils-devel@lists.fedorahosted.org