Hello Kamil,
Sorry for replying late. I subscribed to this list, but for a reason the emails are still
not being delivered to me. I clicked on some more buttons in the interface right now, so
we'll see. In the mean time, I am using the web interface to reply, so please forgive
the awkward formatting that might come out of this.
One additional thing that I noticed today - when I ran the abipkgdiff
comparison on lyx
package, it used up to 6 GB RAM. That is really high and I believe our test clients
don't have that many available.
Ah.
I think the test systems must have that kind of amount of memory. For most packages, the
memory consumption is reasonable. But for some packages where the combination of ELF
binary size and uncompressed debug info size is big, then, well, the tool is going to
require a lot of memory. For instance, it takes 13+GB of memory to compare the ABI of two
instances of the libjvm.so library.
Really, abipkgdiff uses the usual "static analysis" techniques (it's only
that it acts on binaries rather than source code), so it is expected to have the same kind
of memory requirements.
I don't know how abipkgdiff works and whether it
is able to adjust its memory requirements based on the amount of available system memory
(so that it would use less memory on a less equipped system and still work properly).
abipkgdiff performs static analysis of the binary. For a given library, it loads the type
information of each one of the compilation units that make up the final library. There
can be hundreds of thousands of those types in big libraries. abipkgdiff has to represent
them all to analyse them. It needs to have them all in memory because it analyses the
relationships between all the types. And unlike a compiler that generally sees only one
translation unit at a time, abipkgdiff sees the type information of all the translation
units at a time. Just like what a debugger has to do. And abipkgdiff has to do this
*twice*, because it needs to load two libraries to compare the types of the functions and
variables in that library. So yes, that can take a lot of memory. We are working on
optimizations, of course, but the reality is that there are going to be binaries for which
we'll need a lot of memory to operate on. And right now, we cannot do this
incrementally.
Do you know if this is going to be a problem or not?
If the machine doesn't have enough memory, then yes it's going to be a problem.
Our testing showed that in practice, for the vast majority of packages, the memory
consumption is OK. But there are going to be some packages that will require much much
more memory.
And do you know what is the peak memory
usage for very big packages (e.g. what about comparing two kernel packages)?
As Sinny said in another thread, abipkgdiff doesn't support the kernel yet. So we
haven't tested that.
Is there any solution that we could use to limit the memory
consumption of abipkgdiff?
In general, I'd say there is no good solution beside us optimizing the code more to
make it consume less over time. And this is something I am really interested in doing. I
have started already and I intend to keep doing that as part of the normal development
process.
As Sinny pointed out, in the particular case of the lyx package, the tool should *not*
have performed any comparison because that package only contains an executable. The tool
should only act on libraries, at least in a first instalment. The reason why it tried to
compare two lyx executable is because it mistakenly considered that executable as a shared
library. This mistake can happen on Position Independent Executables, due to a bug in
Libabigail. This bug is fixed now, though. So on lyx, the memory consumption should be
very minimal now, as abipkgdiff won't do anything ;-)
I hope this helps.