I hope you're really not saying that, if I request to install
package
foo, that depends on bar, it will also download headers for baz, a
totally unrelated package. I can see that we'd need headers for foo
and bar, but not for baz. I thought the point of the xml files and
the info on provides, filelists, etc, was precisely to enable the
depsolver to avoid having to download the headers for every package.
Just so we don't go off into deeply uninformed space:
yum 2.0.X downloaded all the headers in the headers directory that it
did NOT have installed. It figured this out by reading header.info. This
file stored nevra + rpm location. So yum 2.0.X downloaded this file to
see what new headers it needed, downloaded them, then got on with the
process at hand.
I'm wondering if it would be possible for a depsolver to create
a
(smaller) .hdr file out of info in the .xml files, and feed that to
rpmlib for transaction-verification purposes. This would enable it to
skip the download-header step before downloading the entire package.
Talk to Paul Nasrat - he was working on that a while ago but I think he
got stuck in some rabbit hole debugging something.
Definitely. But couldn't we perhaps do it by intelligently
filtering
information out of the rpm header and, say, generating a single
archive containing all of the info needed for depsolving and for
rpmlib's transaction verification?
you can't do that b/c file conflicts CAN NOT be calculated via rpm w/o
having the full header and/or all the file information present.
I was expecting depsolving wouldn't require all the headers. And
from
what I gather from your reply, it indeed doesn't.
it requires all the headers of the packages involved, yes.
Let's consider two scenarios: 1) using up2date with yum-2.0
(headers/)
repos (whoever claimed up2date supported rpmmd repodata/ misled me :-)
and 2) using yum-2.1 (repodata/) repos.
1) yum 2.0
16MiB) initial download, distro's and empty updates's hdrs
8MiB) daily (on average) downloads of header.info for updates,
downloaded by rhn-applet, considering an average size of almost
30KiB, for 40 weeks. (both FC2 and FC3 updates for i386 have a
header.info this big right now)
16MiB) .hdr files for updates, downloaded by the update installer.
Current FC2 i386 headers/ holds 9832KiB, whereas FC3 i386
headers/ holds 8528KiB, but that doesn't count superseded
updates, whose .hdr files are removed. The assumption is that
each header is downloaded once. 16MiB is a guestimate, that I
believe to be inflated. It doesn't take into account the
duplicate downloads of header.info for updates, under the
assumption that a web proxy would avoid downloading again what
rhn-applet has already downloaded.
----
40MiB) just in metadata over a period of 9 months, total
2) yum 2.1
2.7MiB) initial download, distro's and empty updates'
primary.xml.gz and filelists.xml.gz
68MiB) daily (on average) downloads of primary.xml.gz, downloaded by
rhn-applet, considering an average size of 250KiB (FC2 updates's
is 240KiB, whereas FC3's is 257KiB, plus about 1KiB for
repomd.xml)
16MiB) .hdr files for updates, downloaded by the update installer
(same as in case 1)
192MiB) filelists.xml.gz for updates, downloaded twice a week on
average by the update installer, to solve filename dep.
----
278.7MiB) just in metadata over a period of 9 months, total
Looks like a waste of at least 238.7 MiB per user per 9-month install.
Sure, it's not a lot, only 26.5MiB a month, but it's almost 6 times as
much data being transferred for the very same purpose. How is that a
win? Multiply that by the number of users pounding on your mirrors
and it adds up to hundreds of GiB a month.
Another factor is that you probably won't need filelists.xml.gz
for
every update. Maybe I don't quite understand how often it is needed,
but even if I have to download it only once a month, that's still
64MiB over 9 months, more than the 40MiB total metadata downloaded
over 9 months by yum 2.0.
yum 2.1.x ONLY DOWNLOADS THE XML FILES WHEN IT NEEDS THEM.
go read the code and stop guessing.
it downloads repomd.xml everytime - that's < 1K.
it downloads primary.xml.gz if the file has changed - that's typically <
1M.
it downloads filelists.xml.gz only when there is a file dep that it
cannot resolve with primary.xml.gz.
I don't know how yum 2.0 did it, but up2date surely won't
even try to
download a .hdr file if it already has it in /var/spool/up2date, so
this is not an issue.
yum 2.0.x certainly DID NOT download a .hdr file it already had. Sheesh,
go read the code, stop making suppositions based on anecdotes.
repodata helps the initial download, granted, but it loses terribly
in
the long run.
only as the number of file deps outside of /etc/* and *bin/* increases.
if you keep the file deps in those paths then repodata is a huge win.
-sv