On Mon, 7 Feb 2005, Dag Wieers wrote:
On Sun, 29 Jan 2005, Alexandre Oliva wrote:
> On Jan 29, 2005, "Charles R. Anderson" <cra(a)WPI.EDU> wrote:
>
> > Why all the complication when a general-purpose algorithm, rsync,
> > already exists to solve this problem?
>
> It doesn't do very well on compressed files, and there aren't a lot of
> rsync-enabled web proxies out there.
You may like this RFE:
https://devel.linux.duke.edu/bugzilla/show_bug.cgi?id=391
gzip has an --rsyncable option that is missing from the python zlib
implementation. If python/zlib could be taught this, and createrepo
is able to use it, the metadata would be rsyncable (improves
transferspeed for repo maintainers/mirrors).
My repo-generation script now unzips the createrepo metadata, recompresses
with --rsyncable and recreates the repomd.xml after createrepo's run.
I synced the repository.
I then added a single update package to my repository
And then rsynced again (with a 12kB upstream limitation), and got:
fedora/3/en/i386/dag/repodata/
fedora/3/en/i386/dag/repodata/filelists.xml.gz
961353 100% 306.10kB/s 0:00:03 (68, 16.8% of 208800)
fedora/3/en/i386/dag/repodata/other.xml.gz
412335 100% 89.30kB/s 0:00:04 (69, 16.8% of 208800)
fedora/3/en/i386/dag/repodata/primary.xml.gz
668753 100% 154.79kB/s 0:00:04 (70, 16.8% of 208800)
fedora/3/en/i386/dag/repodata/repomd.xml
1128 100% 4.87kB/s 0:00:00 (71, 16.8% of 208800)
and
fedora/3/en/x86_64/dag/repodata/
fedora/3/en/x86_64/dag/repodata/filelists.xml.gz
1466742 100% 273.93kB/s 0:00:05 (86, 20.9% of 208800)
fedora/3/en/x86_64/dag/repodata/other.xml.gz
674072 100% 79.47kB/s 0:00:08 (87, 20.9% of 208800)
fedora/3/en/x86_64/dag/repodata/primary.xml.gz
1086239 100% 141.68kB/s 0:00:07 (88, 20.9% of 208800)
fedora/3/en/x86_64/dag/repodata/repomd.xml
1128 100% 1.46kB/s 0:00:00 (89, 20.9% of 208800)
Which is a between 6.6x and 25.6x speed improvement for something I have
to update every single sync. (Normally these files are synced with a
12kB/sec rate limitation taking on average 1min per file, now only 6secs)
Now imagine what this would do if Yum had a client-side rsync
implementation. zsync may be something to look at.
-- dag wieers, dag(a)wieers.com,
http://dag.wieers.com/ --
[all I want is a warm bed and a kind word and unlimited power]