Request for Comments: updating RPMs using binary deltas.
by Lamar Owen
Over the past few days on the fedora list, I have participated in a discussion
about 'why can't we distribute patches instead of multimegabyte RPMs for
little changes'.
One of the other participants suggested that this would be better suited to
the -devel list, to which I agree.
To summarize:
1.) A utility for distributing RPM 'deltas' already exists by the name of
rhmask.
2.) Rhmask files may not be optimized for size, since rhmask was designed for
a different problem (how do you distribute patched RPM's for non-open-source
packages: not an issue now, and in fact rhmask is no longer distributed).
3.) A _similar_ thing could be done to reduce both the server storage space
required for updates as well as the download bandwidth for the updates. In
the case of kernel updates, the bandwidth costs have to be huge, particularly
on the server side.
What I am proposing:
1.) Use rsync or something similar to generate an incremental backup of the
patched, unpacked RPM, versus the original distributed RPM (this is not well
known how to do this, but rsync is capable of just copying files that have
changed: see the O'Reilly book 'Linux Server Hacks' hack #42). The delta
must use the original, pristine, as-distributed, RPM as the baseline for
this to not become unwieldy. We would want the file deltas instead of the
whole changed files that an rsync incremental backup would provide.
2.) Package and sign this file, calling it an 'rpmdiff' or something, using
the same basic rpm header structure. This rpmdiff file also carries all the
header info that a traditional errata RPM would carry, including the entire
file list, but would not be directly installable by RPM (unless RPM is
modified to use deltas, but that may be unwieldy).
3.) Locate the pristine RPM. The reason that you can't just install the
rpmdiff and fake out the RPM database as to what files were actually
installed is due to the primary benefit of this technique, and that is when
just a handful of files are changed in, say, the kernel source RPM. The new
kernel-source RPM is installed into a differently versioned directory tree
altogether. This probably would be a 'insert CD#x' type thing, or providing
a URI to the RPM, whether locally stored or on an NFS, FTP, or HTTP server.
We don't want to have to require the user to have all their RPM's installed
in a local disk repository (although that is another thing I'd like to see:
the option of having the installation process install the RPM files
themselves in a local repository that the install tools can then use, but
that's probably not in line with what most people would want.)
4.) The pristine RPM is then 'patched' by the rpmdiff on a file by file level.
The headers from the rpmdiff are used to build the resulting complete RPM
which should be identical with the full errata RPM that would have been
downloaded (except the signature, since the rpmdiff would be signed (and
checked by teh update tools), whereas the reconstructed full rpm would not be
signed, unless the full RPM's signature is transmitted as part of the
rpmdiff). The key to saving space and bandwidth is the use of file-by-file
deltas.
5.) The resulting reconstructed errata RPM is then installed normally, using
up2date or whatever.
6.) There would need to be both a command line 'rpmdiff' tool as well as
up2date support for this to work. Unless the support could be rolled into
rpmlib itself.
7.) The user then enjoys being able to download the updates over dialup and
having a chance to finish in less than five hours, in the case of the
kernel-source update. Dialup users still exist, and they are not going away.
The minor inconvenience of having to insert their original CD's, or have all
the original RPMs stored someplace, IMO outweighs the much larger
inconvenience and cost of the bandwidth problems. If the updates are large
enough one could get compromised in the time it takes to download them. Even
with a high-bandwidth pipe, like a T1 or good DSL, the larger updates,
especially during the testing phases, can take hours to download. I remember
getting beta updates through up2date that took a very long time to download
(and even longer to install, since virtually every package in the whole
distribution had changed, many of which by very few bytes).
8.) The updates repository enjoys being able to service many more users per
hour, since each user takes less time and less bandwidth. And hundreds of GB
are no longer required for a full mirror of all the updates.
9.) Updates get applied quicker, since they take less time. (that would be a
yogi-ism....)
So we trade off the CPU and disk of doing the delta versus the bandwidth of
the download.
Comments?
--
Lamar Owen
Director of Information Technology
Pisgah Astronomical Research Institute
1 PARI Drive
Rosman, NC 28772
(828)862-5554
www.pari.edu
20 years, 5 months
listings package of tetex
by Rui Miguel Silva Seabra
Hi,
In tetex the listings package was removed because of license problems,
but in 2.0.2 it was put back:
* listings package is back (license *is* ok!)
I didn't know there were problems with this package when I made a
document, now I need to compile it again but there's no listings even
though Fedora's tetex is 2.0.2
Is listings still being removed by accident, or is it something else?
Hugs, Rui
--
+ No matter how much you do, you never do enough -- unknown
+ Whatever you do will be insignificant,
| but it is very important that you do it -- Gandhi
+ So let's do it...?
Please AVOID sending me WORD, EXCEL or POWERPOINT attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html
20 years, 5 months
x86_64 linking issues
by Chuck Mead
I am having an issue with several xfce4 packages on x86_64. Libtool
doesn't seem top be able to find and link the packages to the required
libraries. To wit:
*** Warning: linker path does not have real file for library -lXinerama.
*** I have the capability to make that library automatically link in when
*** you link to this library. But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libXinerama and none of the candidates passed a file format test
*** using a file magic. Last file checked: /usr/X11R6/lib64/libXinerama.a
*** Warning: linker path does not have real file for library -lxfcegui4.
*** I have the capability to make that library automatically link in when
*** you link to this library. But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libxfcegui4 and none of the candidates passed a file format test
*** using a file magic. Last file checked: /usr/X11R6/lib64/libXext.so.6.4
*** Warning: linker path does not have real file for library -lxml2.
*** I have the capability to make that library automatically link in when
*** you link to this library. But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libxml2 and none of the candidates passed a file format test
*** using a file magic. Last file checked: /usr/X11R6/lib64/libXext.so.6.4
*** Warning: linker path does not have real file for library -lxfce4util.
*** I have the capability to make that library automatically link in when
*** you link to this library. But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libxfce4util and none of the candidates passed a file format test
*** using a file magic. Last file checked:
/usr/lib/gcc-lib/x86_64-redhat-linux/3.2.3/../../..//libgtk-x11-2.0.so.0.200.4
*** Warning: libtool could not satisfy all declared inter-library
*** dependencies of module libmixer. Therefore, libtool will create
*** a static module, that should work as long as the dlopening
*** application is linked with the -dlopen flag.
And yet the libs it says it cannot find are in ld.so.cache!
[root@stealth root]# strings /etc/ld.so.cache |grep xfce
libxfce4util.so.1
/usr/lib64/libxfce4util.so.1
libxfce4util.so
/usr/lib64/libxfce4util.so
libxfce4mcs-manager.so.1
/usr/lib64/libxfce4mcs-manager.so.1
libxfce4mcs-manager.so
/usr/lib64/libxfce4mcs-manager.so
libxfce4mcs-client.so.1
/usr/lib64/libxfce4mcs-client.so.1
libxfce4mcs-client.so
/usr/lib64/libxfce4mcs-client.so
libxfcegui4.so.1
/usr/lib64/libxfcegui4.so.1
libxfcegui4.so
/usr/lib64/libxfcegui4.so
...and they are of the correct type:
/usr/lib64/libxfce4mcs-client.so.1.0.1: ELF 64-bit LSB shared object,
AMD x86-64, version 1 (SYSV), not stripped
/usr/lib64/libxfce4mcs-manager.so.1.0.1: ELF 64-bit LSB shared object,
AMD x86-64, version 1 (SYSV), not stripped
/usr/lib64/libxfce4util.so.1.0.0: ELF 64-bit LSB shared object,
AMD x86-64, version 1 (SYSV), not stripped
/usr/lib64/libxfcegui4.so.1.0.6: ELF 64-bit LSB shared object,
AMD x86-64, version 1 (SYSV), not stripped
Any ideas?
--
csm
Lunar Project Leader
Disclaimer: "I am not a curmudgeon! No... really..."
Addendum: "Bwahahaha! Fire up the orbital mind-control lasers!"
20 years, 5 months
Re: Request for Comments: updating RPMs using binary deltas
by Jef Spaleta
Lamar Owen wrote:
> Available can mean the install media; it could mean a few GB of space on the
> user's HD (if they chose to install a local repository of the RPMs that they
> installed (which would have to be updated as new RPMs are installed, or be a
> full copy one)). It could mean a download of the original off the update
> server if the user just simply cannot find the original RPM, in which case
> the advantage is negated. They should learn to keep the CD or other local
> repository around anyway to be able to roll back errant updates.
Whoa!!!!! are you saying that you actually expect people to KEEP the
install media around or to KEEP a local repository ondisk. Feel free to
come in to #fedora on irc, and try to TEACH users the lessons you want
them to LEARN. User don't LEARN, if you really think keeping a locally
cached repo or iso images on hand is needed, the distribution is going
to have to ENFORCE that idea on users....and somehow i just don't see a
lot of users being happy about having to have 3 Core
isos + ALL the possible future packages from Extras and Alternatives
hanging around eating up local disk space just to install tiny updates
that may or may not ever materialize. People don't necessarily keep full
install media knocking around..you can't tell them its a good idea, but
they arent going to do it without some arm twisting. People are as
likely to 'borrow' someone's set of install media as they are to get
their friend to burn them a copy.
-jef"I'll learn 'em, I'll learn 'em good"spaleta
20 years, 5 months
RE: initscripts: Can it restart a service automatically if onecrashes?
by Erik LaBianca
>
> erik(a)totalcirculation.com ("Erik LaBianca") writes:
>
> > Another possibility is to use daemontools...
>
> Indeed, they are very usefully. But unfortunately, they are
> not free software (like all the other djb stuff) and can not
> be distributed therefore.
>
>
Are you sure daemontools isn't public domain?
This is all I see:
[erik@mises BUILD]$ head -1 daemontools/src/alloc.c
/* Public domain. */
I've heard tell that djb doesn't like binary redistribution of patched
binaries, but I don't see anything about it in the current daemontools
distro. He also has a treatise[1] about how he declaring code to be in
the public domain effectively relinquishes copyright. He does have a
redistribution page indidicating he doesn't want anyone to ship versions
of his software installed in a "non-standard" location, however [2].
Anyway, it looks like there may be some other replacements that might be
suitable alternative. It sure would be nice to have a simpler (read not
sysvinit) way to add "services" to a fedora system included with the
distro.
1. http://cr.yp.to/publicdomain.html
2. http://cr.yp.to/distributors.html
--erik
20 years, 5 months
Re: Request for Comments: updating RPMs using binary deltas.
by Jef Spaleta
Lamar Owen wrote:
> The rpmdiff would be generated by the build process, and then the >
rpmdiff would be uploaded.
I don't think that was really Seth's point. So let me say what i think
his point was. How much extra space do mirrors have to supply to mirror
the rpmdiffs an an OPTION as well as the fully cooked update rpms. Over
the lifetime of Core, how many different kernel updates do you expect to
see rolled out for example? How much space are you really talking about
mirrors providing for the OPTION of using a collection of rpmdiffs over
the accumulated lifetime of a core release. You can not get away from
providing full rpm update packages. I'm certainly not prepared to
entertain the idea that if I have a custom or alternative package
version install that I need to downgrade all the way back to the iso
release version then apply 7 updates. All those extra steps is bound to
add extra fragility.
> Require the CD/ISO/as-released RPM to be used. Check its signature, or even
> its MD5/SHA checksum to make sure. You may get bit anyway with locally
> generated stuff and full RPM updates.
If there is ANY chance of getting bit because you are using custom rpms
or Fedora Alternatives or 3rd party alternatives...that's pretty much
NOT inline with the project structure and objectives as presented so
far. What is the POINT exactly of having rpmdiffs to confuse things when
trying to use Fedora Alternatives and 3rd party repositories.
I'd MUCH rather concentrate on working out better ways for users to get
access to official mirrors of the update rpms so we don't see the aweful
bottleneck we are seeing right now at the fedora.redhat.com download
server, than complicating the alternative packages situation by putting
efforting into an OPTIONAL rpmdiff idea that can not be garunteed to be
SAFE and EFFECTIVE for people who use custom or aternative rpms. I
personally think that Fedora Core's default set of packages satisfy the
needs perfectly of exactly 3 users in the userbase. I fully expect
everyone but those 3 people to feel a NEED to build or find an
alternative package for at least one Core component.
-jef
20 years, 5 months
The spec file
by Casey James Price
Hi,
I was wondering if anyone has a known working spec file creation script of
spec template. I've seen some templates before, but I am looking to find
one that works for most packages without major changes.
Such a template I have found is here:
Summary :
Name :
Version :
Release :
License : GPL
Group :
Vendor :
Packager : Casey Price <linuxproject(a)aws-sj.com>
Source0 :
Source1 :
Patch0 :
Patch1 :
URL :
BuildRoot : %{_tmppath}/%{name}-buildroot
#Requires :
%description
%prep
rm -rf %buildroot
%setup -q
%build
%configure
make %{_smp_mflags}
%install
rm -rf %{buildroot}
%makeinstall
%clean
rm -rf %{buildroot}
%post -p /sbin/ldconfig
%postun -p /sbin/ldconfig
%files
%defattr(-, root, root)
%doc AUTHORS COPYING Changelog NEWS README TODO
%{_libdir}/*.a
%{_libdir}/*.la
%{_libdir}/*.so
%{_mandir}/man3/*
%{_datadir}/%{name}
%{_bindir}/*
%changelog
* Casey Price <linuxproject(a)aws-sj.com>
- Initial RPM release.
Should it work with the standard ./configure, make and make install
commands? I am still in beginners stages of understanding spec file
creation and composition. Another thing I am curious about, is what
portion of the spec file determines which distribution it will work on, or
if this isn't a factor. Any help is greatly appreciated.
Thanks,
--
Casey Price
20 years, 5 months