hi, we use mock for local package build, but it's very slow. now we install a new host just for mock with 8core, ram disks etc. it seems it still slow. first of all most of the time mock use only one 1 core of the cpu. is there any way to speed up different part of the mock build process? thanks in advance.
ps. anyway is there any better place to discuss it?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/18/2010 09:38 AM, Farkas Levente wrote:
hi, we use mock for local package build, but it's very slow. now we install a new host just for mock with 8core, ram disks etc. it seems it still slow. first of all most of the time mock use only one 1 core of the cpu. is there any way to speed up different part of the mock build process? thanks in advance.
ps. anyway is there any better place to discuss it?
Mock itself is pretty fast, and when it's doing source compilations, it will automatically use as many CPUs as you have available.
The bottleneck, however, is in autoconf. When building a package, in many cases the lion's share of the build time goes to running through autoconf, which cannot run in parallel (to my knowledge) and uses a great deal of disk I/O.
However, one trick you might consider doing is adding in a --cache-file argument to the call to %configure in your makefiles. This will store a copy of the configure results on-disk in a location of your choosing. Builds after the initial creation of this file will refer to it for answers and will run much faster.
- -- Stephen Gallagher RHCE 804006346421761
Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/
On 01/18/2010 03:49 PM, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/18/2010 09:38 AM, Farkas Levente wrote:
hi, we use mock for local package build, but it's very slow. now we install a new host just for mock with 8core, ram disks etc. it seems it still slow. first of all most of the time mock use only one 1 core of the cpu. is there any way to speed up different part of the mock build process? thanks in advance.
ps. anyway is there any better place to discuss it?
Mock itself is pretty fast, and when it's doing source compilations, it will automatically use as many CPUs as you have available.
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow!
On Mon, 18 Jan 2010, Farkas Levente wrote:
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow!
the tar and gzip are mostly BUILDING the cache.
Mock currently makes a cached copy of the buildroot it just created so it doesn't have to redepsolve and download/install the rpms each time.
So the first time you run it makes a cache. You aren't clearing out the cache each time, are you? That would definitely eat up a lot of time.
-sv
On 01/18/2010 04:10 PM, Seth Vidal wrote:
On Mon, 18 Jan 2010, Farkas Levente wrote:
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow!
the tar and gzip are mostly BUILDING the cache.
no tar and gzip used unpacking root cache.
Mock currently makes a cached copy of the buildroot it just created so it doesn't have to redepsolve and download/install the rpms each time.
but have to run yum each time for the package specific depsolve and yum installs (ps axufww:) --------------------------- _ /usr/bin/python -tt /usr/sbin/mock -r testing-i386 --define rhel 5 --define el5 1 --define dist .el5 --rebuild /home/robot/rpm/SRPMS/xyz-4.2.2-5280.el5.src.rpm root 28319 49.5 0.8 255000 34292 pts/1 D 16:15 0:00 | _ /usr/bin/python /usr/bin/yum --installroot /var/lib/mock/testing-i386/root/ install ccache rsync zip --------------------------- and it much slower then the compile itself. it's very annoying when i try to rebuild only a dozen of packages most of the time. eg in this output: --------------------------- INFO: mock.py version 1.0.2 starting... State Changed: init plugins State Changed: start INFO: Start(/home/robot/rpm/SRPMS/xyz-4.2.2-5280.el5.src.rpm) Config(testing-i386) State Changed: lock buildroot State Changed: clean State Changed: init State Changed: lock buildroot Mock Version: 1.0.2 INFO: Mock Version: 1.0.2 INFO: enabled root cache State Changed: unpacking root cache INFO: enabled yum cache State Changed: cleaning yum metadata INFO: enabled ccache State Changed: running yum State Changed: setup State Changed: build INFO: Done(/home/robot/rpm/SRPMS/xyz-4.2.2-5280.el5.src.rpm) Config(testing-i386) 3 minutes 37 seconds INFO: Results and/or logs in: /var/lib/mock/testing-i386/result --------------------------- the time reach the "State Changed: build" is usually more then all other stuff before it.
So the first time you run it makes a cache. You aren't clearing out the cache each time, are you? That would definitely eat up a lot of time.
of course not:-)
On Mon, 18 Jan 2010, Farkas Levente wrote:
the tar and gzip are mostly BUILDING the cache.
no tar and gzip used unpacking root cache.
How slow are your disks? You're not doing any of this to nfs are you?
but have to run yum each time for the package specific depsolve and yum installs (ps axufww:)
_ /usr/bin/python -tt /usr/sbin/mock -r testing-i386 --define rhel 5 --define el5 1 --define dist .el5 --rebuild /home/robot/rpm/SRPMS/xyz-4.2.2-5280.el5.src.rpm root 28319 49.5 0.8 255000 34292 pts/1 D 16:15 0:00 | _ /usr/bin/python /usr/bin/yum --installroot /var/lib/mock/testing-i386/root/ install ccache rsync zip
and it much slower then the compile itself. it's very annoying when i try to rebuild only a dozen of packages most of the time. eg in this output:
How much of this is network access and how much is disk? B/c I doubt very much that you're cpu bound at all.
-sv
On Monday 18 January 2010, Seth Vidal wrote:
On Mon, 18 Jan 2010, Farkas Levente wrote:
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow!
the tar and gzip are mostly BUILDING the cache.
I've been using lzop as the root cache compressor, it makes a difference over gzip here, especially when compressing the cache, but somewhat also when decompressing it. Add this to /etc/mock/site-defaults.cfg to try it out:
config_opts['plugin_conf']['root_cache_opts']['compress_program'] = 'lzop' config_opts['plugin_conf']['root_cache_opts']['extension'] = '.lzo'
For boxes with multiple cores, pigz (and maybe even pbzip2) might be worth looking into. I haven't tried these myself.
Another thing I've been doing is to increase the max age for various root caches, for example to 60 for all EPEL roots, and 30 for released Fedora versions, i.e. something like this in the respective distro config in /etc/mock/*.cfg:
config_opts['plugin_conf']['root_cache_opts']['max_age_days'] = 60
On Mon, Jan 18, 2010 at 17:30, Ville Skyttä ville.skytta@iki.fi wrote:
On Monday 18 January 2010, Seth Vidal wrote:
On Mon, 18 Jan 2010, Farkas Levente wrote:
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow!
the tar and gzip are mostly BUILDING the cache.
I've been using lzop as the root cache compressor, it makes a difference over gzip here, especially when compressing the cache, but somewhat also when decompressing it. Add this to /etc/mock/site-defaults.cfg to try it out:
config_opts['plugin_conf']['root_cache_opts']['compress_program'] = 'lzop' config_opts['plugin_conf']['root_cache_opts']['extension'] = '.lzo'
For boxes with multiple cores, pigz (and maybe even pbzip2) might be worth looking into. I haven't tried these myself.
why we compress at all? disk space is cheep. can we speed up mock to somehow totally disable compressing root_cache? it used the case but store and restore in an uncompressed way? thanks.
Le 03/04/2011 12:31, Farkas Levente a écrit :
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-(
I use for a while a full /dev/shm mock configuration which is really fast (minimum overhead of ~10"), read:
http://blog.famillecollet.com/post/2011/01/11/new-builder-new-mock-configura...
Of course I need a big /dev/shm (6Gio here)
Remi.
Am 03.04.11 13:17, schrieb Remi Collet:
Le 03/04/2011 12:31, Farkas Levente a écrit :
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-(
I use for a while a full /dev/shm mock configuration which is really fast (minimum overhead of ~10"), read:
http://blog.famillecollet.com/post/2011/01/11/new-builder-new-mock-configura...
Of course I need a big /dev/shm (6Gio here)
Remi.
config_opts['plugin_conf']['ccache_opts']['dir'] = "/dev/shm/ccache.fc14x/"
Oh.. /dev/shm is the wrong place here, I think... Should better be /tmp with a tmpfs mounted on.
On Sun, Apr 3, 2011 at 14:17, Harald Hoyer harald.hoyer@gmail.com wrote:
Am 03.04.11 13:17, schrieb Remi Collet:
Le 03/04/2011 12:31, Farkas Levente a écrit :
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-(
I use for a while a full /dev/shm mock configuration which is really fast (minimum overhead of ~10"), read:
http://blog.famillecollet.com/post/2011/01/11/new-builder-new-mock-configura...
Of course I need a big /dev/shm (6Gio here)
Remi.
config_opts['plugin_conf']['ccache_opts']['dir'] = "/dev/shm/ccache.fc14x/"
Oh.. /dev/shm is the wrong place here, I think... Should better be /tmp with a tmpfs mounted on.
may be some kind of docs for mock config would be useful for everyone...
On Mon, 2010-01-18 at 10:10 -0500, Seth Vidal wrote:
So the first time you run it makes a cache. You aren't clearing out the cache each time, are you? That would definitely eat up a lot of time.
Or running builds a long time apart, as the cache gets aged out. On my system (an overclocked Q6600, 4GB RAM) I can get through a trivial build in about 1:30 in mock if it uses a cached environment; minimum of 5 minutes if it doesn't.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/18/2010 10:05 AM, Farkas Levente wrote:
On 01/18/2010 03:49 PM, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/18/2010 09:38 AM, Farkas Levente wrote:
hi, we use mock for local package build, but it's very slow. now we install a new host just for mock with 8core, ram disks etc. it seems it still slow. first of all most of the time mock use only one 1 core of the cpu. is there any way to speed up different part of the mock build process? thanks in advance.
ps. anyway is there any better place to discuss it?
Mock itself is pretty fast, and when it's doing source compilations, it will automatically use as many CPUs as you have available.
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow!
Well, if you know you're going to be running a number of builds inside the same chroot, you can do this:
mock -r <fedora_version> --init
mock -r <fedora_version> --chroot /path/to/chroot/above <package1> repeat as needed
mock -r <fedora_version> --clean
This way, you incur the heavy setup time only once for all packages.
- -- Stephen Gallagher RHCE 804006346421761
Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/
On Mon, Jan 18, 2010 at 04:05:05PM +0100, Farkas Levente wrote:
the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow!
This is very bad performance, on my 2.5GHZ Athlon dual core with 4 GB RAM a very simple src.rpm (pyhunspell-0.1-2.fc12.src.rpm) rebuilds in 18 seconds with tmpfs plugin enabled.
Another bottleneck could be the connection to your Fedora mirror.
Regards Till
On Mon, 18 Jan 2010 15:38:56 +0100 Farkas Levente lfarkas@lfarkas.org wrote:
hi, we use mock for local package build, but it's very slow. now we install a new host just for mock with 8core, ram disks etc. it seems it still slow. first of all most of the time mock use only one 1 core of the cpu. is there any way to speed up different part of the mock build process? thanks in advance.
If you're seeing long times setting up chroots, make sure you're not disabling the root-caching logic. Once it creates a chroot and tars it up, setup times go from minutes to seconds.
If package build times are your problem, you may want to modify the make command used by the specfiles. Mock just does what the specfile says to do (i.e. it just does an rpmbuild), so you might want to add a '-j16' to the make command line to tell make to manage 16 parallel builds.
Clark
On Monday 18 January 2010, Clark Williams wrote:
If package build times are your problem, you may want to modify the make command used by the specfiles. Mock just does what the specfile says to do (i.e. it just does an rpmbuild), so you might want to add a '-j16' to the make command line to tell make to manage 16 parallel builds.
Many packages already pass %{?_smp_mflags} to make (and pretty much all Fedora packages should, unless there's a problem with it documented in the specfile). So instead of modifying specfiles, one can do something like this in /etc/mock/site-defaults.cfg:
config_opts['macros']['%_smp_mflags'] = '-j3'
On Mon, 18 Jan 2010 18:34:44 +0200 Ville Skyttä ville.skytta@iki.fi wrote:
On Monday 18 January 2010, Clark Williams wrote:
If package build times are your problem, you may want to modify the make command used by the specfiles. Mock just does what the specfile says to do (i.e. it just does an rpmbuild), so you might want to add a '-j16' to the make command line to tell make to manage 16 parallel builds.
Many packages already pass %{?_smp_mflags} to make (and pretty much all Fedora packages should, unless there's a problem with it documented in the specfile). So instead of modifying specfiles, one can do something like this in /etc/mock/site-defaults.cfg:
config_opts['macros']['%_smp_mflags'] = '-j3'
Ooooh, I completely forgot about that. Good idea!
Although, I generally use N*2 for build systems (where N == #cores) so that each core is kept busy.
Clark
On 10-01-18 11:34:44, Ville Skyttä wrote: ...
So instead of modifying specfiles, one can do something like this in /etc/mock/site-defaults.cfg:
config_opts['macros']['%_smp_mflags'] = '-j3'
Unless `rpmbuild --showrc` shows a bad definition for _smp_mflags, you're probably better off letting the macro decide. IME, using a number larger than the actual number of CPUs slows the build a lot, due to cache thrashing. Modern CPUs perform well only with hot caches, and each switch to the "other" build process will replace the cache contents.
The OP said that rpmbuild was using the CPUs properly and ran fast, but earlier parts of mock were not and ran slowly, and take most of the time. He indicated that most of the time is spent extracting the build root from the cache.
On Monday 18 January 2010, Tony Nelson wrote:
On 10-01-18 11:34:44, Ville Skyttä wrote: ...
So instead of modifying specfiles, one can do something like this in /etc/mock/site-defaults.cfg:
config_opts['macros']['%_smp_mflags'] = '-j3'
Unless `rpmbuild --showrc` shows a bad definition for _smp_mflags, you're probably better off letting the macro decide.
Possibly, but this was just to point out that there's no need to modify spec files if you want to have a custom value for it.
Anyway, for single core machines, it's better to set %_smp_mflags manually to -jX (X > 1) in any case, not only for build speed reasons, see https://fedoraproject.org/wiki/Packaging/Guidelines#Parallel_make
On Mon, 2010-01-18 at 15:30 -0500, Tony Nelson wrote:
On 10-01-18 11:34:44, Ville Skyttä wrote: ...
So instead of modifying specfiles, one can do something like this in
/etc/mock/site-defaults.cfg:
config_opts['macros']['%_smp_mflags'] = '-j3'
Unless `rpmbuild --showrc` shows a bad definition for _smp_mflags,
or, more easily:
rpm --eval %_smp_mflags
-
Adam Williamson -
Clark Williams -
Farkas Levente -
Harald Hoyer -
Remi Collet -
Seth Vidal -
Stephen Gallagher -
Till Maas -
Tony Nelson -
Ville Skyttä