[cobbler] How to speed up triggers
alastair at alastair-munro.com
alastair at alastair-munro.com
Sat Aug 9 19:39:08 UTC 2014
Hi
Just wanted to share something with the list. When creating triggers to
say run a 'cobbler replicate' via ssh, etc, to remote cobbler servers,
the web page waits for the ssh at the remote end to complete. This
produced quite a delay if you are say updating a system and waiting for
the webpage to refresh. The reason for this is it waits for the ssh to
complete, even if you background it or nohup/at/etc it.
I found a way to daemonize the remote command. Using daemon from here:
http://libslack.org/daemon/. This daemonizes the ssh command, thus
giving an instant response. On centos/rhel 6, just install the rpm.
The other nice thing about daemon is the --name=something switch; daemon
will only allow the command to run if there is not another daemon
running with the same name. Thus you could put the same command in cron
to run every so often on the slave, to ensure its kept up to date. This
means the cron and trigger runs would never clash, or subsequent cron
runs would not clash with an earlier run. For example if you add a new
distro, the replicate command could take a while to run to rsync across
the new distro and the next cron run may run into it.
From our setup we have one master and three slaves. In effect one
cobbler server per site/data center. This means no installs across wan
links and our server builds are always setup with their yum repos in the
local site. However we have simple admin; only one cobbler web and a
standard config across all servers. Only have to remember to set the
server override for systems for the local site cobbler server. We do a
mix of pxe and generated.iso builds (both are quite easy)
# mkdir /usr/local/cobbler
# cd /var/lib/cobbler/triggers/add/system/post/
# cat sync-slaves.sh
#!/bin/bash
# A Munro 6 Aug 2014: sync slaves
# make sure you set up ssh keys for this...
# space delimited list of slaves
SLAVES="gb-wat-svv-0600 us-ham-svv-0600 us-lou-svv-0600"
MASTER=cobbler
SSH="ssh -oConnectTimeout=2 -oStrictHostKeyChecking=no
-oUserKnownHostsFile=/dev/null -oBatchMode=yes -oLogLevel=quiet"
LOG=/var/log/cobbler/triggers.log
[ -f $LOG ] && rm -f $LOG
for h in $SLAVES
do
# slow way:background so they are done in parallel
# however ssh still waits for all the commands to complete
# $SSH $h "cobbler replicate --master=$MASTER --systems=*
--profiles=* --prune" 2>&1 >> $LOG &
# using daemon from http://libslack.org/daemon is fast and
instantaneous. The run gets daemonized remotely and returns instantly
$SSH $h "daemon --name=cob /usr/local/cobbler/sync-master.sh" 2>&1
>> $LOG &
done
# chmod u+x sync-slaves.sh
On the slave:
# cat /usr/local/cobbler/sync-master.sh
cobbler replicate --master=cobbler --systems=* --profiles=* --prune
# chmod u+x /usr/local/cobbler/sync-master.sh
More information about the cobbler
mailing list