[cobbler] How to speed up triggers

alastair at alastair-munro.com alastair at alastair-munro.com
Sat Aug 9 19:39:08 UTC 2014


Hi

Just wanted to share something with the list. When creating triggers to 
say run a 'cobbler replicate' via ssh, etc, to remote cobbler servers, 
the web page waits for the ssh at the remote end to complete. This 
produced quite a delay if you are say updating a system and waiting for 
the webpage to refresh. The reason for this is it waits for the ssh to 
complete, even if you background it or nohup/at/etc it.

I found a way to daemonize the remote command. Using daemon from here: 
http://libslack.org/daemon/. This daemonizes the ssh command, thus 
giving an instant response. On centos/rhel 6, just install the rpm.

The other nice thing about daemon is the --name=something switch; daemon 
will only allow the command to run if there is not another daemon 
running with the same name. Thus you could put the same command in cron 
to run every so often on the slave, to ensure its kept up to date. This 
means the cron and trigger runs would never clash, or subsequent cron 
runs would not clash with an earlier run. For example if you add a new 
distro, the replicate command could take a while to run to rsync across 
the new distro and the next cron run may run into it.

 From our setup we have one master and three slaves. In effect one 
cobbler server per site/data center. This means no installs across wan 
links and our server builds are always setup with their yum repos in the 
local site. However we have simple admin; only one cobbler web and a 
standard config across all servers. Only have to remember to set the 
server override for systems for the local site cobbler server. We do a 
mix of pxe and generated.iso builds (both are quite easy)

# mkdir /usr/local/cobbler
# cd /var/lib/cobbler/triggers/add/system/post/
# cat sync-slaves.sh
#!/bin/bash

# A Munro 6 Aug 2014: sync slaves
# make sure you set up ssh keys for this...

# space delimited list of slaves
SLAVES="gb-wat-svv-0600 us-ham-svv-0600 us-lou-svv-0600"
MASTER=cobbler
SSH="ssh -oConnectTimeout=2 -oStrictHostKeyChecking=no 
-oUserKnownHostsFile=/dev/null -oBatchMode=yes -oLogLevel=quiet"
LOG=/var/log/cobbler/triggers.log

[ -f $LOG ] && rm -f $LOG

for h in $SLAVES
do
#   slow way:background so they are done in parallel
#   however ssh still waits for all the commands to complete
#    $SSH $h "cobbler replicate --master=$MASTER --systems=* 
--profiles=* --prune" 2>&1 >> $LOG  &
#   using daemon from http://libslack.org/daemon is fast and 
instantaneous. The run gets daemonized remotely and returns instantly
     $SSH $h "daemon --name=cob /usr/local/cobbler/sync-master.sh" 2>&1 
 >> $LOG  &
done

# chmod u+x sync-slaves.sh

On the slave:

# cat /usr/local/cobbler/sync-master.sh
cobbler replicate --master=cobbler --systems=* --profiles=* --prune

# chmod u+x /usr/local/cobbler/sync-master.sh



More information about the cobbler mailing list