Ok guys,
I know this isn't strictly "Fed" related, but the group might have suggestions.
My use case, the process has a number of remote servers. I'd like to be able to run a bunch of commands on the remote servers as fast as posible. The initial thought was to loop through the different IPAddresses, and SSH into each server, to run the commands.
However, this process was still "sequential" in that each SSH session needs to complete, prior to running the next server/ssh command.
So, my question/need -- is to figure out how to run a ssh as a background on the local server. This approach would/should allow the process to quickly fire off commands to the remote ipaddresses/servers in a much faster/parallel manner.
sortof:
foreach iplist as ipaddress
ssh user1@ipaddress command & (assuming that the '&' runs the local ssh in the background -- it doesn't!)
Running "stuff" on the remote can occur in a "Screen" session, which would allow the ssh process to fire off the remote cmd, but it still doesn't run the local ssh as a "background" process..
Rick has mentioned/discussed the "Screen" function in past threads.
Thoughts/comments....
I have a script that I use to do this for my classroom, have 20 primary machines, and about 30 other systems, but this is just a single command. I like using the putty plink command to do the connection.
Here is a sub-set of the script. Define the comand and then run the command as a background task to the other system. (Actually, 101 is the machines itself), and then 19 other machines in this one. replace real password with {password}.
cmd="hostname ; dnf clean metadata ; dnf update -y " plink -t 192.168.7.101 -l root -pw {password} "$cmd" & plink -t 192.168.7.201 -l root -pw {password} "$cmd" & plink -t 192.168.7.202 -l root -pw {password} "$cmd" & plink -t 192.168.7.203 -l root -pw {password} "$cmd" & plink -t 192.168.7.204 -l root -pw {password} "$cmd" &
I have another script that is usefull to monitor the progress, since running the 3 update scripts works on 54 different systems.
look3 script while [ -n "`ps -ef | grep plink | grep -v "grep plink"`" ] ; do clear ps -ef | grep plink | cut -d- -f 2 | grep -v grep | cut -d\ -f 2 | sort -n | column -c 70 ps -ef | grep plink | cut -d- -f 2 | grep -v grep | wc -l sleep 10 done echo ""
That displays the IP addresses of the machines that are being processed, and the count of home many are currently running. Open another terminal window, and run time ./look3
That shows the status and number, and then reports the total time it took to complete.
Again, plink is part of putty, so need to install putty if it isn't already.
On 4 Mar 2017 at 9:41, bruce wrote:
From: bruce badouglas@gmail.com Date sent: Sat, 4 Mar 2017 09:41:31 -0500 Subject: Running local SSH as background process - possible?? To: Community support for Fedora users users@lists.fedoraproject.org Send reply to: Community support for Fedora users users@lists.fedoraproject.org
Ok guys,
I know this isn't strictly "Fed" related, but the group might have suggestions.
My use case, the process has a number of remote servers. I'd like to be able to run a bunch of commands on the remote servers as fast as posible. The initial thought was to loop through the different IPAddresses, and SSH into each server, to run the commands.
However, this process was still "sequential" in that each SSH session needs to complete, prior to running the next server/ssh command.
So, my question/need -- is to figure out how to run a ssh as a background on the local server. This approach would/should allow the process to quickly fire off commands to the remote ipaddresses/servers in a much faster/parallel manner.
sortof:
foreach iplist as ipaddress
ssh user1@ipaddress command & (assuming that the '&' runs the local ssh in the background -- it doesn't!)
Running "stuff" on the remote can occur in a "Screen" session, which would allow the ssh process to fire off the remote cmd, but it still doesn't run the local ssh as a "background" process..
Rick has mentioned/discussed the "Screen" function in past threads.
Thoughts/comments.... _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
+----------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor Guam Community College Computer Center mailto:mikes@kuentos.guam.net mailto:msetzerii@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ +----------------------------------------------------------+
http://setiathome.berkeley.edu (Original) Number of Seti Units Returned: 19,471 Processing time: 32 years, 290 days, 12 hours, 58 minutes (Total Hours: 287,489)
BOINC@HOME CREDITS ABC 16613838.513356 | EINSTEIN 129817509.788695 ROSETTA 58150514.717077 | SETI 101559023.269076
On Sat, 4 Mar 2017 09:41:31 -0500 bruce wrote:
ssh user1@ipaddress command & (assuming that the '&' runs the local ssh in the background -- it doesn't!)
I have no idea how it could fail to run it in the background, unless perhaps it is stopping because it is asking for tty input. You can fix that with the -n option and/or -f options.
Or perhaps it is asking for credentials. You can fix that with public keys and an ssh-agent so it doesn't have to ask.
On Sat, Mar 4, 2017 at 11:09 AM, Tom Horsley horsley1953@gmail.com wrote:
On Sat, 4 Mar 2017 09:41:31 -0500 bruce wrote:
ssh user1@ipaddress command & (assuming that the '&' runs the local ssh in the background -- it doesn't!)
I have no idea how it could fail to run it in the background, unless perhaps it is stopping because it is asking for tty input. You can fix that with the -n option and/or -f options.
Or perhaps it is asking for credentials. You can fix that with public keys and an ssh-agent so it doesn't have to ask.
-----
Hey Tom,
My bad forgot to mention that my tests have the pub/priv key set so running the SSH test runs without needing to provide the user passwd.
I'll investigate the -n/-f options though..
thanks
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On Sat, Mar 4, 2017 at 1:28 PM, bruce badouglas@gmail.com wrote:
On Sat, Mar 4, 2017 at 11:09 AM, Tom Horsley horsley1953@gmail.com wrote:
On Sat, 4 Mar 2017 09:41:31 -0500 bruce wrote:
ssh user1@ipaddress command & (assuming that the '&' runs the local ssh in the background -- it doesn't!)
I have no idea how it could fail to run it in the background, unless perhaps it is stopping because it is asking for tty input. You can fix that with the -n option and/or -f options.
Or perhaps it is asking for credentials. You can fix that with public keys and an ssh-agent so it doesn't have to ask.
Hey Tom,
My bad forgot to mention that my tests have the pub/priv key set so running the SSH test runs without needing to provide the user passwd.
I'll investigate the -n/-f options though..
thanks
---
hmmm.....
ok...
tried the SSH without the "-t"...
ssh -vvv crawl_user@67.205.151.11 "ls /crawl_tmp;" &
this works -- no prob... returns as expected...
this doesn't..
ssh -vvv -t crawl_user@67.205.151.11 "ls /crawl_tmp;"
so... what's the use of the "-t" and when is it required/not required!
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 04Mar2017 13:34, bruce badouglas@gmail.com wrote:
tried the SSH without the "-t"...
ssh -vvv crawl_user@67.205.151.11 "ls /crawl_tmp;" &
this works -- no prob... returns as expected...
this doesn't..
ssh -vvv -t crawl_user@67.205.151.11 "ls /crawl_tmp;"
Well, it has no trailing "&"...
-t causes the _remote_ end to allocate a terminal, which causes some things to run differently (eg you can use use screen or tmux; most "ls" commands will columnate output on a terminal, etc etc). However, for background/batch commands you usually don't want this at all. Ssh turns this on if attached to a terminal, and doesn't otherwise - that is what gets you samless terminal access with a plain ssh to a remote host.
Short answer: don't use -t.
You also don't want to read from standard input - this is the -n option.
The -f option is like -n, but completes the ssh connection/authentication before backgrounding it. FOr maximum parallelism you don't want -f.
Basicly you want a loop like this:
set -x # trace the shell so you can see what's happening for host in a b c d do ssh -n "$host" "some command here" & done wait
You can compare that with the -f option flavour:
set -x # trace the shell so you can see what's happening for host in a b c d do ssh -f "$host" "some command here" done
The former is what you probably want; it is what I usually do.
The latter does the authentication phase serially, then backgrounds the ssh. It will be slower, but you could check which sshes failed connection/authentication if that is a concern. Note that the background ssh commands with -f are _not_ children of your shell, and you can't "wait" for them as you can in the first instance. (You can block for them by reading their (shared) stdout, but let us not go there.)
The -t option is a red herring here; its use is more special purpose and will only cause you trouble here.
Cheers, Cameron Simpson cs@zip.com.au
Can't Ansible be used to achieve your objective?
Br,
-vu
On Sat, Mar 4, 2017 at 6:28 PM, Cameron Simpson cs@zip.com.au wrote:
On 04Mar2017 13:34, bruce badouglas@gmail.com wrote:
tried the SSH without the "-t"...
ssh -vvv crawl_user@67.205.151.11 "ls /crawl_tmp;" &
this works -- no prob... returns as expected...
this doesn't..
ssh -vvv -t crawl_user@67.205.151.11 "ls /crawl_tmp;"
Well, it has no trailing "&"...
-t causes the _remote_ end to allocate a terminal, which causes some things to run differently (eg you can use use screen or tmux; most "ls" commands will columnate output on a terminal, etc etc). However, for background/batch commands you usually don't want this at all. Ssh turns this on if attached to a terminal, and doesn't otherwise - that is what gets you samless terminal access with a plain ssh to a remote host.
Short answer: don't use -t.
You also don't want to read from standard input - this is the -n option.
The -f option is like -n, but completes the ssh connection/authentication before backgrounding it. FOr maximum parallelism you don't want -f.
Basicly you want a loop like this:
set -x # trace the shell so you can see what's happening for host in a b c d do ssh -n "$host" "some command here" & done wait
You can compare that with the -f option flavour:
set -x # trace the shell so you can see what's happening for host in a b c d do ssh -f "$host" "some command here" done
The former is what you probably want; it is what I usually do.
The latter does the authentication phase serially, then backgrounds the ssh. It will be slower, but you could check which sshes failed connection/authentication if that is a concern. Note that the background ssh commands with -f are _not_ children of your shell, and you can't "wait" for them as you can in the first instance. (You can block for them by reading their (shared) stdout, but let us not go there.)
The -t option is a red herring here; its use is more special purpose and will only cause you trouble here.
Cheers, Cameron Simpson cs@zip.com.au
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 03/04/2017 06:41 AM, bruce wrote:
Ok guys,
I know this isn't strictly "Fed" related, but the group might have suggestions.
My use case, the process has a number of remote servers. I'd like to be able to run a bunch of commands on the remote servers as fast as posible. The initial thought was to loop through the different IPAddresses, and SSH into each server, to run the commands.
However, this process was still "sequential" in that each SSH session needs to complete, prior to running the next server/ssh command.
So, my question/need -- is to figure out how to run a ssh as a background on the local server. This approach would/should allow the process to quickly fire off commands to the remote ipaddresses/servers in a much faster/parallel manner.
sortof:
foreach iplist as ipaddress
ssh user1@ipaddress command & (assuming that the '&' runs the local ssh in the background -- it doesn't!)
Running "stuff" on the remote can occur in a "Screen" session, which would allow the ssh process to fire off the remote cmd, but it still doesn't run the local ssh as a "background" process..
Rick has mentioned/discussed the "Screen" function in past threads.
Yes, screen is useful. Have a look a the "pdsh" command and its companion, "dshbak". I use it a lot to manage our storage systems.
For example, each of the machines involved has an "/etc/machines" file of:
seg1-r0 seg2-r0 seg3-r0 seg4-r0 seg5-r0 seg6-r0
They also all have shared ssh keys. If I use the command:
pdsh -a "ps ax" | dshbak
on one of them, they all will do a "ps ax" command asynchronously and dshbak will sort the output so that the output of each machine is kept together. Similarly,
pdsh -w seg[1,2]-r0 "ps ax" | grep dshbak
will only run the command on the first two machines. You get the idea. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - Tempt not the dragons of fate, since thou art crunchy and taste - - good with ketchup. - ----------------------------------------------------------------------
-
bruce -
Cameron Simpson -
Michael D. Setzer II -
Rick Stevens -
Tom Horsley -
Vojin Urosevic