I have been trying to capture bash pipe errors as follows:
ret1=0; ret2=0; ret3=0; ret4=0; ret5=0
out=$(\ grep "PATTERN" "${SFILE}"; ret1=$? 2>&1 | \ sed $re1 $re2; ret2=$? 2>&1 | \ sort -n; ret3=$? 2>&1 | \ uniq >> "${TFILE}"; ret4=$? 2>&1\ ); ret5=$?
But the return status variables are not changed should any command errors arise.
How could one go about capturing command errors within pipes?
Thanks! Dan
On Thu, 19 Mar 2009, Daniel B. Thurman wrote:
How could one go about capturing command errors within pipes?
The variable $PIPESTATUS is just for that.
S P Arif Sahari Wibowo wrote:
On Thu, 19 Mar 2009, Daniel B. Thurman wrote:
How could one go about capturing command errors within pipes?
The variable $PIPESTATUS is just for that.
I tried it, seems not to work.
Try capturing a sed error when re1 or re2 has a bad regular expression string. I have not been able to catch the error.
On Thursday, Mar 19th 2009 at 14:11 -0000, quoth Daniel B. Thurman:
=>S P Arif Sahari Wibowo wrote: =>> On Thu, 19 Mar 2009, Daniel B. Thurman wrote: =>> > How could one go about capturing command errors within pipes? =>> =>> The variable $PIPESTATUS is just for that. =>> =>I tried it, seems not to work. => =>Try capturing a sed error when re1 or re2 has a bad regular =>expression string. I have not been able to catch the error.
Sorry, but you'll have to supply something more specific. It's possible for a program to have an error but not return a proper exit status.
But so far, my highly scientific test has revealed that all is good in pipe testing land in the Kingdom of Bash.
578 > true | false | true | true 579 > echo ${PIPESTATUS[@]} 0 1 0 0
Steven W. Orr wrote:
On Thursday, Mar 19th 2009 at 14:11 -0000, quoth Daniel B. Thurman:
=>S P Arif Sahari Wibowo wrote: =>> On Thu, 19 Mar 2009, Daniel B. Thurman wrote: =>> > How could one go about capturing command errors within pipes? =>> =>> The variable $PIPESTATUS is just for that. =>> =>I tried it, seems not to work. => =>Try capturing a sed error when re1 or re2 has a bad regular =>expression string. I have not been able to catch the error.
Sorry, but you'll have to supply something more specific. It's possible for a program to have an error but not return a proper exit status.
But so far, my highly scientific test has revealed that all is good in pipe testing land in the Kingdom of Bash.
578 > true | false | true | true 579 > echo ${PIPESTATUS[@]} 0 1 0 0
OK, I found out why I had a problem!
I setup an intentional sed error: $ (re="-e ''s/b/h/'"; echo "boo" | sed $re | echo "done"; echo ${PIPESTATUS}) done sed: -e expression #1, char 1: unknown command: `'' 0
Same thing but with @ pipe status expression $ (re="-e ''s/b/h/'"; echo "boo" | sed $re | echo "done"; echo ${PIPESTATUS[@]}) done sed: -e expression #1, char 1: unknown command: `'' 0 1 0
My problem was that the last command was OK, of which PIPESTATUS was reporting, thus I was not properly capturing the exact status location where the error occurred! duh!
Your example showed me the way!
Thanks! Dan
Daniel B. Thurman wrote:
Steven W. Orr wrote:
On Thursday, Mar 19th 2009 at 14:11 -0000, quoth Daniel B. Thurman:
=>S P Arif Sahari Wibowo wrote: =>> On Thu, 19 Mar 2009, Daniel B. Thurman wrote: =>> > How could one go about capturing command errors within pipes? =>> =>> The variable $PIPESTATUS is just for that. =>> =>I tried it, seems not to work. => =>Try capturing a sed error when re1 or re2 has a bad regular =>expression string. I have not been able to catch the error.
Sorry, but you'll have to supply something more specific. It's possible for a program to have an error but not return a proper exit status.
But so far, my highly scientific test has revealed that all is good in pipe testing land in the Kingdom of Bash.
578 > true | false | true | true 579 > echo ${PIPESTATUS[@]} 0 1 0 0
OK, I found out why I had a problem!
I setup an intentional sed error: $ (re="-e ''s/b/h/'"; echo "boo" | sed $re | echo "done"; echo ${PIPESTATUS}) done sed: -e expression #1, char 1: unknown command: `'' 0
Same thing but with @ pipe status expression $ (re="-e ''s/b/h/'"; echo "boo" | sed $re | echo "done"; echo ${PIPESTATUS[@]}) done sed: -e expression #1, char 1: unknown command: `'' 0 1 0
My problem was that the last command was OK, of which PIPESTATUS was reporting, thus I was not properly capturing the exact status location where the error occurred! duh!
Your example showed me the way!
Thanks! Dan
Oops, not the LAST command but the FIRST command, I think. $PIPESTATUS is the same as $PIPESTATUS[0]
Dan
On 19Mar2009 12:19, Daniel B. Thurman dant@cdkkt.com wrote:
OK, I found out why I had a problem!
I setup an intentional sed error: $ (re="-e ''s/b/h/'"; echo "boo" | sed $re | echo "done"; echo ${PIPESTATUS}) done sed: -e expression #1, char 1: unknown command: `'' 0
[ I'm sure you know everything I say below based on previous posts, but it seems worth saying purely for other readers... ]
Ok, but I strongly recommend you never us a regexp unquoted - the necessary backslash nesting gets nasty real fast. A better way is like this:
re='s/b/h/' sed -e "$re"
or for complex stuff, write a file called "sedf" (for example) containing a full sed script:
s/b/h/ /foo/d /bah/b again
etc. Then:
sed -f sedf
The point here, unrelated to the pipe exit status issue, is to keep the sed stuff easy to write and undamaged by shell substitution.
On the pipe stuff, another approach (less convenient than PIPESTATUS), is a flag approach like you original attempt, but using files:
badness=/tmp/foo$$bad { x || touch $badness; } \ | { y || touch $badness; } \ | { z || touch $badness; } if [ -f $badness ] then rm $badness echo badness happened >&2 fi
It has the advantage of working in non-pipe circumstances (nested subshells, etc).
Cheers,
Cameron Simpson wrote:
On 19Mar2009 12:19, Daniel B. Thurman dant@cdkkt.com wrote:
OK, I found out why I had a problem!
I setup an intentional sed error: $ (re="-e ''s/b/h/'"; echo "boo" | sed $re | echo "done"; echo ${PIPESTATUS}) done sed: -e expression #1, char 1: unknown command: `'' 0
[ I'm sure you know everything I say below based on previous posts, but it seems worth saying purely for other readers... ]
Ok, but I strongly recommend you never us a regexp unquoted - the necessary backslash nesting gets nasty real fast. A better way is like this:
re='s/b/h/' sed -e "$re"
or for complex stuff, write a file called "sedf" (for example) containing a full sed script:
s/b/h/ /foo/d /bah/b again
etc. Then:
sed -f sedf
The point here, unrelated to the pipe exit status issue, is to keep the sed stuff easy to write and undamaged by shell substitution.
On the pipe stuff, another approach (less convenient than PIPESTATUS), is a flag approach like you original attempt, but using files:
badness=/tmp/foo$$bad { x || touch $badness; } \ | { y || touch $badness; } \ | { z || touch $badness; } if [ -f $badness ] then rm $badness echo badness happened >&2 fi
It has the advantage of working in non-pipe circumstances (nested subshells, etc).
Cheers,
Another person sent me private email warning of the double-quotes, but thanks for your information, it is interesting!
This is what I ended up doing, so please give your constructive critiques, if you so like?
TRACKER="Tracker.log" SFILE="Bad_Hosts_File.txt" TFILE="Bad_Hosts_File_$$.txt" pat="EACC" re1="-e '/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/'" re2="-e '/^.*cdkkt.com$/d'" rex="sed $re1 $re2"
echo -en "\033[0;36m>> \033[0;35mExtracting data from:\n \ [\033[0;34m${TRACKER}\033[0;35m and appending to\n \ [\033[0;34m${TFILE}\033[0;35m]\033[0m"
out=$(grep "$pat" "${TRACKER}" | \ eval "$rex" | sort -n | \ uniq >> "${TFILE}"); ret="$?";
if [ "$ret" -gt 0 ] || \ [ "$PIPESTATUS[0]" -gt 0 ] || \ [ "$PIPESTATUS[1]" -gt 0 ] || \ [ "$PIPESTATUS[2]" -gt 0 ] || \ [ "$PIPESTATUS[3]" -gt 0 ]; then echo -e " \033[0;31mFailed.\n \ \033[0;35m-- ErrorStatus:<ret=$ret>,\ PipeStatus:<$PIPESTATUS[@]>\n\ -- out:<$out>\033[0m" echo -e "\033[0;31m>> \033[0;31m$PROG exited.\033[0m\n" exit 1 fi echo -e " \033[0;32mDone.\033[0m";
I guess the ugly part is the multiple $PIPESTATUS[n] in the if statement test block. but perhaps this is the best I can do?
Dan
On 19Mar2009 18:55, Daniel B. Thurman dant@cdkkt.com wrote:
Cameron Simpson wrote:
Ok, but I strongly recommend you never us a regexp unquoted - the necessary backslash nesting gets nasty real fast. A better way is like this: re='s/b/h/' sed -e "$re"
[...]
The point here, unrelated to the pipe exit status issue, is to keep the sed stuff easy to write and undamaged by shell substitution. [...]
Another person sent me private email warning of the double-quotes, but thanks for your information, it is interesting!
This is what I ended up doing, so please give your constructive critiques, if you so like?
Untested, but here is how I would write each of you code pieces were I doing it myself:
TRACKER="Tracker.log" SFILE="Bad_Hosts_File.txt" TFILE="Bad_Hosts_File_$$.txt"
tracker=Tracker.log sfile=Bad_Hosts_File.txt tfile=Bad_Hosts_File_$$.txt
Your quotes are unneeded here. Plenty of people find the "" marks and in similar cases, to use ${foo} instead of $foo elsewhere, but I find the syntactic noise annoying if there's no other necessity.
Also, you should never use UPPER CASE names for script local variables (i.e. that are not to be exported). I give some background why here:
http://markmail.org/message/awimhb3nmu6ssvwp
pat="EACC"
pat=EACC
re1="-e '/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/'" re2="-e '/^.*cdkkt.com$/d'"
Note that this "$" should probably be backslashed. As it happens, $/ is not a shell special variable, so it is left alone. However, has you used $# or one of the others ($!, et al) there would have been substitution done, mangling your sed code.
rex="sed $re1 $re2"
I would strongly reommend putting this in a sed script file, perhaps like this:
cat >sedf$$ <<X /$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/ /^.*cdkkt.com$/d X rex="sed -f sedf$$"
Hmm, actually, maybe not. Maybe like this:
sedline="/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/; /^.*cdkkt.com$/d"
and then:
sed -e "$sedline"
in your pipeline lower down.
echo -en "\033[0;36m>> \033[0;35mExtracting data from:\n \ [\033[0;34m${TRACKER}\033[0;35m and appending to\n \ [\033[0;34m${TFILE}\033[0;35m]\033[0m"
I usually put escape sequences into their own variables for readbility, like this:
http://www.cskk.ezoshosting.com/cs/rc/shell/colour_ansi
Then you can talk about ${tty_green}, ${tty_normal} etc in your text; somewhat more readable. You can do even better than this, actually. Since not all terminals use the ANSI colour escapes, it is better to use tput, like this:
http://www.cskk.ezoshosting.com/cs/css/bin/with-colour
and then in your script:
esc_tracker=`with-colour green echo "$TRACKER"`
and then use ${esc_tracker} in messages. And so forth.
out=$(grep "$pat" "${TRACKER}" | \ eval "$rex" | sort -n | \ uniq >> "${TFILE}"); ret="$?";
if [ "$ret" -gt 0 ] || \ [ "$PIPESTATUS[0]" -gt 0 ] || \ [ "$PIPESTATUS[1]" -gt 0 ] || \ [ "$PIPESTATUS[2]" -gt 0 ] || \ [ "$PIPESTATUS[3]" -gt 0 ]; then echo -e " \033[0;31mFailed.\n \ \033[0;35m-- ErrorStatus:<ret=$ret>,\ PipeStatus:<$PIPESTATUS[@]>\n\ -- out:<$out>\033[0m" echo -e "\033[0;31m>> \033[0;31m$PROG exited.\033[0m\n" exit 1 fi echo -e " \033[0;32mDone.\033[0m";
I guess the ugly part is the multiple $PIPESTATUS[n] in the if statement test block. but perhaps this is the best I can do?
Nah. You can do this:
if out=$(grep "$pat" "$tracker" | $rex | sort -un >>"$tfile") then case " $PIPESTATUS" in *' '[1-9]*) echo exit ok, but PIPESTATUS=$PIPESTATUS ;; *)echo exit ok and PIPESTATUS all zeroes ;; esac else echo exit not ok fi
Um, you are aware that $out will always be empty? You have sent standard out to the temp file. Standard error is not collected by back ticks or $(); it will be displayed directly and not land in $out. You probably need to go:
out=$(exec 2>&1; grep .... >>"$tfile")
An easy test is like this:
out=$(ls /no/such/file >>temp) echo "out=[$out]"
versus:
out=$(exec 2>&1; ls /no/such/file >>temp) echo "out=[$out]"
Cheers,
On Friday, Mar 20th 2009 at 00:39 -0000, quoth Cameron Simpson:
=>Your quotes are unneeded here. Plenty of people find the "" marks and in =>similar cases, to use ${foo} instead of $foo elsewhere, but I find the =>syntactic noise annoying if there's no other necessity. => =>Also, you should never use UPPER CASE names for script local variables (i.e. =>that are not to be exported). I give some background why here:
Yes, yes yes a thousand times yes. I encourage people to always quote in an informed manner. This also can be expressed in a style that uses minimal quotes. 1. Don't quote if not necessary. 2. Use single quotes if quotes are needed but no variable interpolation is used. 3. Only use double quotes if you need variable interpolation.
Cameron Simpson wrote:
On 19Mar2009 18:55, Daniel B. Thurman dant@cdkkt.com wrote:
Cameron Simpson wrote:
Ok, but I strongly recommend you never us a regexp unquoted - the necessary backslash nesting gets nasty real fast. A better way is like this: re='s/b/h/' sed -e "$re"
[...]
The point here, unrelated to the pipe exit status issue, is to keep the sed stuff easy to write and undamaged by shell substitution. [...]
Another person sent me private email warning of the double-quotes, but thanks for your information, it is interesting!
This is what I ended up doing, so please give your constructive critiques, if you so like?
Untested, but here is how I would write each of you code pieces were I doing it myself:
TRACKER="Tracker.log" SFILE="Bad_Hosts_File.txt" TFILE="Bad_Hosts_File_$$.txt"
tracker=Tracker.log sfile=Bad_Hosts_File.txt tfile=Bad_Hosts_File_$$.txt
Your quotes are unneeded here. Plenty of people find the "" marks and in similar cases, to use ${foo} instead of $foo elsewhere, but I find the syntactic noise annoying if there's no other necessity.
Also, you should never use UPPER CASE names for script local variables (i.e. that are not to be exported). I give some background why here:
http://markmail.org/message/awimhb3nmu6ssvwp
pat="EACC"
pat=EACC
re1="-e '/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/'" re2="-e '/^.*cdkkt.com$/d'"
Note that this "$" should probably be backslashed. As it happens, $/ is not a shell special variable, so it is left alone. However, has you used $# or one of the others ($!, et al) there would have been substitution done, mangling your sed code.
rex="sed $re1 $re2"
I would strongly reommend putting this in a sed script file, perhaps like this:
cat >sedf$$ <<X /$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/ /^.*cdkkt.com$/d X rex="sed -f sedf$$"
Hmm, actually, maybe not. Maybe like this:
sedline="/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/; /^.*cdkkt.com$/d"
and then:
sed -e "$sedline"
in your pipeline lower down.
echo -en "\033[0;36m>> \033[0;35mExtracting data from:\n \ [\033[0;34m${TRACKER}\033[0;35m and appending to\n \ [\033[0;34m${TFILE}\033[0;35m]\033[0m"
I usually put escape sequences into their own variables for readbility, like this:
http://www.cskk.ezoshosting.com/cs/rc/shell/colour_ansi
Then you can talk about ${tty_green}, ${tty_normal} etc in your text; somewhat more readable. You can do even better than this, actually. Since not all terminals use the ANSI colour escapes, it is better to use tput, like this:
http://www.cskk.ezoshosting.com/cs/css/bin/with-colour
and then in your script:
esc_tracker=`with-colour green echo "$TRACKER"`
and then use ${esc_tracker} in messages. And so forth.
out=$(grep "$pat" "${TRACKER}" | \ eval "$rex" | sort -n | \ uniq >> "${TFILE}"); ret="$?";
if [ "$ret" -gt 0 ] || \ [ "$PIPESTATUS[0]" -gt 0 ] || \ [ "$PIPESTATUS[1]" -gt 0 ] || \ [ "$PIPESTATUS[2]" -gt 0 ] || \ [ "$PIPESTATUS[3]" -gt 0 ]; then echo -e " \033[0;31mFailed.\n \ \033[0;35m-- ErrorStatus:<ret=$ret>,\ PipeStatus:<$PIPESTATUS[@]>\n\ -- out:<$out>\033[0m" echo -e "\033[0;31m>> \033[0;31m$PROG exited.\033[0m\n" exit 1 fi echo -e " \033[0;32mDone.\033[0m";
I guess the ugly part is the multiple $PIPESTATUS[n] in the if statement test block. but perhaps this is the best I can do?
Nah. You can do this:
if out=$(grep "$pat" "$tracker" | $rex | sort -un >>"$tfile") then case " $PIPESTATUS" in *' '[1-9]*) echo exit ok, but PIPESTATUS=$PIPESTATUS ;; *)echo exit ok and PIPESTATUS all zeroes ;; esac else echo exit not ok fi
Um, you are aware that $out will always be empty? You have sent standard out to the temp file. Standard error is not collected by back ticks or $(); it will be displayed directly and not land in $out. You probably need to go:
out=$(exec 2>&1; grep .... >>"$tfile")
An easy test is like this:
out=$(ls /no/such/file >>temp) echo "out=[$out]"
versus:
out=$(exec 2>&1; ls /no/such/file >>temp) echo "out=[$out]"
Cheers,
Thanks a LOT! I have updated my code following your recommendations! Guys like you are sure a help!
It sure was an eye-opener for me when it comes to assigning variables with double-quotes. I thought it was needed, but little did I know!
Thanks for the education, it is appreciated! Dan
On Friday, Mar 20th 2009 at 00:39 -0000, quoth Cameron Simpson:
=>On 19Mar2009 18:55, Daniel B. Thurman dant@cdkkt.com wrote: =>> Cameron Simpson wrote: =>>> Ok, but I strongly recommend you never us a regexp unquoted - the =>>> necessary backslash nesting gets nasty real fast. A better way is like =>>> this: =>>> re='s/b/h/' =>>> sed -e "$re" =>[...] =>>> The point here, unrelated to the pipe exit status issue, is to keep the =>>> sed stuff easy to write and undamaged by shell substitution. [...] =>> =>> Another person sent me private email =>> warning of the double-quotes, but thanks =>> for your information, it is interesting! =>> =>> This is what I ended up doing, so please give =>> your constructive critiques, if you so like? => =>Untested, but here is how I would write each of you code pieces were I doing =>it myself: => =>> TRACKER="Tracker.log" =>> SFILE="Bad_Hosts_File.txt" =>> TFILE="Bad_Hosts_File_$$.txt" => =>tracker=Tracker.log =>sfile=Bad_Hosts_File.txt =>tfile=Bad_Hosts_File_$$.txt => =>Your quotes are unneeded here. Plenty of people find the "" marks and in =>similar cases, to use ${foo} instead of $foo elsewhere, but I find the =>syntactic noise annoying if there's no other necessity. => =>Also, you should never use UPPER CASE names for script local variables (i.e. =>that are not to be exported). I give some background why here: => => http://markmail.org/message/awimhb3nmu6ssvwp
One historical tidbit I'd like to share with you. Back when the TRex was tromping around, I worked at Data General during Soul Of A New Machine. The deal there was that all external variables under AOS had to be uppercase. The reason for this was because we sometimes needed to access variables by name at link-time and the CLI (Command Line Interface, a sort of macro processor) was case insensitive, so all commands caused everything to be flipped to uppercase, no matter what we wanted. The result was that all external variables ended up as uppercase, just in case. :-)
=> =>> pat="EACC" => =>pat=EACC => =>> re1="-e '/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/'" =>> re2="-e '/^.*cdkkt.com$/d'" => =>Note that this "$" should probably be backslashed. As it happens, $/ is =>not a shell special variable, so it is left alone. However, has you used =>$# or one of the others ($!, et al) there would have been substitution =>done, mangling your sed code. => =>> rex="sed $re1 $re2" => =>I would strongly reommend putting this in a sed script file, perhaps =>like this: => => cat >sedf$$ <<X => /$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/ => /^.*cdkkt.com$/d => X => rex="sed -f sedf$$" => =>Hmm, actually, maybe not. Maybe like this: => => sedline="/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/; /^.*cdkkt.com$/d" => =>and then: => => sed -e "$sedline" => =>in your pipeline lower down. => =>> echo -en "\033[0;36m>> \033[0;35mExtracting data from:\n \ =>> [\033[0;34m${TRACKER}\033[0;35m and appending to\n \ =>> [\033[0;34m${TFILE}\033[0;35m]\033[0m" => =>I usually put escape sequences into their own variables for readbility, =>like this: => => http://www.cskk.ezoshosting.com/cs/rc/shell/colour_ansi => =>Then you can talk about ${tty_green}, ${tty_normal} etc in your text; =>somewhat more readable. You can do even better than this, actually. =>Since not all terminals use the ANSI colour escapes, it is better to use =>tput, like this: => => http://www.cskk.ezoshosting.com/cs/css/bin/with-colour => =>and then in your script: => => esc_tracker=`with-colour green echo "$TRACKER"` => =>and then use ${esc_tracker} in messages. And so forth. => =>> out=$(grep "$pat" "${TRACKER}" | \ =>> eval "$rex" | sort -n | \ =>> uniq >> "${TFILE}"); ret="$?"; =>> =>> if [ "$ret" -gt 0 ] || \ =>> [ "$PIPESTATUS[0]" -gt 0 ] || \ =>> [ "$PIPESTATUS[1]" -gt 0 ] || \ =>> [ "$PIPESTATUS[2]" -gt 0 ] || \ =>> [ "$PIPESTATUS[3]" -gt 0 ]; then =>> echo -e " \033[0;31mFailed.\n \ =>> \033[0;35m-- ErrorStatus:<ret=$ret>,\ =>> PipeStatus:<$PIPESTATUS[@]>\n\ =>> -- out:<$out>\033[0m" =>> echo -e "\033[0;31m>> \033[0;31m$PROG exited.\033[0m\n" =>> exit 1 =>> fi =>> echo -e " \033[0;32mDone.\033[0m"; =>> =>> I guess the ugly part is the multiple $PIPESTATUS[n] =>> in the if statement test block. but perhaps this is the best =>> I can do? => =>Nah. You can do this: => => if out=$(grep "$pat" "$tracker" | $rex | sort -un >>"$tfile") => then => case " $PIPESTATUS" in => *' '[1-9]*) => echo exit ok, but PIPESTATUS=$PIPESTATUS => ;; => *)echo exit ok and PIPESTATUS all zeroes => ;; => esac => else => echo exit not ok => fi => =>Um, you are aware that $out will always be empty? You have sent =>standard out to the temp file. Standard error is not collected by back =>ticks or $(); it will be displayed directly and not land in $out. =>You probably need to go: => => out=$(exec 2>&1; grep .... >>"$tfile") => =>An easy test is like this: => => out=$(ls /no/such/file >>temp) => echo "out=[$out]" => =>versus: => => out=$(exec 2>&1; ls /no/such/file >>temp) => echo "out=[$out]" => =>Cheers, =>-- =>Cameron Simpson cs@zip.com.au DoD#743 =>http://www.cskk.ezoshosting.com/cs/ => =>The aim of AI is to make computers act like the ones in the movies. => - Graham Mann => =>-- =>fedora-list mailing list =>fedora-list@redhat.com =>To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list =>Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines =>
Steven W. Orr wrote:
On Friday, Mar 20th 2009 at 00:39 -0000, quoth Cameron Simpson:
=>On 19Mar2009 18:55, Daniel B. Thurman dant@cdkkt.com wrote: =>> Cameron Simpson wrote: =>>> Ok, but I strongly recommend you never us a regexp unquoted - the =>>> necessary backslash nesting gets nasty real fast. A better way is like =>>> this: =>>> re='s/b/h/' =>>> sed -e "$re" =>[...] =>>> The point here, unrelated to the pipe exit status issue, is to keep the =>>> sed stuff easy to write and undamaged by shell substitution. [...] =>> =>> Another person sent me private email =>> warning of the double-quotes, but thanks =>> for your information, it is interesting! =>> =>> This is what I ended up doing, so please give =>> your constructive critiques, if you so like? => =>Untested, but here is how I would write each of you code pieces were I doing =>it myself: => =>> TRACKER="Tracker.log" =>> SFILE="Bad_Hosts_File.txt" =>> TFILE="Bad_Hosts_File_$$.txt" => =>tracker=Tracker.log =>sfile=Bad_Hosts_File.txt =>tfile=Bad_Hosts_File_$$.txt => =>Your quotes are unneeded here. Plenty of people find the "" marks and in =>similar cases, to use ${foo} instead of $foo elsewhere, but I find the =>syntactic noise annoying if there's no other necessity. => =>Also, you should never use UPPER CASE names for script local variables (i.e. =>that are not to be exported). I give some background why here: => => http://markmail.org/message/awimhb3nmu6ssvwp
One historical tidbit I'd like to share with you. Back when the TRex was tromping around, I worked at Data General during Soul Of A New Machine. The deal there was that all external variables under AOS had to be uppercase. The reason for this was because we sometimes needed to access variables by name at link-time and the CLI (Command Line Interface, a sort of macro processor) was case insensitive, so all commands caused everything to be flipped to uppercase, no matter what we wanted. The result was that all external variables ended up as uppercase, just in case. :-)
=> =>> pat="EACC" => =>pat=EACC => =>> re1="-e '/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/'" =>> re2="-e '/^.*cdkkt.com$/d'" => =>Note that this "$" should probably be backslashed. As it happens, $/ is =>not a shell special variable, so it is left alone. However, has you used =>$# or one of the others ($!, et al) there would have been substitution =>done, mangling your sed code. => =>> rex="sed $re1 $re2" => =>I would strongly reommend putting this in a sed script file, perhaps =>like this: => => cat >sedf$$ <<X => /$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/ => /^.*cdkkt.com$/d => X => rex="sed -f sedf$$" => =>Hmm, actually, maybe not. Maybe like this: => => sedline="/$pat/s/^.*Received:[from\s+(.*)\s+(.*/\1/; /^.*cdkkt.com$/d" => =>and then: => => sed -e "$sedline" => =>in your pipeline lower down. => =>> echo -en "\033[0;36m>> \033[0;35mExtracting data from:\n \ =>> [\033[0;34m${TRACKER}\033[0;35m and appending to\n \ =>> [\033[0;34m${TFILE}\033[0;35m]\033[0m" => =>I usually put escape sequences into their own variables for readbility, =>like this: => => http://www.cskk.ezoshosting.com/cs/rc/shell/colour_ansi => =>Then you can talk about ${tty_green}, ${tty_normal} etc in your text; =>somewhat more readable. You can do even better than this, actually. =>Since not all terminals use the ANSI colour escapes, it is better to use =>tput, like this: => => http://www.cskk.ezoshosting.com/cs/css/bin/with-colour => =>and then in your script: => => esc_tracker=`with-colour green echo "$TRACKER"` => =>and then use ${esc_tracker} in messages. And so forth. => =>> out=$(grep "$pat" "${TRACKER}" | \ =>> eval "$rex" | sort -n | \ =>> uniq >> "${TFILE}"); ret="$?"; =>> =>> if [ "$ret" -gt 0 ] || \ =>> [ "$PIPESTATUS[0]" -gt 0 ] || \ =>> [ "$PIPESTATUS[1]" -gt 0 ] || \ =>> [ "$PIPESTATUS[2]" -gt 0 ] || \ =>> [ "$PIPESTATUS[3]" -gt 0 ]; then =>> echo -e " \033[0;31mFailed.\n \ =>> \033[0;35m-- ErrorStatus:<ret=$ret>,\ =>> PipeStatus:<$PIPESTATUS[@]>\n\ =>> -- out:<$out>\033[0m" =>> echo -e "\033[0;31m>> \033[0;31m$PROG exited.\033[0m\n" =>> exit 1 =>> fi =>> echo -e " \033[0;32mDone.\033[0m"; =>> =>> I guess the ugly part is the multiple $PIPESTATUS[n] =>> in the if statement test block. but perhaps this is the best =>> I can do? => =>Nah. You can do this: => => if out=$(grep "$pat" "$tracker" | $rex | sort -un >>"$tfile")
Please note:
When I tried `sort -un', the data was truncated, i.e. there is data loss. So, when I went back to my original code using 'sort -n | uniq', there is no data loss. There seems to be a problem using the `sort -un' method.
What I do in my code, is to create a copy of the sorted and uniq'd original file to a temp file, and then append new data to the temp file, then sorted and uniq the temp file back into the original file. The result was a file that ended up much smaller than the original file!
=> then => case " $PIPESTATUS" in => *' '[1-9]*) => echo exit ok, but PIPESTATUS=$PIPESTATUS => ;; => *)echo exit ok and PIPESTATUS all zeroes => ;; => esac => else => echo exit not ok => fi => =>Um, you are aware that $out will always be empty? You have sent =>standard out to the temp file. Standard error is not collected by back =>ticks or $(); it will be displayed directly and not land in $out. =>You probably need to go: => => out=$(exec 2>&1; grep .... >>"$tfile") => =>An easy test is like this: => => out=$(ls /no/such/file >>temp) => echo "out=[$out]" => =>versus: => => out=$(exec 2>&1; ls /no/such/file >>temp) => echo "out=[$out]" => =>Cheers, =>-- =>Cameron Simpson cs@zip.com.au DoD#743 =>http://www.cskk.ezoshosting.com/cs/ => =>The aim of AI is to make computers act like the ones in the movies. => - Graham Mann => =>-- =>fedora-list mailing list =>fedora-list@redhat.com =>To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list =>Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines =>
On 21Mar2009 16:47, Daniel B. Thurman dant@cdkkt.com wrote:
=>> out=$(grep "$pat" "${TRACKER}" | \ =>> eval "$rex" | sort -n | \ =>> uniq >> "${TFILE}"); ret="$?";
[...]
=> if out=$(grep "$pat" "$tracker" | $rex | sort -un >>"$tfile")
Please note:
When I tried `sort -un', the data was truncated, i.e. there is data loss. So, when I went back to my original code using 'sort -n | uniq', there is no data loss. There seems to be a problem using the `sort -un' method.
Well, they do mean slightly different things.
"sort -un" sorts and returns the first row of each set of rows that sorted equal. (i.e. "1 foo" and "1 bah" sort equal (numeric) and only "1 foo" is returned. (See "man sort" for the details, and "man 1p sort" for what you may portably expect on multiple UNIX platforms.)
"uniq" discards repeated identical lines. "1 foo" and "1 foo" are identical, but not "1 bah". (And uniq requires sorted input; the repeated lines must be adjacent in the input.)
It is often correct to replace "sort -n | uniq" with "sort -un", but I was clearly wrong to do so here.
What I do in my code, is to create a copy of the sorted and uniq'd original file to a temp file, and then append new data to the temp file, then sorted and uniq the temp file back into the original file. The result was a file that ended up much smaller than the original file!
Yah, see discussion above.
Cheers,
On Saturday, Mar 21st 2009 at 22:41 -0000, quoth Cameron Simpson:
=>On 21Mar2009 16:47, Daniel B. Thurman dant@cdkkt.com wrote: =>>> =>> out=$(grep "$pat" "${TRACKER}" | \ =>>> =>> eval "$rex" | sort -n | \ =>>> =>> uniq >> "${TFILE}"); ret="$?"; =>[...] =>>> => if out=$(grep "$pat" "$tracker" | $rex | sort -un >>"$tfile") =>>> =>> Please note: =>> =>> When I tried `sort -un', the data was truncated, i.e. =>> there is data loss. So, when I went back to my original =>> code using 'sort -n | uniq', there is no data loss. There =>> seems to be a problem using the `sort -un' method. => =>Well, they do mean slightly different things. => =>"sort -un" sorts and returns the first row of each set of rows that =>sorted equal. (i.e. "1 foo" and "1 bah" sort equal (numeric) and only "1 =>foo" is returned. (See "man sort" for the details, and "man 1p sort" for =>what you may portably expect on multiple UNIX platforms.) => =>"uniq" discards repeated identical lines. "1 foo" and "1 foo" are =>identical, but not "1 bah". (And uniq requires sorted input; the =>repeated lines must be adjacent in the input.) => =>It is often correct to replace "sort -n | uniq" with "sort -un", but I was =>clearly wrong to do so here. => =>> What I do in my code, is to create a copy of the sorted =>> and uniq'd original file to a temp file, and then append =>> new data to the temp file, then sorted and uniq the temp =>> file back into the original file. The result was a file that =>> ended up much smaller than the original file!
Fascinating. I never noticed that behaviour.
given
518 > cat > xxx 1 foo 1 bah 519 > sort -un xxx 1 foo
even though foo comes after bar. But look, it gets better.
522 > sort -k 1,1 -n xxx 1 bah 1 foo *523 > sort -k 1,1 -nu xxx 1 foo
So, even if I sort on the first column, the -u still selects the second record.
I guess I'm convinced that the only safe way is to *never* use -u.
On 22Mar2009 11:09, Steven W. Orr steveo@syslang.net wrote: | On Saturday, Mar 21st 2009 at 22:41 -0000, quoth Cameron Simpson: | =>On 21Mar2009 16:47, Daniel B. Thurman dant@cdkkt.com wrote: | =>> When I tried `sort -un', the data was truncated, i.e. | =>> there is data loss. So, when I went back to my original | =>> code using 'sort -n | uniq', there is no data loss. There | =>> seems to be a problem using the `sort -un' method. | => | =>Well, they do mean slightly different things. | => | =>"sort -un" sorts and returns the first row of each set of rows that | =>sorted equal. (i.e. "1 foo" and "1 bah" sort equal (numeric) and only "1 | =>foo" is returned. (See "man sort" for the details, and "man 1p sort" for | =>what you may portably expect on multiple UNIX platforms.) [...] | =>It is often correct to replace "sort -n | uniq" with "sort -un", but I was | =>clearly wrong to do so here. | | Fascinating. I never noticed that behaviour.
Neither had I until Daniel reported his problem. Then I read the manual page more closely.
[...] | I guess I'm convinced that the only safe way is to *never* use -u.
For plenty of data the -u option is just fine. It's only where you have extra data on the line not related to the sort criteria that it matters. Clearly, almost all the data I've sorted has been unique (columns of numbers, passwd and group files, etc), or I've not cared about which line I end up with (sorting lists of words).
So from my point of view, Daniel's use case is uncommon.
I intend using -u as frequently as before. Like anything, I must keep its real semantics in the back of my mind for when I do have data not unique on the sort keys, which must be kept.
Cheers,