More efficient use of GNU Parallel

Jim Meyering jim at meyering.net
Sun Jun 5 17:08:37 UTC 2011


[I've Cc'd iwhd-devel;  we try to keep all project-related mail public. ]

Ole Tange wrote:
> On https://www.aeolusproject.org/redmine/projects/iwhd/repository/revisions/master/entry/t/exercise
> it says:
>
> for i in $(seq $nb); do
>   # Create a bucket.
>   local b=http://localhost:$port/b-$i
>   curl_w -XPUT $b > http-code || fail=1
>   test "$(cat http-code)" = 201 || fail=1
> done
>
> Could that be replaced with:
>
> seq $nb | parallel curl --write-out '%{http_code}'  -XPUT
> http://localhost:$port/b-{} > http-code || fail=1
> grep -q 201 http-code || fail=1

Hi Ole,

Thank you for the suggestions!

I presume you mean this:

    seq $nb | parallel curl --write-out '%{http_code}' -XPUT \
      http://localhost:$port/b-{} > http-code || fail=1
    grep -q 201 http-code || fail=1

It looks like it could be.
They're slightly different in that the former stops upon
the first instance of a 201 failure, while the latter
reports all of them at the end, but that's fine.

There is the question of readability.
For those not intimately familiar with GNU parallel,
it's probably not obvious that the latter command runs
curl $nb times.  But that's ok too.  I can add a comment.
This makes me think that while the following code tests
creating many *objects* in parallel, we would benefit
by also creating many *buckets* in parallel.

So yes, this looks worthwhile, if only to parallelize
(probably using many more than $nb, just to exercise the code)
the bucket creating set-up.

> Later it says:
>
>   seq 20000 > data
>   for i in $(seq $nb); do
>     local b=http://localhost:$port/b-$i
>     for j in $(seq --format=%04g 50); do
>       # Create an object in that bucket.
>       local obj=$b/9756a40c-7668-11e0-$j-0015c5f4d7e4
>       echo "curl -S -T - $obj < data"
>     done
>   done |parallel --halt-on-error=2 -j $n_procs || fail=1
>
> With GNU Parallel 20110522 you can do the double loop in a single line:
>
> seq 20000 > data
> parallel --halt-on-error=2 -j 150% \
> curl -S -T - http://localhost:$port/b-{1}/9756a40c-7668-11e0-{2}-0015c5f4d7e4 \
> \< data :::: <(seq $nb) <(seq --format=%04g 50) || fail=1

Nice feature.  However, I'm reluctant to use it right now
because it'd require that newer version, which I'd then have to
test for.  FYI, <(...) works fine with modern shells like bash
and zsh, but is not specified by POSIX, so would fail with some
shells commonly used for /bin/sh; thus we'd need two more temporary
files in place of those uses:

    $ bash -c 'cat <(echo 1)'
    1
    $ dash -c 'cat <(echo 1)'
    dash: Syntax error: "(" unexpected
    [Exit 2]

Also, and perhaps more importantly, an explicit doubly-nested loop
seems more readable to me than using GNU parallel's new feature.


More information about the iwhd-devel mailing list