How to use shell variables with GNU parallel? - shell

Question

This is the content of list.csv:
Apple,Red,10
Banana,Yellow,3
Coconut,White,18
Suppose I have this GNU parallel command:
parallel -a list.csv -j0 -C, \
color=`echo {2} | sed 's/e/eee/g' | ./capitalize.sh` ";" \
echo "{2}" ";" \
echo "$color" ";"
To get:
Red
REEED
Yellow
YEEELLOW
White
WHITEEE
Why isn't the color variable being defined/printed?
EDIT 20151218:
Now that I got the quoting right, I'd like to introduce a function reading a variable from another function, and reading $0.
This is a working example without GNU parallel (I made grep case-insensitive before posting, to facilitate testing without ./capitalize.sh).
while read line; do
doit() {
color=`echo $1 | cut -d, -f2 | sed 's/e/eee/g' | ./capitalize.sh`
}
export -f doit
get_key() {
key=`grep -i $color $0 | cut -d, -f2`
}
export -f get_key
#note that I would use parallel's `-C,` here instead of `cut`.
doit $line #get CSV's 2nd element and make it look like the one in script.
get_key #extract this element's value from the script's comments.
echo "color: $color"
echo "key: $key"
done < list.csv
#Key database in the shell script
# REEED,r-key
# YEEELLOW,y-key
# WHITEEE,w-key
Working output:
color: REEED
key: r-key
color: YEEELLOW
key: y-key
color: WHITEEE
key: w-key

Answer 1

This should work:
parallel -a list.csv -j0 -C, 'color=`echo {2} | sed "s/e/eee/g" | ./capitalize.sh`' ";" echo "{2}" ";" echo '"$color"' ";"
You are being hit by inadequate quoting. It might be easier to use a function:
doit() {
color=`echo $2 | sed 's/e/eee/g' | ./capitalize.sh`
echo "$2"
echo "$color"
}
export -f doit
parallel -a list.csv -j0 -C, doit
If this is the real goal you might want to use {= =} instead which is made for similar situations:
parallel -a list.csv -j0 -C, echo {2}";" echo '{=2 s/e/eee/g; $_=uc($_) =}'
If you are using $color several times, then --rpl can introduce a shorthand:
parallel --rpl '{clr} s/e/eee/g; $_=uc($_)' -a list.csv -j0 -C, echo {2}";" echo '{2clr} and again: {2clr}'
From the xargs afficionados I would really like to see a solution using xargs that:
guarantees not mixing output from different jobs - even if the lines are 60k long (e.g. the value of $color is 60k long)
sends stdout to stdout, and stderr to stderr
does not skip jobs even if the list of jobs (list.csv) is bigger than the number of available processes in the process table - even if capitalize.sh takes a full minute to run (xargs -P0)

Answer 2

The idea is to use a single function to do everything.
#!/bin/bash
#Key database in the shell script
# REEED,r-key
# YEEELLOW,y-key
# WHITEEE,w-key
doit() {
# get CSV's 2nd element and make it look like the one in script.
color=`echo $3 | cut -d, -f2 | sed 's/e/eee/g' | ./capitalize.sh`
#extract this element's value from the script's comments.
key=`grep -i $color $1 | cut -d, -f2`
echo "color: $color"
echo "key: $key"
}
export -f doit
#note that I would use parallel's `-C,` here instead of `cut`.
parallel -C, doit $0 < list.csv