Using a subshell for parameter substitution with diff - bash

Question

I'm writing a shell script, and in an effort to make it shorter and easier to read, I'm trying to use nested subshells to pass parameters to diff.
Here's what I have:
if
diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' > /dev/null;
then
echo There is no difference between the files. > ./participants-by-state-results.txt;
else
diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' > ./participants-by-state-results.txt;
fi
When I run the script, I keep getting diff: extra operand 'AL'
I'd appreciate any insight into why this is failing. I think I'm pretty close. Thanks!

Answer 1

Your code is unreadable because the lines are so long:
if diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' \
'$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' \
> /dev/null;
then
echo There is no difference between the files. > ./participants-by-state-results.txt;
else
diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' \
'$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' \
> ./participants-by-state-results.txt;
fi
Repeating whole commands like that is also fairly nasty. You also have major problems with your use of single quotes; you only have one sort in each set of commands, apparently operating on the combined outputs of two identical awk commands (whereas you probably need two separate sorts, one for the output of each awk command); you're not using the -F option to awk when you could; you are repeating the gargantuan file names all over the place; and finally, it appears that you are probably wanting to use process substitution, but not actually doing so.
Let's take a step back and formulate the question clearly.
Given two files (new-participants-by-state.csv and current-participants-by-state.csv) find the first pipe-separated field on each line of each file, sort the lists of those fields, and compare the results of the two sorted lists.
If there are no differences, write a message into the output file participants-by-state-results.txt; otherwise, list the differences in the output file.
So, we could use:
oldfile='current-participants-by-state.csv'
newfile='new-participants-by-state.csv'
outfile='participants-by-state-results.txt'
tmpfile=${TMPDIR:-/tmp}/xx.$$
awk -F'|' '{print $1}' $oldfile | sort > $tmpfile.1
awk -F'|' '{print $1}' $newfile | sort > $tmpfile.2
if diff -iy $tmpfile.1 $tmpfile.2 > $outfile
then echo "There is no difference between the files" > $outfile
fi
rm -f $tmpfile.?
If this was going to be the final script, we'd want to put trap handling in place so that the temporary files are not left around unless the script is killed dead with SIGKILL.
However, we can now use process substitution to avoid the temporary files:
oldfile='current-participants-by-state.csv'
newfile='new-participants-by-state.csv'
outfile='participants-by-state-results.txt'
if diff -iy <(awk -F'|' '{print $1}' $oldfile | sort) \
<(awk -F'|' '{print $1}' $newfile | sort) > $outfile
then echo "There is no difference between the files" > $outfile
fi
Note how the code carefully preserves symmetries where there are symmetries. Note the use of shortish variable names to avoid the repetition of long file names. Note that the diff command is run just once, not twice - throwing away results which are needed later is not very sensible.
You could compress the output I/O redirection even more using:
{
if diff -iy <(awk -F'|' '{print $1}' $oldfile | sort) \
<(awk -F'|' '{print $1}' $newfile | sort)
then echo "There is no difference between the files"
fi
} > $outfile
That sends the standard output of the enclosed commands to the file.
Of course, CSV might not be the appropriate nomenclature if the files are pipe-separated rather than comma-separated, but that's another matter altogether.
I'm also assuming that the status from diff -iy works as suggested by the original script; I've not validated that usage of the diff command.

Answer 2

There are several problems here.
First, you're putting various arguments in single-quotes, which prevents any interpretation being done on them (for example, $(....) doesn't do anything special inside single-quotes). You're probably thinking of double-quotes, but those aren't what you want either.
Which brings us to the second problem, that diff and sort expect to be given filenames as arguments, and they operate on the data in those files; you're trying to pass the data directly as arguments, which doesn't work (and I suspect that's the origin of the error you're getting: diff expects exactly two filenames, you're passing more than two participant names, and AL happened to be third on the list and hence the one that diff panicked on). The usual way to do this is to use intermediate files (and multiple lines in the script), but bash actually has a way of doing this without either of those: process substitution. Essentially, what it does is run one command with output (or input, but we need output in this case) sent to a named pipe; then it passes the name of the pipe as an argument to another command. For example, diff <(command1) <(command2) will give you the differences between the outputs of command1 and command2. Note that since this is a bash-only feature, you must start the script with #!/bin/bash, not #!/bin/sh.
Third, there's a missing close-parenthesis that makes it a little hard to tell what's supposed to happen. Are both files supposed to be sorted before the comparison, or only the new-participants file?
Fourth, since the final comparison ignores case (-i), you'd better use a case-insensitive sort (-f) as well.
Finally, you're doing all of the processing twice if there are any differences. I'd recommend running the comparison once into a file, then if there were no differences just ignore/overwrite the (empty) file.
Oh, and just a stylistic thing: you don't need semicolons at the end of lines in bash. You only need semicolons if you're putting more than one command on the same line (and a few other cases like before then in an if statement).
Anyway, here's my rewrite:
#!/bin/bash
if
diff -iy <(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv | sort -f) <(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv | sort -f) >./participants-by-state-results.txt
then
echo "There is no difference between the files." > ./participants-by-state-results.txt
fi