Is there a way to compute permutations in bash? - bash

Question

I have multiple files (nearly 1000 of them) separated by space and I need to compute permutations with join function between each one of them using the second columns only. The important thing is that the comparison must not be repeated, that's why the permutation.
For instance, a small example with 3 files A.txt B.txt and C.txt
The general idea is to get A B comparison, A C and B C. Neither B A nor C A nor C B
The 101 code would be
join -1 2 -2 2 A.txt B.txt | cut -d ' ' -f1 > AB.txt
join -1 2 -2 2 A.txt C.txt | cut -d ' ' -f1 > AC.txt
join -1 2 -2 2 B.txt C.txt | cut -d ' ' -f1 > BC.txt
Is there a way to accomplish this for thousand of files? I tried using a for loop, but toasted my brains out, and now I'm trying with a while loop. But I better get some orientation first.

Answer 1

As the number of iterations is quite large performance becomes an issue. Here is an optimized version of Matty's answer, using an array, to divide the number of iterations by 2 (half a million instead of a million) and to avoid a test:
declare -a files=( *.txt )
declare -i len=${#files[#]}
declare -i lenm1=$(( len - 1 ))
for (( i = 0; i < lenm1; i++ )); do
a="${files[i]}"
ab="${a%.txt}"
for (( j = i + 1; j < len; j++ )); do
b="${files[j]}"
join -1 2 -2 2 "$a" "$b" | cut -d ' ' -f1 > "$ab$b"
done
done
But consider that bash was not designed for such intensive tasks with half a million iterations. There might be a better (more efficient) way to accomplish what you want.

Answer 2

It looks like what you are after can be accomplished with two nested for loops and a lexicographic comparison to maintain alphabetical order?
# prints pairs of filenames
for f in dir/*; do
for g in dir/*; do
if [[ "$f" < "$g" ]]; then # ensure alphabetical order
echo $f $g
fi
done
done

Answer 3

Here's why you don't want to use bash for this:
First create 1000 files
seq 1000 | xargs touch
Now, distinct pairs with bash
time {
files=(*)
len=${#files[#]}
for ((i=0; i<len-1; i++)); do
a=${files[i]}
for ((j=i+1; j<len; j++)); do
b=${files[j]}
echo "$a $b"
done
done >/dev/null
}
real 0m5.091s
user 0m4.818s
sys 0m0.262s
Versus, for example, the same in perl:
time {
perl -e '
opendir my $dh, ".";
my #files = sort grep {$_ != "." && $_ != ".."} readdir $dh;
closedir $dh;
for (my $i = 0; $i < #files - 1; $i++) {
my $a = $files[$i];
for (my $j = $i + 1; $j < #files; $j++) {
my $b = $files[$j];
print "$a $b\n";
}
}
' > /dev/null
}
real 0m0.131s
user 0m0.120s
sys 0m0.006s