I am trying to write a program that will print out every line from a file with another line of that file added at the end, basically creating pairs from a portion of each line. If the line is the same, it will do nothing. Also, it must avoid repeating the same pairs. A B is the same as B A
In short
FileInput:
otherstuff A
otherstuff B
otherstuff C
otherstuff D
Output:
A B
A C
A D
B C
B D
C D
I was trying to do this with a BASH script, but was having trouble because I could not get my nested while loops to work. It would read the first line, compare it to each other line, and then stop (Basically only outputting the first 3 lines in the example output above, the outer while loop only ran once).
I also suspect I might be able to do this using MATLAB, so suggestions using that are also welcome.
Here is the bash script that I have thus far. As I said, it is no printing out correctly for me, as the outer loop only runs once.
#READS IN file from terminal
FILE1=$1
#START count at 0
count0=
exec 3<&0
exec 0< $FILE1
while read LINEa; do
while read LINEb; do
eventIDa=$(echo $LINEa | cut -c20-23)
eventIDb=$(echo $LINEb | cut -c20-23)
echo $eventIDa $eventIDb
done
done
Using bash:
#!/bin/bash
[ -f "$1" ] || { echo >&2 "File not found"; exit 1; }
mapfile -t lines < <(cut -c20-23 <"$1" | sort | uniq)
for i in ${!lines[#]}; do
elem1=${lines[$i]}
unset lines[$i]
for elem2 in "${lines[#]}"; do
echo "$elem1" "$elem2"
done
done
This will read a file given as a parameter on the command line, sort and filter out duplicates, and output all combinations. You can modify the parameter to cut to adjust to your particular input file.
Due to the particular way you seem to indent to use cut, your input example above won't work. Instead, use something with the correct line length, such as:
123456789012345678 A
123456789012345678 B
123456789012345678 C
123456789012345678 D
Assuming the otherstuff is not relevant (otherwise you can of course add it later) this should do the trick in Matlab:
combnk({'A' 'B' 'C' 'D'},2)
Related
Given a file so that in any line can be more than one word, and exists a single space between any word to other, for example:
a a a a
b b b b
c c
d d
a a a a
How can I create array so that in the cell number i will be the line number i , but WITHOUT DUPLICATES BETWEEN THE ELEMENTS IN THE ARRAY !
In according to the file above, we will need create this array:
Array[0]="a a a a" , Array[1]="b b b b" , Array[2]="c c" , Array[3]=d d.
(The name of the file pass to the script as argument).
I know how to create array that will contain all the lines. Something like that:
Array=()
while read line; do
Array=("${Array[#]}" "${line}")
done < $1
But how can I pass to the while read.. the sorting (and uniq) output of the file?
You should be able to use done < <(sort "$1" | uniq) in place of done < $1.
The <() syntax creates a file-like object from a subshell to execute a separate set of commands.
I have one queston about nested loop with bash.
I have an input files with one file name per line (full path)
I read this file and then i make a nest loop:
for i in $filelines ; do
echo $i
for j in $filelines ; do
./program $i $j
done
done
The program I within the loop is pretty low.
Basically it compare the file A with the file B.
I want to skip A vs A comparison (i.e comparing one file with itslef) AND
I want to avoid permutation (i.e. for file A and B, only perform A against B and not B against A).
What is the simplest to perform this?
Version 2: this one takes care of permutations
#!/bin/bash
tmpunsorted="/tmp/compare_unsorted"
tmpsorted="/tmp/compare_sorted"
>$tmpunsorted
while read linei
do
while read linej
do
if [ $linei != $linej ]
then
echo $linei $linej | tr " " "\n" | sort | tr "\n" " " >>$tmpunsorted
echo >>$tmpunsorted
fi
done <filelines
done <filelines
sort $tmpunsorted | uniq > $tmpsorted
while read linecompare
do
echo "./program $linecompare"
done <$tmpsorted
# Cleanup
rm -f $tmpunsorted
rm -f $tmpsorted
What is done here:
I use the while loop to read each line, twice, i and j
if the value of the lines is the same, forget them, no use to consider them
if they are different, output them into a file ($tmpunsorted). And they are sorted in alphebetical order before going tothe $tmpunsorted file. This way the arguments are always in the same order. So a b and b a will be same in the unsorted file.
I then apply sort | uniq on $tmpunsorted, so the result is a list of individual argument pairs.
finally loop on the $tmpsorted file, and call the program on each individual pair.
Since I do not have your program, I did an echo, which you should remove to use the script.
I want to run a while loop from output I get from MySQL, but my output is being cut off.
Example output I get from MySQL is:
123 nfs://192.168.1.100/full/Some.file.1.txt
124 nfs://192.168.1.100/full/A second file 2.txt
My loop looks like so:
mysql -uuser -ppass queue -ss -e 'select id,path from queue where status = 0' | while read a b c
do
echo $a
echo $b
done
The result for $b cuts off after nfs://192.168.1.100/full/A.
How can I have it output the whole sentence?
Your second filename contains spaces, so that is where the field is cut off.
Since it is the last field of the output, you can just skip field c:
mysql -uuser -ppass queue -ss -e 'select id,path from queue where status = 0' | while read a b
do
echo $a
echo $b
done
The last field in read will have all remaining fields.
Problem is that you are reading each line into 3 variables using:
read a b c
And since your input line also contains a whitespace e.g.
124 nfs://192.168.1.100/full/A second file 2.txt
with the default IFS it is setting 3 variables as:
a=124
b=nfs://192.168.1.100/full/A
c=second file 2.txt
Since c is the last parameter in read it is reading rest of the line in c.
To fix your script you can just do:
read a b
I have two programs that produce data on stdout, and I'd like to paste their output together. I can successfully do this like so:
paste <(./prog1) <(./prog2)
But I find that this method will print all lines from both inputs,
and what I really want is to stop paste after either input program is finished.
So if ./prog1 produces the output:
a
b
c
But ./prog2 produces:
Hello
World
I would expect the output:
a Hello
b World
Also note that one of the input programs may actually produce infinite output, and I want to be able to handle that case as well. For example, if my inputs are yes and ./prog2, I should get:
y Hello
y World
Use join instead, with a variation on the Schwartzian transform:
numbered () {
nl -s- -ba -nrz
}
join -j 1 <(prog1 | numbered) <(prog2 | numbered) | sed 's/^[^-]*-//'
Piping to nl numbers each line, and join -1 1 will join corresponding lines with the same number. The extra lines in the longer file will have no join partner and be omitted. Once the join is complete, pipe through sed to remove the line numbers.
Here's one solution:
while IFS= read -r -u7 a && IFS= read -r -u8 b; do echo "$a $b"; done 7<$file1 8<$file2
This has the slightly annoying effect of ignoring the last line of an input file if it is not terminated with a newline (but such a file is not a valid text file).
You can wrap this in a function, of course:
paste_short() {
(
while IFS= read -r -u7 a && IFS= read -r -u8 b; do
echo "$a $b"
done
) 7<"$1" 8<"$2"
}
Consider using awk:
awk 'FNR==NR{a[++i]=$0;next} FNR>i{exit}
{print a[FNR], $0}' <(printf "hello\nworld\n") <(printf "a\nb\nc\n")
hello a
world b
Keep the longer output producing program as your 2nd input.
I understand the format below...
while read line
do
etc...
However, I saw this yesterday and haven't been able to figure out what var would be in the following:
while read pkg var
do
etc...
Thanks
while loop will read the var one by one , but assign the last parts to one var.
For example, I have a file like:
a b c d
when run the command
$ while read x y
do
echo $x
echo $y
done < file
Resule:
a
b c d
You will get "b c d" to $y.
Of course, if you only assign one var (line), then $line will get the whole line.
The read builtin will read multiple whitespace-separated (or, really, separated by whatever is in $IFS) values.
echo a b c | (read x y z; echo "$y")
#=> b
If there are more fields than variables passed to read, the last variable gets the rest of the line.