I am currently writing a bash script to find names that are available in File1 but not available in File2.
File1:
"Name"
"Jeff"
"Michael"
"Ringo"
"John"
File2:
"Name"
"Jeff"
"Michael"
"John"
"Bert"
From the example above, it should return "Ringo". So far, I am running a for loop to extract it.
for q in `cat File1 | tail -n +2 | sort`;do grep $q File2 >> output.txt;done
However, it would take forever to run it on ~150,000 records. So, is there a better solution you can share for this?
Thanks in advance for the answers.
comm is a standard utility for this.
tail -n +2 File1 | sort -u > tmp1
sort -u File2 > tmp2
comm -23 tmp1 tmp2 > output.txt
rm tmp1 tmp2
With bash, the temporary file cleanup can be avoided:
comm -23 \
<(tail -n +2 File1 | sort -u) \
<(sort -u File2) \
> output.txt
Note that sort works fine on files that do not fit in memory (implementations generally use mergesort with temporary files if memory usage would become too high). comm itself requires minimal memory. I believe overall runtime is O(n*log(n))
I think you're looking for diff(1). If you have the GNU version, this flag with some output processing to get just the first column.
--suppress-common-lines
do not output common lines
--side-by-side, -y
output in two columns
But diff requires lines to be in the same order in both files. If that's not your case, grep with multiple expressions and the invert flag -v/--invert-match and -E/--extended-regexp might work better.
Also note that I am using command substitution instead of a for loop to run it in one go. The (x|y) extended regexp searches x OR y.
grep --invert-match --extended-regexp \
"^( $(uniq file2 | tr '\n' '|') )$" \
file1
Related
I have two large files (sets of filenames). Roughly 30.000 lines in each file. I am trying to find a fast way of finding lines in file1 that are not present in file2.
For example, if this is file1:
line1
line2
line3
And this is file2:
line1
line4
line5
Then my result/output should be:
line2
line3
This works:
grep -v -f file2 file1
But it is very, very slow when used on my large files.
I suspect there is a good way to do this using diff(), but the output should be just the lines, nothing else, and I cannot seem to find a switch for that.
Can anyone help me find a fast way of doing this, using bash and basic Linux binaries?
EDIT: To follow up on my own question, this is the best way I have found so far using diff():
diff file2 file1 | grep '^>' | sed 's/^>\ //'
Surely, there must be a better way?
The comm command (short for "common") may be useful comm - compare two sorted files line by line
#find lines only in file1
comm -23 file1 file2
#find lines only in file2
comm -13 file1 file2
#find lines common to both files
comm -12 file1 file2
The man file is actually quite readable for this.
You can achieve this by controlling the formatting of the old/new/unchanged lines in GNU diff output:
diff --new-line-format="" --unchanged-line-format="" file1 file2
The input files should be sorted for this to work. With bash (and zsh) you can sort in-place with process substitution <( ):
diff --new-line-format="" --unchanged-line-format="" <(sort file1) <(sort file2)
In the above new and unchanged lines are suppressed, so only changed (i.e. removed lines in your case) are output. You may also use a few diff options that other solutions don't offer, such as -i to ignore case, or various whitespace options (-E, -b, -v etc) for less strict matching.
Explanation
The options --new-line-format, --old-line-format and --unchanged-line-format let you control the way diff formats the differences, similar to printf format specifiers. These options format new (added), old (removed) and unchanged lines respectively. Setting one to empty "" prevents output of that kind of line.
If you are familiar with unified diff format, you can partly recreate it with:
diff --old-line-format="-%L" --unchanged-line-format=" %L" \
--new-line-format="+%L" file1 file2
The %L specifier is the line in question, and we prefix each with "+" "-" or " ", like diff -u
(note that it only outputs differences, it lacks the --- +++ and ## lines at the top of each grouped change).
You can also use this to do other useful things like number each line with %dn.
The diff method (along with other suggestions comm and join) only produce the expected output with sorted input, though you can use <(sort ...) to sort in place. Here's a simple awk (nawk) script (inspired by the scripts linked-to in Konsolebox's answer) which accepts arbitrarily ordered input files, and outputs the missing lines in the order they occur in file1.
# output lines in file1 that are not in file2
BEGIN { FS="" } # preserve whitespace
(NR==FNR) { ll1[FNR]=$0; nl1=FNR; } # file1, index by lineno
(NR!=FNR) { ss2[$0]++; } # file2, index by string
END {
for (ll=1; ll<=nl1; ll++) if (!(ll1[ll] in ss2)) print ll1[ll]
}
This stores the entire contents of file1 line by line in a line-number indexed array ll1[], and the entire contents of file2 line by line in a line-content indexed associative array ss2[]. After both files are read, iterate over ll1 and use the in operator to determine if the line in file1 is present in file2. (This will have have different output to the diff method if there are duplicates.)
In the event that the files are sufficiently large that storing them both causes a memory problem, you can trade CPU for memory by storing only file1 and deleting matches along the way as file2 is read.
BEGIN { FS="" }
(NR==FNR) { # file1, index by lineno and string
ll1[FNR]=$0; ss1[$0]=FNR; nl1=FNR;
}
(NR!=FNR) { # file2
if ($0 in ss1) { delete ll1[ss1[$0]]; delete ss1[$0]; }
}
END {
for (ll=1; ll<=nl1; ll++) if (ll in ll1) print ll1[ll]
}
The above stores the entire contents of file1 in two arrays, one indexed by line number ll1[], one indexed by line content ss1[]. Then as file2 is read, each matching line is deleted from ll1[] and ss1[]. At the end the remaining lines from file1 are output, preserving the original order.
In this case, with the problem as stated, you can also divide and conquer using GNU split (filtering is a GNU extension), repeated runs with chunks of file1 and reading file2 completely each time:
split -l 20000 --filter='gawk -f linesnotin.awk - file2' < file1
Note the use and placement of - meaning stdin on the gawk command line. This is provided by split from file1 in chunks of 20000 line per-invocation.
For users on non-GNU systems, there is almost certainly a GNU coreutils package you can obtain, including on OSX as part of the Apple Xcode tools which provides GNU diff, awk, though only a POSIX/BSD split rather than a GNU version.
Like konsolebox suggested, the posters grep solution
grep -v -f file2 file1
actually works great (faster) if you simply add the -F option, to treat the patterns as fixed strings instead of regular expressions. I verified this on a pair of ~1000 line file lists I had to compare. With -F it took 0.031 s (real), while without it took 2.278 s (real), when redirecting grep output to wc -l.
These tests also included the -x switch, which are necessary part of the solution in order to ensure totally accuracy in cases where file2 contains lines which match part of, but not all of, one or more lines in file1.
So a solution that does not require the inputs to be sorted, is fast, flexible (case sensitivity, etc) is:
grep -F -x -v -f file2 file1
This doesn't work with all versions of grep, for example it fails in macOS, where a line in file 1 will be shown as not present in file 2, even though it is, if it matches another line that is a substring of it. Alternatively you can install GNU grep on macOS in order to use this solution.
If you're short of "fancy tools", e.g. in some minimal Linux distribution, there is a solution with just cat, sort and uniq:
cat includes.txt excludes.txt excludes.txt | sort | uniq --unique
Test:
seq 1 1 7 | sort --random-sort > includes.txt
seq 3 1 9 | sort --random-sort > excludes.txt
cat includes.txt excludes.txt excludes.txt | sort | uniq --unique
# Output:
1
2
This is also relatively fast, compared to grep.
whats the speed of as sort and diff?
sort file1 -u > file1.sorted
sort file2 -u > file2.sorted
diff file1.sorted file2.sorted
Use combine from moreutils package, a sets utility that supports not, and, or, xor operations
combine file1 not file2
i.e give me lines that are in file1 but not in file2
OR give me lines in file1 minus lines in file2
Note: combine sorts and finds unique lines in both files before performing any operation but diff does not. So you might find differences between output of diff and combine.
So in effect you are saying
Find distinct lines in file1 and file2 and then give me lines in file1 minus lines in file2
In my experience, it's much faster than other options
This seems quick for me :
comm -1 -3 <(sort file1.txt) <(sort file2.txt) > output.txt
$ join -v 1 -t '' file1 file2
line2
line3
The -t makes sure that it compares the whole line, if you had a space in some of the lines.
You can use Python:
python -c '
lines_to_remove = set()
with open("file2", "r") as f:
for line in f.readlines():
lines_to_remove.add(line.strip())
with open("f1", "r") as f:
for line in f.readlines():
if line.strip() not in lines_to_remove:
print(line.strip())
'
Using of fgrep or adding -F option to grep could help. But for faster calculations you could use Awk.
You could try one of these Awk methods:
http://www.linuxquestions.org/questions/programming-9/grep-for-huge-files-826030/#post4066219
The way I usually do this is using the --suppress-common-lines flag, though note that this only works if your do it in side-by-side format.
diff -y --suppress-common-lines file1.txt file2.txt
I have two large files (sets of filenames). Roughly 30.000 lines in each file. I am trying to find a fast way of finding lines in file1 that are not present in file2.
For example, if this is file1:
line1
line2
line3
And this is file2:
line1
line4
line5
Then my result/output should be:
line2
line3
This works:
grep -v -f file2 file1
But it is very, very slow when used on my large files.
I suspect there is a good way to do this using diff(), but the output should be just the lines, nothing else, and I cannot seem to find a switch for that.
Can anyone help me find a fast way of doing this, using bash and basic Linux binaries?
EDIT: To follow up on my own question, this is the best way I have found so far using diff():
diff file2 file1 | grep '^>' | sed 's/^>\ //'
Surely, there must be a better way?
The comm command (short for "common") may be useful comm - compare two sorted files line by line
#find lines only in file1
comm -23 file1 file2
#find lines only in file2
comm -13 file1 file2
#find lines common to both files
comm -12 file1 file2
The man file is actually quite readable for this.
You can achieve this by controlling the formatting of the old/new/unchanged lines in GNU diff output:
diff --new-line-format="" --unchanged-line-format="" file1 file2
The input files should be sorted for this to work. With bash (and zsh) you can sort in-place with process substitution <( ):
diff --new-line-format="" --unchanged-line-format="" <(sort file1) <(sort file2)
In the above new and unchanged lines are suppressed, so only changed (i.e. removed lines in your case) are output. You may also use a few diff options that other solutions don't offer, such as -i to ignore case, or various whitespace options (-E, -b, -v etc) for less strict matching.
Explanation
The options --new-line-format, --old-line-format and --unchanged-line-format let you control the way diff formats the differences, similar to printf format specifiers. These options format new (added), old (removed) and unchanged lines respectively. Setting one to empty "" prevents output of that kind of line.
If you are familiar with unified diff format, you can partly recreate it with:
diff --old-line-format="-%L" --unchanged-line-format=" %L" \
--new-line-format="+%L" file1 file2
The %L specifier is the line in question, and we prefix each with "+" "-" or " ", like diff -u
(note that it only outputs differences, it lacks the --- +++ and ## lines at the top of each grouped change).
You can also use this to do other useful things like number each line with %dn.
The diff method (along with other suggestions comm and join) only produce the expected output with sorted input, though you can use <(sort ...) to sort in place. Here's a simple awk (nawk) script (inspired by the scripts linked-to in Konsolebox's answer) which accepts arbitrarily ordered input files, and outputs the missing lines in the order they occur in file1.
# output lines in file1 that are not in file2
BEGIN { FS="" } # preserve whitespace
(NR==FNR) { ll1[FNR]=$0; nl1=FNR; } # file1, index by lineno
(NR!=FNR) { ss2[$0]++; } # file2, index by string
END {
for (ll=1; ll<=nl1; ll++) if (!(ll1[ll] in ss2)) print ll1[ll]
}
This stores the entire contents of file1 line by line in a line-number indexed array ll1[], and the entire contents of file2 line by line in a line-content indexed associative array ss2[]. After both files are read, iterate over ll1 and use the in operator to determine if the line in file1 is present in file2. (This will have have different output to the diff method if there are duplicates.)
In the event that the files are sufficiently large that storing them both causes a memory problem, you can trade CPU for memory by storing only file1 and deleting matches along the way as file2 is read.
BEGIN { FS="" }
(NR==FNR) { # file1, index by lineno and string
ll1[FNR]=$0; ss1[$0]=FNR; nl1=FNR;
}
(NR!=FNR) { # file2
if ($0 in ss1) { delete ll1[ss1[$0]]; delete ss1[$0]; }
}
END {
for (ll=1; ll<=nl1; ll++) if (ll in ll1) print ll1[ll]
}
The above stores the entire contents of file1 in two arrays, one indexed by line number ll1[], one indexed by line content ss1[]. Then as file2 is read, each matching line is deleted from ll1[] and ss1[]. At the end the remaining lines from file1 are output, preserving the original order.
In this case, with the problem as stated, you can also divide and conquer using GNU split (filtering is a GNU extension), repeated runs with chunks of file1 and reading file2 completely each time:
split -l 20000 --filter='gawk -f linesnotin.awk - file2' < file1
Note the use and placement of - meaning stdin on the gawk command line. This is provided by split from file1 in chunks of 20000 line per-invocation.
For users on non-GNU systems, there is almost certainly a GNU coreutils package you can obtain, including on OSX as part of the Apple Xcode tools which provides GNU diff, awk, though only a POSIX/BSD split rather than a GNU version.
Like konsolebox suggested, the posters grep solution
grep -v -f file2 file1
actually works great (faster) if you simply add the -F option, to treat the patterns as fixed strings instead of regular expressions. I verified this on a pair of ~1000 line file lists I had to compare. With -F it took 0.031 s (real), while without it took 2.278 s (real), when redirecting grep output to wc -l.
These tests also included the -x switch, which are necessary part of the solution in order to ensure totally accuracy in cases where file2 contains lines which match part of, but not all of, one or more lines in file1.
So a solution that does not require the inputs to be sorted, is fast, flexible (case sensitivity, etc) is:
grep -F -x -v -f file2 file1
This doesn't work with all versions of grep, for example it fails in macOS, where a line in file 1 will be shown as not present in file 2, even though it is, if it matches another line that is a substring of it. Alternatively you can install GNU grep on macOS in order to use this solution.
If you're short of "fancy tools", e.g. in some minimal Linux distribution, there is a solution with just cat, sort and uniq:
cat includes.txt excludes.txt excludes.txt | sort | uniq --unique
Test:
seq 1 1 7 | sort --random-sort > includes.txt
seq 3 1 9 | sort --random-sort > excludes.txt
cat includes.txt excludes.txt excludes.txt | sort | uniq --unique
# Output:
1
2
This is also relatively fast, compared to grep.
whats the speed of as sort and diff?
sort file1 -u > file1.sorted
sort file2 -u > file2.sorted
diff file1.sorted file2.sorted
Use combine from moreutils package, a sets utility that supports not, and, or, xor operations
combine file1 not file2
i.e give me lines that are in file1 but not in file2
OR give me lines in file1 minus lines in file2
Note: combine sorts and finds unique lines in both files before performing any operation but diff does not. So you might find differences between output of diff and combine.
So in effect you are saying
Find distinct lines in file1 and file2 and then give me lines in file1 minus lines in file2
In my experience, it's much faster than other options
This seems quick for me :
comm -1 -3 <(sort file1.txt) <(sort file2.txt) > output.txt
$ join -v 1 -t '' file1 file2
line2
line3
The -t makes sure that it compares the whole line, if you had a space in some of the lines.
You can use Python:
python -c '
lines_to_remove = set()
with open("file2", "r") as f:
for line in f.readlines():
lines_to_remove.add(line.strip())
with open("f1", "r") as f:
for line in f.readlines():
if line.strip() not in lines_to_remove:
print(line.strip())
'
Using of fgrep or adding -F option to grep could help. But for faster calculations you could use Awk.
You could try one of these Awk methods:
http://www.linuxquestions.org/questions/programming-9/grep-for-huge-files-826030/#post4066219
The way I usually do this is using the --suppress-common-lines flag, though note that this only works if your do it in side-by-side format.
diff -y --suppress-common-lines file1.txt file2.txt
I perform this operation very often, and I am looking for a shortcut. Is there any way I can do the following without having to write to a temp file?
cut -k 3-5 file1 > temp1
cat temp1 file2 | sort > outfile
Thanks!
Like this:
cut -k 3-5 file1 | cat - file2 | sort > outfile
There may be ancient versions of cat which do not take - to mean standard input.
Just do them in sequence:
(cut -k 3-5 file1; cat file2) | sort > outfile
This has the added advantage of working in any Bourne-based shell without requiring bash- or zsh-specific features.
This should do it:
cat <(cut -k 3-5 file1) file2 | sort > outfile
I just wanna to substract one CSV-File from another one, but not if the lines are the same. Instead of comparing the lines I'd like to look if the lines matching in one field.
e.g. the first file
EMAIL;NAME;SALUTATION;ID
foo#bar.com;Foo;Mr;1
bar#foo.com;Bar;Ms;2
and the second file
EMAIL;NAME
foo#bar.com;Foo
the resultfile should be
EMAIL;NAME;SALUTATION;ID
bar#foo.com;Bar;Ms;2
I think u know what I mean ;)
How is that possible in bash? It's easy for me doing this in Java, but I realy like to learn how to do that in bash. Also I can substract by comparing the lines using sort
#! / bin / bash
echo "Substracting Files..."
sort "/tmp/list1.csv" "/tmp/list2.csv" "/tmp/list2.csv" | uniq -u >> /tmp/subList.csv
echo "Files successfully substracted."
But the lines arn't the same tuple. So I have to compare line with keys.
Any suggestions? Thanks a lot.. Nils
One possible solution coming to my mind is this one (working with bash):
grep -v -f <(cut -d ";" -f1 /tmp/list2.csv) /tmp/list1.csv
That means:
cut -d ";" -f1 /tmp/list2.csv: Extract the first column of the second file.
grep -f some_file: Use a file as pattern source.
<(some_command): This is a process substitution. It executes the command and feeds the output to a named pipe which then can be used as file input to grep -f.
grep -v: Print only the lines not matching the pattern(s).
Update: the solution to the question, via join and awk.
join --header -1 1 -2 1 -t";" --nocheck-order -v 1 1.csv 2.csv | | awk 'NR==1 {print gensub(";[^;]\\+$","","g");next} 1'
These were the inverse answers:
$ join -1 1 -2 1 -t";" --nocheck-order -o 1.1,1.2,1.3,1.4 1.csv 2.csv
EMAIL;NAME;SALUTATION;ID
foo#bar.com;Foo;Mr;1
join to the rescue.
Or the skipping of printing the NAME field without -o:
$ join -1 1 -2 1 -t";" --nocheck-order 1.csv 2.csv | awk 'BEGIN {FS=";" ; OFS=";"} {$NF=""; print }'
(But it still prints a plus ;˛after the last field.
HTH
I need a script to edit files. Im going a bit crazy about this already :).
I've got two files:
143956;lorem
143957;ipsum
143958;lala
143959;vuvu
and second
512;143956;15
2156;143957;15
153;143958;4968
2156;143959;486
And what I need is to put those two files together in this way:
512;143956;lorem;15
2156;143957;ipsum;15
153;143958;lala;4968
2156;143959;vuvu;486
That doesn't look that difficult and probably paste would be enough, but theres a catch. There are rows which are only in one of those files but aren't in the other. In such case I need to wait on the line on the second file and still comparing to the first.
Example:
143956;lorem
143957;ipsum
143959;vuvu //here "lulu" is missing, will compare with 3rd line (143958) but the script wont declare this as "not found" but keep on searching till finds 143959 (which is already on 4 in this case).
512;143956;15
2156;143957;15
153;143958;4968
2156;143959;486
The output would look like this then:
512;143956;lorem;15
2156;143957;ipsum;15
2156;143959;vuvu;486
Or better this way
512;143956;lorem;15
2156;143957;ipsum;15
153;143958;*WAS NOT FOUND*;4968
2156;143959;vuvu;486
But I can finish that on my own...
Hope this is understandable. Thanks very much for any help.
Using Bash process substitution (<()) and the join utility:
join -t \; -1 1 -2 2 -o 2.1,2.2,1.2,2.3 <(sort file1) <(sort -t \; -k2,2 file2)
Or you can presort the files.
To output the records that appear in file2 but don't appear in file1:
join -t \; -1 1 -2 2 -v 2 -o 2.1,2.2,1.2,2.3 <(sort file1) <(sort -t \; -k2,2 file2) | sed 's/;;/;*WAS NOT FOUND*;/'
If the first file isn't too large, you can do (test1 and test2 are two files in the order you specified):
#!/bin/sh
for line in `cat test2`; do
number=`echo "$line" | grep -o ";[0-9]*;" | sed 's/;//g'`
repl=`grep "$number;" test1`
if [ -z "$repl" ]; then
echo "$line" | sed "s#;$number;#;$number;*WAS NOT FOUND*;#g"
else
echo "$line" | sed "s#;$number;#;$repl;#g"
fi
done