I need to output all lines in file1.txt, except those that are found in file2.txt, by matching the entirely lines.
E.g., file1.txt:
Cats eat fish.
Mice eat cheese.
Eagles can fly.
Mountains are tall.
E.g., file2.txt:
Cats eat fish.
Birds can fly.
Trees are tall.
E.g., output:
Mice eat cheese.
Eagles fly.
Mountains are tall.
I have used the following command:
grep -v -x -f file1.txt file2.txt
This appears to work, however, when the files are of certain lengths, it often reports grep: memory exhausted, so I need an alternative that will not create this memory problem.
The order of the lines is important, so they should not be sorted.
Any tool typically found within a default Linux install will be acceptable.
How can I output the lines of file1.txt, except those found in file2.txt, without encountering a memory problem?
Try:
grep -Fxvf file2.txt file1.txt
References: find difference between two text files with one item per line
try:
rm -f out.txt && while read -r line; do echo "checking if line $line exists in file2.txt"; if `grep -Fxq "$line" file2.txt`; then echo "$line exists in other for"; else echo "$line" >> out.txt; fi; done < file.txt
Explaination:
this deletes the output file (in case of continuous use...), then it checks line by line if one exists in the other.
As a bash file it's clearer:
rm out.txt
while read -r line
do
echo "checking if line $line exists in file2.txt"
if `grep -Fxq "$line" file2.txt`
then
echo "$line exists in other file"
else
echo "$line" >> out.txt
fi
done < file.txt
And the obvious generalization:
while read -r line
do
echo "checking if line $line exists in $2"
if `grep -Fxq "$line" $2`
then
echo "$line exists in $2"
else
echo "$line" >> $3
fi
done < $1
Where the first and second arguments would be the files and the outut file would be the third argument
Related
I want to be able to comment and uncomment lines which are "managed" using a bash script.
I am trying to write a script which will update all of the config lines which have the word #managed after them and remove the preceeding # if it exists.
The rest of the config file needs to be left unchanged. The config file looks like this:
configFile.txt
#config1=abc #managed
#config2=abc #managed
config3=abc #managed
config3=abc
This is the script I have created so far. It iterates the file, finds lines which contain "#managed" and detects if they are currently commented.
I need to then write this back to the file, how do I do that?
manage.sh
#!/bin/bash
while read line; do
STR='#managed'
if grep -q "$STR" <<< "$line"; then
echo "debug - this is managed"
firstLetter=${$line:0:1}
if [ "$firstLetter" = "#" ]; then
echo "Remove the initial # from this line"
fi
fi
echo "$line"
done < configFile.txt
With your approach using grep and sed.
str='#managed$'
file=ConfigFile.txt
grep -q "^#.*$str" "$file" && sed "/^#.*$str/s/^#//" "$file"
Looping through files ending in *.txt
#!/usr/bin/env bash
str='#managed$'
for file in *.txt; do
grep -q "^#.*$str" "$file" &&
sed "/^#.*$str/s/^#//" "$file"
done
In place editing with sed requires the -i flag/option but that varies from different version of sed, the GNU version does not require an -i.bak args, while the BSD version does.
On a Mac, ed should be installed by default, so just replace the sed part with.
printf '%s\n' "g/^#.*$str/s/^#//" ,p Q | ed -s "$file"
Replace the Q with w to actually write back the changes to the file.
Remove the ,p if no output to stdout is needed/required.
On a side note, embedding grep and sed in a shell loop that reads line-by-line the contents of a text file is considered a bad practice from shell users/developers/coders. Say the file has 100k lines, then grep and sed would have to run 100k times too!
This sed one-liner should do the trick:
sed -i.orig '/#managed/s/^#//' configFile.txt
It deletes the # character at the beginning of the line if the line contains the string #managed.
I wouldn't do it in bash (because that would be slower than sed or awk, for instance), but if you want to stick with bash:
#! /bin/bash
while IFS= read -r line; do
if [[ $line = *'#managed'* && ${line:0:1} = '#' ]]; then
line=${line:1}
fi
printf '%s\n' "$line"
done < configFile.txt > configFile.tmp
mv configFile.txt configFile.txt.orig && mv configFile.tmp configFile.txt
I'm trying to create a bash script that first looks for a name and then checks whether a certain filename, for example vacation021.jpg exists in the file system, and if it exists I want to append the filename to a .txt file.
I'm having a lot of issues with this, I'm still very new to bash.
This is as far as I've gotten.
> oldFiles.txt
files=$(grep "jane " list.txt)
for i in $files; do
if test -e vacation021.jpg;
then echo $i >> oldFiles.txt; fi
done
This however appends all the separate words in the list.txt to the oldFiles.txt.
Any help would be much appreciated.
for i in $files will iterate over each word in $files, not the lines. If you want to iterate over the lines, pipe the output of grep to the loop:
grep 'jane ' list.txt | while read -r i; do
if test -e vacation021.jpg
then printf "%s" "%i"
fi
done > oldFiles.txt
But as mentioned in the comments, unless the vacation021.jpg file is going to be created or deleted during the loop, you can simply use a single if:
if test -e vacation021.jpg
then
grep 'jane ' list.txt > oldFiles.txt
fi
I have a file which consists of number of users which I need to compare it with multiple files and print if any particular user is not present in the files with filename.
#!/bin/bash
awk '{print $1}' $1 | while read -r line; do
if ! grep -q "$line" *.txt;
then
echo "$line User doesn't exist"
fi
done
In the above script, passing user_list file as $1 and can able to find the users for single target file, but it fails for multiple files.
File contents:
user_list:
Johnny
Stella
Larry
Jack
One of the multiple files contents:
root:x:0:0:root:/root:/bin/bash
Stella:x:1:1:Admin:/bin:/bin/bash
Jack:x:2:2:admin:/sbin:/bin/bash
Usage:
./myscript user_list.txt
Desired output:
File1:
Stella doesn't exist
Jack doesn't exist
File2:
Larry doesn't exist
Johnny doesn't exist
Any suggestion here to achieve it for multiple files with printing filename headers?
Use a for loop to iterate over each file and execute your code for each file separately.
#!/bin/bash
for f in *.txt; do
echo $f:
awk '{print $1}' $1 | while read -r line; do
if ! grep -q "$line" $f
then
echo "$line doesn't exist"
fi
done
echo
done
You could do simply
for file in *.txt; do
echo "$file"
grep -Fvf "$file" "$1"
echo
done
This might do what you want.
#!/usr/bin/env bash
for f in *.txt; do ##: Loop through the *.txt files
for j; do ##: Loop through the argument files, file1 file2 file3
printf '\n%s:\n' "$j" ##: Print one of the multiple files.
while read -r lines; do ##: Read line-by-line & one-by-one all the *.txt files
if ! grep -q "$lines" "$j"; then ##: If grep did not found a match.
printf "%s not doesn't exist.\n" "$lines" ##: Print the desired output.
fi
done < "$f"
done
done
The *.txt files should be in the current directory, otherwise add the absolute path, e.g. /path/to/files/*.txt
Howto use.
./myscript file1 file2 file3 ...
The downside is you're running grep line-by-line on each files as opposed to what #Quasimodo did.
I would like to make a bash script that uses 2 arguments, file1 file2 that copies all lines from the file1 that contains the letter b into file2 . I have found the solution to determine if a string is contains the letter
if [[ $string == *"b"* ]]; then
echo "It's there!"
fi
I just can figure how to apply this code to my problem, and run through each line of a random file.
In the course description i have found that this problem can be solved with the usage of head -n tail -n cat echo wc -c wc -l wc -w if case test , but we don't have to limit ourselves to the usage of just these commands.
This is the reason why grep has been invented:
grep "b" file1.txt >>file2.txt
(This copies all lines from file1.txt, containing the character b, to file2.txt)
How can I move all of the lines in a file which contain at least one matching phrase to the end of the file? E.g., file:
Do you like to read books?
Yes, do you like to watch movies?
No, but the fish does.
If the search phrases were "book" and "movie", then the first two lines above would move to the end of the file, e.g.:
No, but the fish does.
Do you like to read books?
Yes, do you like to watch movies?
Here's a quick and dirty way:
(grep -v -e movie -e book file; grep -e movie -e book file) >newfile
Is this what you're looking for?
grep "match1|match2" input.txt > tmp1
grep -v "match1|match2" input.txt > tmp2
cat tmp2 tmp1 > output.txt
rm tmp1 tmp2
Or, as pointed by Kevin, without using temporary files:
cat <(grep "match1|match2" input.txt) <(grep -v "match1|match2" input.txt) > output.txt
here is another way in full bash:
#!/bin/bash -
declare -a HEAD
declare -a BOTTOM
while read -r line
do
case "$line" in
*book*|*movie*)
BOTTOM[${#BOTTOM[*]}]="${line}";
;;
*)
HEAD[${#HEAD[*]}]="${line}";
;;
esac # --- end of case ---
done < "$1"
for i in "${HEAD[#]}" "${BOTTOM[#]}"; do echo $i; done