I'm trying to create a bash script that first looks for a name and then checks whether a certain filename, for example vacation021.jpg exists in the file system, and if it exists I want to append the filename to a .txt file.
I'm having a lot of issues with this, I'm still very new to bash.
This is as far as I've gotten.
> oldFiles.txt
files=$(grep "jane " list.txt)
for i in $files; do
if test -e vacation021.jpg;
then echo $i >> oldFiles.txt; fi
done
This however appends all the separate words in the list.txt to the oldFiles.txt.
Any help would be much appreciated.
for i in $files will iterate over each word in $files, not the lines. If you want to iterate over the lines, pipe the output of grep to the loop:
grep 'jane ' list.txt | while read -r i; do
if test -e vacation021.jpg
then printf "%s" "%i"
fi
done > oldFiles.txt
But as mentioned in the comments, unless the vacation021.jpg file is going to be created or deleted during the loop, you can simply use a single if:
if test -e vacation021.jpg
then
grep 'jane ' list.txt > oldFiles.txt
fi
Related
I have a file of strings that I need to search for in another file. So I've tried the following code:
#!/bin/bash
while read name; do
#echo $name
grep "$name" file.txt > results.txt
done < missing.txt
The echo line confirms the file is being read into the variable, but my results file is always empty. Doing the grep command on its own works, I'm obviously missing something very basic here but I have been stuck for a while and can't figure it out.
I've also tried without quotes around the variable. Can someone tell me what I'm missing? Thanks a bunch
Edit - input file was DOS format, set file format to unix and works fine now
Use grep's -f option: Then you only need a single grep call and no loop.
grep -f missing.txt file.txt > results.txt
If the contents of "missing.txt" are fixed strings, not regular expressions, this will speed up the process:
grep -F -f missing.txt file.txt > results.txt
And if you want to find the words of missing.txt in the other file, not partial words
grep -F -w -f missing.txt file.txt > results.txt
My first guess is that you are overwriting your results.txt file in every iteration of the while loop (with the single >). If it is the case you should at least have the result for the very last line in your missing.txt file. Then I think it would suffice to do something like
#!/bin/bash
while read name; do
#echo "$name"
grep "$name" file.txt
done < missing.txt > results.txt
I have a file which consists of number of users which I need to compare it with multiple files and print if any particular user is not present in the files with filename.
#!/bin/bash
awk '{print $1}' $1 | while read -r line; do
if ! grep -q "$line" *.txt;
then
echo "$line User doesn't exist"
fi
done
In the above script, passing user_list file as $1 and can able to find the users for single target file, but it fails for multiple files.
File contents:
user_list:
Johnny
Stella
Larry
Jack
One of the multiple files contents:
root:x:0:0:root:/root:/bin/bash
Stella:x:1:1:Admin:/bin:/bin/bash
Jack:x:2:2:admin:/sbin:/bin/bash
Usage:
./myscript user_list.txt
Desired output:
File1:
Stella doesn't exist
Jack doesn't exist
File2:
Larry doesn't exist
Johnny doesn't exist
Any suggestion here to achieve it for multiple files with printing filename headers?
Use a for loop to iterate over each file and execute your code for each file separately.
#!/bin/bash
for f in *.txt; do
echo $f:
awk '{print $1}' $1 | while read -r line; do
if ! grep -q "$line" $f
then
echo "$line doesn't exist"
fi
done
echo
done
You could do simply
for file in *.txt; do
echo "$file"
grep -Fvf "$file" "$1"
echo
done
This might do what you want.
#!/usr/bin/env bash
for f in *.txt; do ##: Loop through the *.txt files
for j; do ##: Loop through the argument files, file1 file2 file3
printf '\n%s:\n' "$j" ##: Print one of the multiple files.
while read -r lines; do ##: Read line-by-line & one-by-one all the *.txt files
if ! grep -q "$lines" "$j"; then ##: If grep did not found a match.
printf "%s not doesn't exist.\n" "$lines" ##: Print the desired output.
fi
done < "$f"
done
done
The *.txt files should be in the current directory, otherwise add the absolute path, e.g. /path/to/files/*.txt
Howto use.
./myscript file1 file2 file3 ...
The downside is you're running grep line-by-line on each files as opposed to what #Quasimodo did.
I am relatively new to bash and I am testing my code for the first case.
counter=1
for file in not_processed/*.txt; do
if [ $counter -le 1 ]; then
grep -v '2018-07' $file > bis.txt;
counter=$(($counter+1));
fi;
done
I want to subtract all the lines containing '2018-07' from my file. The new file needs to be named $file_bis.txt.
Thanks
With sed or awk it's much easier and faster to process complex files.
sed -n '/2018-07/p' not_processed/*.txt
then you get the output in your console. If you want you can pipe the output to a new file.
sed -n '/2018-07/p' not_processed/*.txt >> out.txt
This is to do it on all files in not_processed/*.txt
for file in not_processed/*.txt
do
grep -v '2018-07' $file > "$file"_bis.txt
done
And this is to do it only on the first 2 files in not_processed/*.txt
for file in $(ls not_processed/*.txt|head -2)
do
grep -v '2018-07' $file > "$file"_bis.txt
done
Don't forget to add "" on $file, because otherwise bash considers $file_bis as a new variable, which has no assigned value.
I don't understood why you are using a counter and if condition for this simple requirement. Use below script which will fulfill you requirement:-
#first store all the files in a variable
files=$(ls /your/path/*.txt)
# now use a for loop
for file in $files;
do
grep '2018-07' $file >> bis.txt
done
Better avoid for loop here as below single line is suffice
grep -h '2018-07' /your/path/*.txt > bis.txt
I was working on a program that could transfer files using sftp program:
sftp -oBatchMode=no -b ${BATCH_FILE} user#$123.123.123.123:/home << EOF
bye
EOF
One of my requirement is I must have a BATCH_FILE use with sftp and the batch file was generate using following script:
files=$(ls -1 ${SRC_PATH}/*.txt)
echo "$files" > ${TEMP_FILE}
while read file
do
if [ -s "${file}" ]
then
echo ${file} >> "${PARSE_FILE}" ## line 1
fi
done < ${TEMP_FILE}
awk '$0="put "$0' ${PARSE_FILE} > ${BATCH_FILE}
Somehow my program doesn't able to handle files with space in it. I did try using following code to replace line 1 but failed, the output of this will show filename\.txt.
newfile=`echo $file | tr ' ' '\\ '`
echo ${newfile} >> "${PARSE_FILE}"
In order to handle file name with space, how could I append a \ in front of the space within a file name?
THE PROBLEM
The problem is that tr SET1 SET2 will replace the Nth character in SET1 with the Nth character in SET2, which means that you are effectively replacing every space by \, instead of adding a backslash before every space.
PROPOSED SOLUTION
Instead of manually trying to fix the missing spaces, upon using your variable that might contain spaces; wrap it in quotes and let the shell handle the trouble for you.
See the below example:
$ echo $FILENAME
file with spaces.txt
$ ls $FILENAME
ls: cannot access file: No such file or directory
ls: cannot access with: No such file or directory
ls: cannot access spaces.txt: No such file or directory
$ ls "$FILENAME"
file with spaces.txt
But I really wanna replace stuff..
Well, if you really want a command to change every ' ' (space) into '\ ' (backslash, space) you could use sed with a basic replace-pattern, as the below:
$ echo "file with spaces.txt" | sed 's, ,\\ ,g'
file\ with\ spaces.txt
I haven't looked too closely at what you're trying to do there, but I do know that bash can handle filenames with spaces in them if you double-quote them. Why not try quoting every filename variable and see if that works? You're quoting some of them but not all yet.
Like try these: "${newfile}" or just "$newfile" "$file" "$tempfile" etc...
You can further simplify your code if you're using Bash:
function generate_batch_file {
for FILE in "${SRC_PATH}"/*.txt; do
[[ -s $FILE ]] && echo "put {$FILE// /\\ }"
done
}
sftp -oBatchMode=no -b <(generate_batch_file) user#$123.123.123.123:/home <<< "bye"
you can try to rename the file to work and rename it again after it has done.
I am trying to modify a bash script to remove a glob of malicious code from a large number of files.
The community will benefit from this, so here it is:
#!/bin/bash
grep -r -l 'var createDocumentFragm' /home/user/Desktop/infected_site/* > /home/user/Desktop/filelist.txt
for i in $(cat /home/user/Desktop/filelist.txt)
do
cp -f $i $i.bak
done
for i in $(cat /home/user/Desktop/filelist.txt)
do
$i | sed 's/createDocumentFragm.*//g' > $i.awk
awk '/<\/SCRIPT>/{p=1;print}/<\/script>/{p=0}!p'
This is where the script bombs out with this message:
+ for i in '$(cat /home/user/Desktop/filelist.txt)'
+ sed 's/createDocumentFragm.*//g'
+ /home/user/Desktop/infected_site/index.htm
I get 2 errors and the script stops.
/home/user/Desktop/infected_site/index.htm: line 1: syntax error near unexpected token `<'
/home/user/Desktop/infected_site/index.htm: line 1: `<html><head><script>(function (){ '
I have the first 2 parts done.
The files containing createDocumentfragm have been enumerated in a text file correctly.
The files in the textfile.txt have been duplicated, in their original location with a .bak added to them IE: infected_site/some_directory/infected_file.htm and infected_file.htm.bak
effectively making sure we have a backup.
All I need to do now is write an AWK command that will use the list of files in filelist.txt, use the entire glob of malicious text as a pattern, and remove it from the files. Using just the uppercase script as the starting point, and the lower case script is too generic and could delete legitimate text
I suspect this may help me, but I don't know how to use it correctly.
http://backreference.org/2010/03/13/safely-escape-variables-in-awk/
Once I have this part figured out, and after you have verified that the files weren't mangled you can do this to clean out the bak files:
for i in $(cat /home/user/Desktop/filelist.txt)
do
rm -f $i.bak
done
Several things:
You have:
$i | sed 's/var createDocumentFragm.*//g' > $i.awk
You should probably meant this (using your use of cat which we'll talk about in a moment):
cat $i | sed 's/var createDocumentFragm.*//g' > $i.awk
You're treating each file in your file list as if it was a command and not a file.
Now, about your use of cat. If you're using cat for almost anything but concatenating multiple files together, you probably are doing something not quite right. For example, you could have done this:
sed 's/var createDocumentFragm.*//g' "$i" > $i.awk
I'm also a bit confused about the awk statement. Exactly what file are you using awk on? Your awk statement is using STDIN and STDOUT, so it's reading file names from the for loop and then printing the output on the screen. Is the sed statement suppose to feed into the awk statement?
Note that I don't have to print out my file to STDOUT, then pipe that into sed. The sed command can take the file name directly.
You also want to avoid for loops over a list of files. That is very inefficient, and can cause problems with the command line getting overloaded. Not a big issue today, but can affect you when you least suspect it. What happens is that your $(cat /home/user/Desktop/filelist.txt) must execute first before the for loop can even start.
A little rewriting of your program:
cd ~/Desktop
grep -r -l 'var createDocumentFragm' infected_site/* > filelist.txt
while read file
do
cp -f "$file" "$file.bak"
sed 's/var createDocumentFragm.*//g' "$file" > "$i.awk"
awk '/<\/SCRIPT>/{p=1;print}/<\/script>/{p=0}!p'
done < filelist.txt
We can use one loop, and we made it a while loop. I could even feed the grep into that while loop:
grep -r -l 'var createDocumentFragm' infected_site/* | while read file
do
cp -f "$file" "$file.bak"
sed 's/var createDocumentFragm.*//g' "$file" > "$i.awk"
awk '/<\/SCRIPT>/{p=1;print}/<\/script>/{p=0}!p'
done < filelist.txt
and then I don't even have to create a temporary file.
Let me know what's going on with the awk. I suspect you wanted something like this:
grep -r -l 'var createDocumentFragm' infected_site/* | while read file
do
cp -f "$file" "$file.bak"
sed 's/var createDocumentFragm.*//g' "$file" \
| awk '/<\/SCRIPT>/{p=1;print}/<\/script>/{p=0}!p' > "$i.awk"
done < filelist.txt
Also note I put quotes around file names. This helps prevent problems if file name has a space in it.