remove file starting with space in shell scripting - bash

I'm trying to write a shell script to cleanup a directory by deleting files that match particular patterns. My code works with all patterns but if the file name starts with space. Although we can delete a file starting with space by rm \ *however if I pass this pattern to my script it won't delete files starting with space. Here is my code:
for file in *;do
for pattern in $*; do
if [[ -f "$file" && "$file" == $pattern ]]; then
rm "$file"
fi
done
done
I also tried this simpler code, but the same problem!
for pattern in $*; do
if [[ -f $pattern ]]; then
rm $pattern
fi
done
Could you please help me why there is a problem just with files starting with space?!

Rather than $*, if you use the special parameter $#, the items in the list will start with quotes around them. You still have to quote the variables where you use them.
Reworking the second example, that would be
for pattern in "$#"; do
if [[ -f "$pattern" ]]; then
rm -f "$pattern"
fi
done

this is really a challenging one
for starters please see below example
[shravan#localhost mydir]$ ls " myfile"
myfile
[shravan#localhost mydir]$ echo $vr1
" myfile"
[shravan#localhost mydir]$ ls $vr1
ls: ": No such file or directory
ls: myfile": No such file or directory
[shravan#localhost mydir]$ vr2=" myfile"
[shravan#localhost mydir]$ echo $vr2
myfile
You can see above that ls " myfile" is working but it is not working after assigning this value in variable vr1 or vr2.
So we cannot do check of file if it exists or not.
For solution keep all you patterns in a file and all patterns in double quotes. see example below.
[shravan#localhost mydir]$ touch " myfile"
[shravan#localhost mydir]$ touch my.pl
[shravan#localhost mydir]$ ls
exe.sh findrm inp input myfile my.pl pattern text text1
[shravan#localhost mydir]$ cat inp
" myfile"
"my.pl"
[shravan#localhost mydir]$ cat inp | xargs rm
[shravan#localhost mydir]$ ls
exe.sh findrm inp input pattern text text1
The files are removed. Or if you have lot of patterns and dont want to add quotes to them use below.
cat inp | awk '{print "\""$0"\""}' | xargs rm
Yes if file is not found then it will give error for that file that
rm: cannot remove ` myfile': No such file or directory

for file in *;do
for pattern in "$#"; do
if [[ -f "$file" && "$file" == $pattern ]]; then
rm "$file"
fi
done
done
If we simply change $# to quoted "$#" then each individual argument would be wrapped in double quotation and no space would be lost. On the other hand we need a quoted string at the right of == operator, because when the '==' operator is used inside [[ ]], the string to the right of the operator is considered a pattern. But here we will not quote $pattern since all arguments in the list include double quotation.

Related

Removing spec characters with sed and mv but keeping / for destination, using full path

I am trying to remove special characters from specific files in files.txt. I need the mv command to use the full path to write the corrected file to the same location. The source and destination directories both contain spaces.
files.txt
/home/user/scratch/test2 2/capital:lets?.log
/home/user/scratch/test2 2/:31apples.tif
/home/user/scratch/test2 2/??testdoc1.txt
script.sh
#!/bin/bash
set -x
while IFS="" read -r p || [ -n "$p" ]
do
printf '%s\n' "$p"
mv "$p" $(echo "$p" | sed -e 's#[^A-Za-z0-9._-/]#_#g')
done < /home/user/scratch/files.txt
Here is the error that I get:
+ IFS=
+ read -r p
+ printf '%s\n' '/home/user/scratch/test2 2/??testdoc1.txt'
/home/user/scratch/test2 2/??testdoc1.txt
++ sed -e 's#[^A-Za-z0-9._-/]#_#g'
sed: -e expression #1, char 22: Invalid range end
++ echo '/home/user/scratch/test2 2/??testdoc1.txt'
+ mv '/home/user/scratch/test2 2/??testdoc1.txt'
mv: missing destination file operand after '/home/user/scratch/test2 2/??testdoc1.txt'
If I remove the / from sed -e 's#[^A-Za-z0-9._-]#_#g' command it will try to write the file like this:
++ sed -e 's#[^A-Za-z0-9._-]#_#g'
++ echo '/home/user/scratch/test2 2/??testdoc1.txt'
+ mv '/home/user/scratch/test2 2/??testdoc1.txt' _home_user_scratch_test2_2___testdoc1.txt
I have tried changing the delimiter in sed to something other than a / but the issue persists. If I try using mv "$p" "$(echo "$p" | sed -e 's|/[^/]*/\{0,1\}$||;s|^$|/|')" mv errors with this is the same file.
Am I approaching this problem wrong? This feels like it should have been an easier task.
EDIT:
The solution below gives me an issue with the file itself:
' echo '/mnt/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development/.Memeo 40'\'' flat w:boat plane.xls.plist
/mnt/data/bucket//Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development/.Memeo 40' flat w:boat plane.xls.plist
+ dir='/mnt/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development'
= */* ]]/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development/.Memeo 40' flat w:boat plane.xls.plist
' file='.Memeo 40'\'' flat w:boat plane.xls.plist
+ echo .Memeo '40'\''' flat w:boat $'plane.xls.plist\r'
.Memeo 40' flat w:boat plane.xls.plist
+ echo /mnt/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development
/mnt/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development
The actual filename is: .Memeo 40' flat w:boat plane.xls.plist
Why is it changing the filename when trying to do the move?
There are two problems in your substitution:
In the character class description [^A-Za-z0-9._-/], the last part
_-/ is interpreted as a range of characters between _ and /,
which is invalid. To avoid this, you need to escape the hyphen character
with a backslash, or put the hyphen at the beginning or the end of the
character class.
The directory name test2 2 includes the special character and
the sed command converts the directory name into test2_2,
which does not exist. Assuming you want to change the filenames only
keeping the directory names as is, we need to process the directory names
and filenames separately.
Then would you please try the following:
set -x
while IFS= read -r p || [ -n "$p" ]; do
echo "$p"
dir=${p%/*} # extract directory name
[[ $p = */* ]] || dir="." # in case $p does not contain "/"
file=${p##*/} # extract filename
mv -- "$p" "$dir/${file//[^-A-Za-z0-9._]/_}"
done < /home/user/scratch/files.txt

Looping list of folder path containing comma "," and spaces results in error

The folowing code work great but when the folder path contain "," and spaces make error
dir data/ > folder_file.txt
IFS=$'\n'
for file in "`cat folder_file.txt`"
do
printf 'File found: %s\n' "$file"
ls "data/$file/" #-----------> "," and "space" brook this task
done
any idea ? to escape special character
it work now any other advice's to make it better
IFS=$'\n'
a=0
for file in out/*; do
ls "$file" > html_file.txt
for file2 in `cat html_file.txt`; do
echo $file
mv "$file""/""$file2" "$file""/""page_"$a
let a=$a+1
done
a=0
done
This is how you loop on the content of a directory:
#!/bin/bash
shopt -s nullglob
for file in data/*; do
printf 'File found: %s\n' "$file"
ls "$file"
done
We use the shell options nullglob so that the glob * expands to nothing (and hence the loop is void) in case there are no matches.

rename file names work in command prompt but not in bash script

I'm trying to rename commands in a bash script. If I run for example:
echo /home/scientist/mySalesData/campaignData_1482386214.24417.csv | sed 's/\(.*\)\(_.*\)/mv \"&" \"\1.csv\"/' | bash
It works fine and gives me campaignData.csv in the directory /home/scientist/mySalesData/ .
However, if I put this in a bash script as follows:
for f in /home/scientist/SalesData/*; do
if [ -f "$f" ];
cp "$f" /home/scientist/SalesForce/SalesData/Backups/
echo $f$ | sed 's/\(.*\)\(_.*\)/mv \"&" \"\1.csv\"/' | bash |
fi
done
I get:
mv: cannot stat '/home/scientist/SalesData/campaignData_1482386214.24417.csv$': No such file or directory
Any help would be much appreciated!
cd "$srcdir"
for f in *; do
if [ -f "$f" ]; then
cp "./$f" "$dstdir/${f%_*}.csv"
fi
done
The % is the strip shortest suffix pattern operator.
You have a trailing $ here:
echo $f$
remove that (and quote the expansion):
echo "$f"
You could use here string too:
sed ... <<<"$f"

Find file names in other Bash files using grep

How do I loop through a list of Bash file names from an input text file and grep each file in a directory for each file name (to see if the file name is contained in the file) and output to text all file names that weren't found in any files?
#!/bin/sh
# This script will be used to output any unreferenced bash files
# included in the WebAMS Project
# Read file path of bash files and file name input
SEARCH_DIR=$(awk -F "=" '/Bash Dir/ {print $2}' bash_input.txt)
FILE_NAME=$(awk -F "=" '/Input File/ {print $2}' bash_input.txt)
echo $SEARCH_DIR
echo $FILE_NAME
exec<$FILE_NAME
while read line
do
echo "IN WHILE"
if (-z "$(grep -lr $line $SEARCH_DIR)"); then
echo "ENTERED"
echo $filename
fi
done
Save this as search.sh, updating SEARCH_DIR as appropriate for your environment:
#!/bin/bash
SEARCH_DIR=some/dir/here
while read filename
do
if [ -z "$(grep -lr $filename $SEARCH_DIR)" ]
then
echo $filename
fi
done
Then:
chmod +x search.sh
./search.sh files-i-could-not-find.txt
It could be possible through grep and find commands,
while read -r line; do (find . -type f -exec grep -l "$line" {} \;); done < file
OR
while read -r line; do grep -rl "$line"; done < file
-r --> recursive
-l --> files-with-matches(Displays the filenames which contains the search string)
It will read all the filenames present inside the input file and search for the filenames which contains the readed filenames. If it found any, then it returns the corresponding filename.
You're using regular parentheses instead of square brackets in your if statement.
The square brackets are a test command. You're running a test (in your case, whether a string has zero length or not. If the test is successful, the [ ... ] command returns an exit code of zero. The if statement sees that exit code and runs the then clause of the if statement. Otherwise, if an else statement exists, that is run instead.
Because the [ .. ] are actually commands, you must leave a blank space around each side.
Right
if [ -z "$string" ]
Wrong
if [-z "$string"] # Need white space around the brackets
Sort of wrong
if [ -z $sting ] # Won't work if "$string" is empty or contains spaces
By the way, the following are the same:
if test -z "$string"
if [ test -z "$string" ]
Be careful with that grep command. If there are spaces or newlines in the string returned, it may not do what you think it does.

How do I use Bash to create a copy of a file with an extra suffix before the extension?

This title is a little confusing, so let me break it down. Basically I have a full directory of files with various names and extensions:
MainDirectory/
image_1.png
foobar.jpeg
myFile.txt
For an iPad app, I need to create copies of these with the suffix #2X appended to the end of all of these file names, before the extension - so I would end up with this:
MainDirectory/
image_1.png
image_1#2X.png
foobar.jpeg
foobar#2X.jpeg
myFile.txt
myFile#2X.txt
Instead of changing the file names one at a time by hand, I want to create a script to take care of it for me. I currently have the following, but it does not work as expected:
#!/bin/bash
FILE_DIR=.
#if there is an argument, use that as the files directory. Otherwise, use .
if [ $# -eq 1 ]
then
$FILE_DIR=$1
fi
for f in $FILE_DIR/*
do
echo "Processing $f"
filename=$(basename "$fullfile")
extension="${filename##*.}"
filename="${filename%.*}"
newFileName=$(echo -n $filename; echo -n -#2X; echo -n $extension)
echo Creating $newFileName
cp $f newFileName
done
exit 0
I also want to keep this to pure bash, and not rely on os-specific calls. What am I doing wrong? What can I change or what code will work, in order to do what I need?
#!/bin/sh -e
cd "${1-.}"
for f in *; do
cp "$f" "${f%.*}#2x.${f##*.}"
done
It's very easy to do that with awk in one line like this:
ls -1 | awk -F "." ' { print "cp " $0 " " $1 "#2X." $2 }' | sh
with ls -1 you get just the bare list of files, then you pipe awk to use the dot (.) as separator. Then you build a shell command to create a copy of each file.
I suggest to run the command without the last sh pipe before, in order to check the cp commands are correct. Like this:
ls -1 | awk -F "." ' { print "cp " $0 " " $1 "#2X." $2 }'

Resources