The folowing code work great but when the folder path contain "," and spaces make error
dir data/ > folder_file.txt
IFS=$'\n'
for file in "`cat folder_file.txt`"
do
printf 'File found: %s\n' "$file"
ls "data/$file/" #-----------> "," and "space" brook this task
done
any idea ? to escape special character
it work now any other advice's to make it better
IFS=$'\n'
a=0
for file in out/*; do
ls "$file" > html_file.txt
for file2 in `cat html_file.txt`; do
echo $file
mv "$file""/""$file2" "$file""/""page_"$a
let a=$a+1
done
a=0
done
This is how you loop on the content of a directory:
#!/bin/bash
shopt -s nullglob
for file in data/*; do
printf 'File found: %s\n' "$file"
ls "$file"
done
We use the shell options nullglob so that the glob * expands to nothing (and hence the loop is void) in case there are no matches.
Related
I am trying to remove special characters from specific files in files.txt. I need the mv command to use the full path to write the corrected file to the same location. The source and destination directories both contain spaces.
files.txt
/home/user/scratch/test2 2/capital:lets?.log
/home/user/scratch/test2 2/:31apples.tif
/home/user/scratch/test2 2/??testdoc1.txt
script.sh
#!/bin/bash
set -x
while IFS="" read -r p || [ -n "$p" ]
do
printf '%s\n' "$p"
mv "$p" $(echo "$p" | sed -e 's#[^A-Za-z0-9._-/]#_#g')
done < /home/user/scratch/files.txt
Here is the error that I get:
+ IFS=
+ read -r p
+ printf '%s\n' '/home/user/scratch/test2 2/??testdoc1.txt'
/home/user/scratch/test2 2/??testdoc1.txt
++ sed -e 's#[^A-Za-z0-9._-/]#_#g'
sed: -e expression #1, char 22: Invalid range end
++ echo '/home/user/scratch/test2 2/??testdoc1.txt'
+ mv '/home/user/scratch/test2 2/??testdoc1.txt'
mv: missing destination file operand after '/home/user/scratch/test2 2/??testdoc1.txt'
If I remove the / from sed -e 's#[^A-Za-z0-9._-]#_#g' command it will try to write the file like this:
++ sed -e 's#[^A-Za-z0-9._-]#_#g'
++ echo '/home/user/scratch/test2 2/??testdoc1.txt'
+ mv '/home/user/scratch/test2 2/??testdoc1.txt' _home_user_scratch_test2_2___testdoc1.txt
I have tried changing the delimiter in sed to something other than a / but the issue persists. If I try using mv "$p" "$(echo "$p" | sed -e 's|/[^/]*/\{0,1\}$||;s|^$|/|')" mv errors with this is the same file.
Am I approaching this problem wrong? This feels like it should have been an easier task.
EDIT:
The solution below gives me an issue with the file itself:
' echo '/mnt/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development/.Memeo 40'\'' flat w:boat plane.xls.plist
/mnt/data/bucket//Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development/.Memeo 40' flat w:boat plane.xls.plist
+ dir='/mnt/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development'
= */* ]]/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development/.Memeo 40' flat w:boat plane.xls.plist
' file='.Memeo 40'\'' flat w:boat plane.xls.plist
+ echo .Memeo '40'\''' flat w:boat $'plane.xls.plist\r'
.Memeo 40' flat w:boat plane.xls.plist
+ echo /mnt/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development
/mnt/data/bucket/Desktop/For_the_New_Director/Part Number Assignment/__Prod_Development
The actual filename is: .Memeo 40' flat w:boat plane.xls.plist
Why is it changing the filename when trying to do the move?
There are two problems in your substitution:
In the character class description [^A-Za-z0-9._-/], the last part
_-/ is interpreted as a range of characters between _ and /,
which is invalid. To avoid this, you need to escape the hyphen character
with a backslash, or put the hyphen at the beginning or the end of the
character class.
The directory name test2 2 includes the special character and
the sed command converts the directory name into test2_2,
which does not exist. Assuming you want to change the filenames only
keeping the directory names as is, we need to process the directory names
and filenames separately.
Then would you please try the following:
set -x
while IFS= read -r p || [ -n "$p" ]; do
echo "$p"
dir=${p%/*} # extract directory name
[[ $p = */* ]] || dir="." # in case $p does not contain "/"
file=${p##*/} # extract filename
mv -- "$p" "$dir/${file//[^-A-Za-z0-9._]/_}"
done < /home/user/scratch/files.txt
I'm trying to run some command with looping through all files in a directory. The code is:
#!/bin/bash
shopt -s nullglob
INPUT_DIR=$1
OUTPUT_DIR=$2
: ${INPUT_DIR:="."}
: ${OUTPUT_DIR:="."}
files="$INPUT_DIR/*.ttf"
for file in $files
do
base_file=${file##*/}
output="$OUTPUT_DIR/${base_file%.*}.woff"
ttf2woff "$file" "$output" || exit 1
done
I'd expect the double qoutes around $INPUT_DIR/*.ttf would do the magic but apparently it's not:
$> ttf2woff_multi "/Users/ozan/Dropbox/Graphic Library/Google Fonts/fonts-master/ofl/raleway"
Can't open input file (/Users/ozan/Dropbox/Graphic)
and when I print out $FILES I get: /Users/ozan/Dropbox/Graphic Library/Google
What am I missing here?
Edit: files="$INPUT_DIR"/*.ttf instead of files="$INPUT_DIR/*.ttf" doesn't work either...
In addition to the array solution, (which is a good solution), you can also make use of read with process substitution:
INPUT_DIR=${1:=.}
OUTPUT_DIR=${2:=.}
[ -d "$INPUT_DIR" -a -d "$OUTPUT_DIR" ] || {
printf "error: invalid directory specified (INPUT_DIR or OUTPUT_DIR)\n"
exit 1
}
while IFS= read -r file; do
base_file=${file##*/}
output="$OUTPUT_DIR/${base_file%.*}.woff"
ttf2woff "$file" "$output" || exit 1
done < <(find "$INPUT_DIR" -type f -iname "*.ttf")
Since you want to loop through a list of files, better store them in an array:
files=("$INPUT_DIR"/*.ttf)
for file in "${files[#]}"
do
base_file=${file##*/}
output="$OUTPUT_DIR/${base_file%.*}.woff"
ttf2woff "$file" "$output" || exit 1
done
Note you were saying "$INPUT_DIR/*.ttf" whereas I am suggesting "$INPUT_DIR"/*.ttf. This is to allow the globbing to behave as intended and expand properly.
The key point here, as Cyrus mentions in comments, is the fact of not quoting, since they prevent globbing.
See an example with some files.
$ ls f*
f1 f2 f3
Store with double quotes... it just matches the string itself:
$ files=("f*")
$ for f in "${files[#]}"; do echo "$f"; done
f*
See how it is expanded if we do not quote:
$ files=(f*)
$ for f in "${files[#]}"; do echo "$f"; done
f1
f2
f3
I'm trying to write a shell script to cleanup a directory by deleting files that match particular patterns. My code works with all patterns but if the file name starts with space. Although we can delete a file starting with space by rm \ *however if I pass this pattern to my script it won't delete files starting with space. Here is my code:
for file in *;do
for pattern in $*; do
if [[ -f "$file" && "$file" == $pattern ]]; then
rm "$file"
fi
done
done
I also tried this simpler code, but the same problem!
for pattern in $*; do
if [[ -f $pattern ]]; then
rm $pattern
fi
done
Could you please help me why there is a problem just with files starting with space?!
Rather than $*, if you use the special parameter $#, the items in the list will start with quotes around them. You still have to quote the variables where you use them.
Reworking the second example, that would be
for pattern in "$#"; do
if [[ -f "$pattern" ]]; then
rm -f "$pattern"
fi
done
this is really a challenging one
for starters please see below example
[shravan#localhost mydir]$ ls " myfile"
myfile
[shravan#localhost mydir]$ echo $vr1
" myfile"
[shravan#localhost mydir]$ ls $vr1
ls: ": No such file or directory
ls: myfile": No such file or directory
[shravan#localhost mydir]$ vr2=" myfile"
[shravan#localhost mydir]$ echo $vr2
myfile
You can see above that ls " myfile" is working but it is not working after assigning this value in variable vr1 or vr2.
So we cannot do check of file if it exists or not.
For solution keep all you patterns in a file and all patterns in double quotes. see example below.
[shravan#localhost mydir]$ touch " myfile"
[shravan#localhost mydir]$ touch my.pl
[shravan#localhost mydir]$ ls
exe.sh findrm inp input myfile my.pl pattern text text1
[shravan#localhost mydir]$ cat inp
" myfile"
"my.pl"
[shravan#localhost mydir]$ cat inp | xargs rm
[shravan#localhost mydir]$ ls
exe.sh findrm inp input pattern text text1
The files are removed. Or if you have lot of patterns and dont want to add quotes to them use below.
cat inp | awk '{print "\""$0"\""}' | xargs rm
Yes if file is not found then it will give error for that file that
rm: cannot remove ` myfile': No such file or directory
for file in *;do
for pattern in "$#"; do
if [[ -f "$file" && "$file" == $pattern ]]; then
rm "$file"
fi
done
done
If we simply change $# to quoted "$#" then each individual argument would be wrapped in double quotation and no space would be lost. On the other hand we need a quoted string at the right of == operator, because when the '==' operator is used inside [[ ]], the string to the right of the operator is considered a pattern. But here we will not quote $pattern since all arguments in the list include double quotation.
How do I loop through a list of Bash file names from an input text file and grep each file in a directory for each file name (to see if the file name is contained in the file) and output to text all file names that weren't found in any files?
#!/bin/sh
# This script will be used to output any unreferenced bash files
# included in the WebAMS Project
# Read file path of bash files and file name input
SEARCH_DIR=$(awk -F "=" '/Bash Dir/ {print $2}' bash_input.txt)
FILE_NAME=$(awk -F "=" '/Input File/ {print $2}' bash_input.txt)
echo $SEARCH_DIR
echo $FILE_NAME
exec<$FILE_NAME
while read line
do
echo "IN WHILE"
if (-z "$(grep -lr $line $SEARCH_DIR)"); then
echo "ENTERED"
echo $filename
fi
done
Save this as search.sh, updating SEARCH_DIR as appropriate for your environment:
#!/bin/bash
SEARCH_DIR=some/dir/here
while read filename
do
if [ -z "$(grep -lr $filename $SEARCH_DIR)" ]
then
echo $filename
fi
done
Then:
chmod +x search.sh
./search.sh files-i-could-not-find.txt
It could be possible through grep and find commands,
while read -r line; do (find . -type f -exec grep -l "$line" {} \;); done < file
OR
while read -r line; do grep -rl "$line"; done < file
-r --> recursive
-l --> files-with-matches(Displays the filenames which contains the search string)
It will read all the filenames present inside the input file and search for the filenames which contains the readed filenames. If it found any, then it returns the corresponding filename.
You're using regular parentheses instead of square brackets in your if statement.
The square brackets are a test command. You're running a test (in your case, whether a string has zero length or not. If the test is successful, the [ ... ] command returns an exit code of zero. The if statement sees that exit code and runs the then clause of the if statement. Otherwise, if an else statement exists, that is run instead.
Because the [ .. ] are actually commands, you must leave a blank space around each side.
Right
if [ -z "$string" ]
Wrong
if [-z "$string"] # Need white space around the brackets
Sort of wrong
if [ -z $sting ] # Won't work if "$string" is empty or contains spaces
By the way, the following are the same:
if test -z "$string"
if [ test -z "$string" ]
Be careful with that grep command. If there are spaces or newlines in the string returned, it may not do what you think it does.
I have 2 files sorted by numerically. I need help with shell script to read these 2 files and do a 1:1 mapping and rename the filenames with the mapped case#;
For example:
cat case.txt
10_80
10_90
cat files.txt
A BCD_x 1.pdf
A BCD_x 2.pdf
ls pdf_dir
A BCD_x 1.pdf A BCD_x 2.pdf
Read these 2 txt and rename the pdf files in pdf_dir :
A BCD_x 1.pdf as A BCD_10_80.pdf
A BCD_x 1.pdf as A BCD_10_90.pdf
Use paste to create the "mapping", then shell facilities to do the renaming.
shopt -s extglob
while IFS=$'\t' read file replacement; do
echo mv "$file" "${file/x +([0-9])/$replacement}"
done < <(paste files.txt case.txt)
remove "echo" when you're satisfied.
Using awk:
awk 'FNR==NR{a[FNR]=$0;next}
{f=$0; sub(/_x /, "_" a[FNR] " "); system("mv \"" f "\" \"" $0 "\"")}' case.txt files.txt
Using normal array and sed substitution -
Removing echo before mv will provide you the move capability.
You can change the /path/to/pdf_dir/ to specify your path to desired directory
#!/bin/bash
i=0
while read line
do
arr[i]="$line"
((i=i+1));
done < files.txt
i=0
while read case
do
newFile=$(echo "${arr[i]}" | sed "s/x/"$case"/")
echo mv /path/to/pdf_dir/"${arr[i]}" /path/to/pdf_dir/"$newFile"
((i=i+1))
done < case.txt
If you have Bash 4.0 this could help:
#!/bin/bash
declare -A MAP
I=0
IFS=''
while read -r CASE; do
(( ++I ))
MAP["A BCD_x ${I}.pdf"]="A BCD_${CASE}.pdf"
done < case.txt
while read -r FILE; do
__=${MAP[$FILE]}
[[ -n $__ ]] && echo mv "$FILE" "$__" ## Remove echo when things seem right already.
done < files.txt
Note: Make sure you run the script in UNIX file format.