Update 01/12/2022
With triplee's helpful suggestions, I resolved it to take both files & directories by adding a comma in between f and d, the final code now looks like this:
while read -r old new;
do echo "replacing ${old} by ${new}" >&2
find '/path/to/dir' -depth -type d,f -name "$old" -exec rename
"s/${old}/${new}/" {} ';'
done <input.txt
Thank you!
Original request:
I am trying to rename a list of files (from $old to $new), all present in $homedir or in subdirectories in $homedir.
In the command line this line works to rename files in the subfolders:
find ${homedir}/ -name ${old} -exec rename "s/${old}/${new}/" */${old} ';'
However, when I want to implement this line in a simple bash script getting the $old and $new filenames from input.txt, it doesn't work anymore...
input.txt looks like this:
name_old name_new
name_old2 name_new2
etc...
the script looks like this:
#!/bin/bash
homedir='/path/to/dir'
cat input.txt | while read old new;
do
echo 'replacing' ${old} 'by' ${new}
find ${homedir}/ -name ${old} -exec rename "s/${old}/${new}/" */${old} ';'
done
After running the script, the text line from echo with $old and $new filenames being replaced is printed for the entire loop, but no files are renamed. No error is printed either. What am I missing? Your help would be greatly appreaciated!
I checked whether the $old and $new variables were correctly passed to the find -exec rename command, but because they are printed by echo that doesn't seem to be the issue.
If you add an echo, like -exec echo rename ..., you'll see what actually gets executed. I'd say that both the path to $old is wrong (you're not using the result of find in the -exec clause), and */$old isn't quoted and might be expanded by the shell before find ever gets to see it.
You're also having most other expansions unquoted, which can lead to all sorts of trouble.
You could do it in pure Bash (drop echo when output looks good):
shopt -s globstar
for f in **/"$old"; do echo mv "$f" "${f/%*/$new}"; done
Or with rename directly, though this would run into trouble if too many files match (drop -n when output looks good):
rename -n "s/$old\$/$new/" **/"$old"
Or with GNU find, using -execdir to run in the same directory as the matching file (drop echo when output looks good):
find -type f -name "$old" -execdir echo mv "$old" "$new" \;
And finally, a version with find that spawns just a single subshell (drop echo when output looks right):
find -type f -name "$old" -exec bash -c '
new=$1
shift
for f; do
echo mv "$f" "${f/%*/$new}"
done
' bash "$new" {} +
The argument to rename should be the file itself, not */${old}. You also have a number of quoting errors, and a useless cat).
#!/bin/bash
while read -r old new;
do
echo "replacing ${old} by ${new}" >&2
find /path/to/dir -name "$old" -exec rename "s/${old}/${new}/" {} ';'
done <input.txt
Running find multiple times on the same directory is hugely inefficient, though. Probably a better solution is to find all files in one go, and abort if it's not one of the files on the list.
find /path/to/dir -type f -exec sh -c '
for f in "$#"; do
awk -v f="$f" "f==\$1 { print \"s/\" \$1 \"/\" \$2 \"/\" }" "$0" |
xargs -I _ -r rename _ "$f"
done' input.txt {} +
(Untested; probably try with echo before you run this live.)
In Bash, I wish to rename many files based upon a predefined dict with many replacement strings. I have multiple files nested in a directory tree, like:
./aa
./b/aa
./b/bb
./c/aa
./c/d/ee
I have a "sed script" dict.sed whose contents is like:
s|aa|xx|g
s|ee|yy|g
Can I recursively find and rename files matching aa and ee to xx and yy, respectively, and preserving the directory structure, using said sed script?
At the moment I have:
function rename_from_sed() {
IFS='|' read -ra SEDCMD <<< "$1"
find "." -type f -name "${SEDCMD[1]}" -execdir mv {} "${SEDCMD[2]}" ';'
}
while IFS='' read -r line || [[ -n "${line}" ]]; do
rename_from_sed "$line"
done < "dict.sed"
This seems to work, but is there a better way, perhaps using sed -f dict.sed instead of parsing dict.sed line-by-line? The current approach means I need to specify the delimiter, and is also limited to exact filenames.
The closest other answer is probably https://stackoverflow.com/a/11709995/3329384 but that also has the limitation that directories may also be renamed.
That seems odd: iterating over the lines in a sed script. You want:
Can I recursively find and rename files
So iterate over files, not over the sed script.
find .... | while IFS= read -r file; do
newfile=$(sed -f dict.sed <<<"$file")
mv "$file" "$newfile"
done
Related https://mywiki.wooledge.org/BashFAQ/001 and https://mywiki.wooledge.org/BashFAQ/020
but is there a better way
Sure - do not use Bash and do use Sed. Write the whole script in Python or Perl - it will be 1000 times faster to use a tool which will do all the operations in a single thread without any fork() calls.
Anyway, you can also parse sed script line by line and run a single find call.
find "." -type f '(' -name aa -execdir mv {} xx ';' ')' -o '(' -name ee -execdir mv {} yy ';' ')'
You would have to build such call, like:
findargs=()
while IFS='|' read -r _ f t _; do
if ((${#findargs[#]})); then findargs+=('-o'); fi
findargs+=( '(' -name "$f" -execdir {} "$t" ';' ')' )
done < "dict.sed"
find . -type f "${findargs[#]}"
Assuming lines in the file dict is in the form of from|to, below is an implementation in pure bash, without using find and sed.
#!/bin/bash
shopt -s globstar nullglob
while IFS='|' read -r from to; do
for path in ./**/*"$from"*; do
[[ -f $path ]] || continue
basename=${path##*/}
echo mv "$path" "${path%/*}/${basename//"$from"/"$to"}"
done
done < dict
--
$ cat dict
aa|xx
ee|yy
Drop the echo if you are satisfied with the output.
I'm making a bash script that crawls through a directory and outputs all files of a certain type into a text file. I've got that working, it just also writes out a bunch of output to console I don't want (the names of the files)
Here's the relevant code so far, tmpFile is the file I'm writing to:
for DIR in `find . -type d` # Find problem directories
do
for FILE in `ls "$DIR"` # Loop through problems in directory
do
if [[ `echo ${FILE} | grep -e prob[0-9]*_` ]]; then
`echo ${FILE} >> ${tmpFile}`
fi
done
done
The files I'm putting into the text file are in the format described by the regex prob[0-9]*_ (something like prob12345_01)
Where I pipe the output from echo ${FILE} into grep, it still outputs to stdout, something I want to avoid. I think it's a simple fix, but it's escaping me.
All this can be done in one single find command. Consider this:
find . -type f -name "prob[0-9]*_*" -exec echo {} >> ${tmpFile} \;
EDIT:
Even simpler: (Thanks to #GlennJackman)
find . -type f -name "prob[0-9]*_*" >> $tmpFile
To answer your specific question, you can pass -q to grep for silent output.
if echo "hello" | grep -q el; then
echo "found"
fi
But since you're already using find, this can be done with just one command:
find . -regex ".*prob[0-9]*_.*" -printf '%f\n' >> ${tmpFile}
find's regex is a match on the whole path, which is why the leading and trailing .* is needed.
The -printf '%f\n' prints the file name without directory, to match what your script is doing.
what you want to do is, read the output of the find command,
for every entry find returned, you want to get all (*) the files under that location
and then you want to check whether that filename matches the pattern you want
if it matches then add it to the tmpfile
while read -r dir; do
for file in "$dir"/*; do # will not match hidden files, unless dotglob is set
if [[ "$file" =~ prob[0-9]*_ ]]; then
echo "$file" >> "$tmpfile"
fi
done < <(find . -type d)
however find can do that alone
anubhava got me there ;)
so look his answer on how that's done
I'm trying to find a bash script that will recursively look for files with a .bx extension, and remove this extension. The filenames are in no particular format (some are hidden files with "." prefix, some have spaces in the name, etc.), and not all files have this extension.
I'm not sure how to find each file with the .bx extension (in and below my cwd) and remove it. Thanks for the help!
find . -name '*.bx' -type f | while read NAME ; do mv "${NAME}" "${NAME%.bx}" ; done
find -name "*.bx" -print0 | xargs -0 rename 's/\.bx//'
Bash 4+
shopt -s globstar
shopt -s nullglob
shopt -s dotglob
for file in **/*.bx
do
mv "$file" "${file%.bx}"
done
Assuming you are in the folder from where you want to do this
find . -name "*.bx" -print0 | xargs -0 rename .bx ""
for blah in *.bx ; do mv ${blah} ${blah%%.bx}
Here is another version which does the following:
Finds out files based on $old_ext variable (right now set to .bx) in and below cwd, stores them in $files
Replaces those files' extension to nothing (or something new depending on $new_ext variable, currently set to .xyz)
The script uses dirname and basename to find out file-path and file-name respectively.
#!/bin/bash
old_ext=".bx"
new_ext=".xyz"
files=$(find ./ -name "*${old_ext}")
for file in $files
do
file_name=$(basename $file $old_ext)
file_path=$(dirname $file)
new_file=${file_path}/${file_name}${new_ext}
#echo "$file --> $new_file"
mv "$file" "$new_file"
done
Extra: How to remove any extension from filenames
find -maxdepth 1 -type f | sed 's/.\///g'| grep -E [.] | while read file; do mv $file ${file%.*}; done
will cut starting from last dot, i.e. pet.cat.dog ---> pet.cat
find -maxdepth 1 -type f | sed 's/.\///g'| grep -E [.] | while read file; do mv $file ${file%%.*}; done
will cut starting from first dot, i.e. pet.cat.dog ---> pet
"-maxdepth 1" limits operation to current directory, "-type f" is used to select files only. Sed & grep combination is used to pick only filenames with dot. Number of percent signs in "mv" command will define actual cut point.
I have a bash shell script that loops through all child directories (but not files) of a certain directory. The problem is that some of the directory names contain spaces.
Here are the contents of my test directory:
$ls -F test
Baltimore/ Cherry Hill/ Edison/ New York City/ Philadelphia/ cities.txt
And the code that loops through the directories:
for f in `find test/* -type d`; do
echo $f
done
Here's the output:
test/Baltimore
test/Cherry
Hill
test/Edison
test/New
York
City
test/Philadelphia
Cherry Hill and New York City are treated as 2 or 3 separate entries.
I tried quoting the filenames, like so:
for f in `find test/* -type d | sed -e 's/^/\"/' | sed -e 's/$/\"/'`; do
echo $f
done
but to no avail.
There's got to be a simple way to do this.
The answers below are great. But to make this more complicated - I don't always want to use the directories listed in my test directory. Sometimes I want to pass in the directory names as command-line parameters instead.
I took Charles' suggestion of setting the IFS and came up with the following:
dirlist="${#}"
(
[[ -z "$dirlist" ]] && dirlist=`find test -mindepth 1 -type d` && IFS=$'\n'
for d in $dirlist; do
echo $d
done
)
and this works just fine unless there are spaces in the command line arguments (even if those arguments are quoted). For example, calling the script like this: test.sh "Cherry Hill" "New York City" produces the following output:
Cherry
Hill
New
York
City
First, don't do it that way. The best approach is to use find -exec properly:
# this is safe
find test -type d -exec echo '{}' +
The other safe approach is to use NUL-terminated list, though this requires that your find support -print0:
# this is safe
while IFS= read -r -d '' n; do
printf '%q\n' "$n"
done < <(find test -mindepth 1 -type d -print0)
You can also populate an array from find, and pass that array later:
# this is safe
declare -a myarray
while IFS= read -r -d '' n; do
myarray+=( "$n" )
done < <(find test -mindepth 1 -type d -print0)
printf '%q\n' "${myarray[#]}" # printf is an example; use it however you want
If your find doesn't support -print0, your result is then unsafe -- the below will not behave as desired if files exist containing newlines in their names (which, yes, is legal):
# this is unsafe
while IFS= read -r n; do
printf '%q\n' "$n"
done < <(find test -mindepth 1 -type d)
If one isn't going to use one of the above, a third approach (less efficient in terms of both time and memory usage, as it reads the entire output of the subprocess before doing word-splitting) is to use an IFS variable which doesn't contain the space character. Turn off globbing (set -f) to prevent strings containing glob characters such as [], * or ? from being expanded:
# this is unsafe (but less unsafe than it would be without the following precautions)
(
IFS=$'\n' # split only on newlines
set -f # disable globbing
for n in $(find test -mindepth 1 -type d); do
printf '%q\n' "$n"
done
)
Finally, for the command-line parameter case, you should be using arrays if your shell supports them (i.e. it's ksh, bash or zsh):
# this is safe
for d in "$#"; do
printf '%s\n' "$d"
done
will maintain separation. Note that the quoting (and the use of $# rather than $*) is important. Arrays can be populated in other ways as well, such as glob expressions:
# this is safe
entries=( test/* )
for d in "${entries[#]}"; do
printf '%s\n' "$d"
done
find . -type d | while read file; do echo $file; done
However, doesn't work if the file-name contains newlines. The above is the only solution i know of when you actually want to have the directory name in a variable. If you just want to execute some command, use xargs.
find . -type d -print0 | xargs -0 echo 'The directory is: '
Here is a simple solution which handles tabs and/or whitespaces in the filename. If you have to deal with other strange characters in the filename like newlines, pick another answer.
The test directory
ls -F test
Baltimore/ Cherry Hill/ Edison/ New York City/ Philadelphia/ cities.txt
The code to go into the directories
find test -type d | while read f ; do
echo "$f"
done
The filename must be quoted ("$f") if used as argument. Without quotes, the spaces act as argument separator and multiple arguments are given to the invoked command.
And the output:
test/Baltimore
test/Cherry Hill
test/Edison
test/New York City
test/Philadelphia
This is exceedingly tricky in standard Unix, and most solutions run foul of newlines or some other character. However, if you are using the GNU tool set, then you can exploit the find option -print0 and use xargs with the corresponding option -0 (minus-zero). There are two characters that cannot appear in a simple filename; those are slash and NUL '\0'. Obviously, slash appears in pathnames, so the GNU solution of using a NUL '\0' to mark the end of the name is ingenious and fool-proof.
You could use IFS (internal field separator) temporally using :
OLD_IFS=$IFS # Stores Default IFS
IFS=$'\n' # Set it to line break
for f in `find test/* -type d`; do
echo $f
done
IFS=$OLD_IFS
<!>
Why not just put
IFS='\n'
in front of the for command? This changes the field separator from < Space>< Tab>< Newline> to just < Newline>
find . -print0|while read -d $'\0' file; do echo "$file"; done
I use
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for f in $( find "$1" -type d ! -path "$1" )
do
echo $f
done
IFS=$SAVEIFS
Wouldn't that be enough?
Idea taken from http://www.cyberciti.biz/tips/handling-filenames-with-spaces-in-bash.html
Don't store lists as strings; store them as arrays to avoid all this delimiter confusion. Here's an example script that'll either operate on all subdirectories of test, or the list supplied on its command line:
#!/bin/bash
if [ $# -eq 0 ]; then
# if no args supplies, build a list of subdirs of test/
dirlist=() # start with empty list
for f in test/*; do # for each item in test/ ...
if [ -d "$f" ]; then # if it's a subdir...
dirlist=("${dirlist[#]}" "$f") # add it to the list
fi
done
else
# if args were supplied, copy the list of args into dirlist
dirlist=("$#")
fi
# now loop through dirlist, operating on each one
for dir in "${dirlist[#]}"; do
printf "Directory: %s\n" "$dir"
done
Now let's try this out on a test directory with a curve or two thrown in:
$ ls -F test
Baltimore/
Cherry Hill/
Edison/
New York City/
Philadelphia/
this is a dirname with quotes, lfs, escapes: "\''?'?\e\n\d/
this is a file, not a directory
$ ./test.sh
Directory: test/Baltimore
Directory: test/Cherry Hill
Directory: test/Edison
Directory: test/New York City
Directory: test/Philadelphia
Directory: test/this is a dirname with quotes, lfs, escapes: "\''
'
\e\n\d
$ ./test.sh "Cherry Hill" "New York City"
Directory: Cherry Hill
Directory: New York City
ps if it is only about space in the input, then some double quotes worked smoothly for me...
read artist;
find "/mnt/2tb_USB_hard_disc/p_music/$artist" -type f -name *.mp3 -exec mpg123 '{}' \;
To add to what Jonathan said: use the -print0 option for find in conjunction with xargs as follows:
find test/* -type d -print0 | xargs -0 command
That will execute the command command with the proper arguments; directories with spaces in them will be properly quoted (i.e. they'll be passed in as one argument).
#!/bin/bash
dirtys=()
for folder in *
do
if [ -d "$folder" ]; then
dirtys=("${dirtys[#]}" "$folder")
fi
done
for dir in "${dirtys[#]}"
do
for file in "$dir"/\*.mov # <== *.mov
do
#dir_e=`echo "$dir" | sed 's/[[:space:]]/\\\ /g'` -- This line will replace each space into '\ '
out=`echo "$file" | sed 's/\(.*\)\/\(.*\)/\2/'` # These two line code can be written in one line using multiple sed commands.
out=`echo "$out" | sed 's/[[:space:]]/_/g'`
#echo "ffmpeg -i $out_e -sameq -vcodec msmpeg4v2 -acodec pcm_u8 $dir_e/${out/%mov/avi}"
`ffmpeg -i "$file" -sameq -vcodec msmpeg4v2 -acodec pcm_u8 "$dir"/${out/%mov/avi}`
done
done
The above code will convert .mov files to .avi. The .mov files are in different folders and
the folder names have white spaces too. My above script will convert the .mov files to .avi file in the same folder itself. I don't know whether it help you peoples.
Case:
[sony#localhost shell_tutorial]$ ls
Chapter 01 - Introduction Chapter 02 - Your First Shell Script
[sony#localhost shell_tutorial]$ cd Chapter\ 01\ -\ Introduction/
[sony#localhost Chapter 01 - Introduction]$ ls
0101 - About this Course.mov 0102 - Course Structure.mov
[sony#localhost Chapter 01 - Introduction]$ ./above_script
... successfully executed.
[sony#localhost Chapter 01 - Introduction]$ ls
0101_-_About_this_Course.avi 0102_-_Course_Structure.avi
0101 - About this Course.mov 0102 - Course Structure.mov
[sony#localhost Chapter 01 - Introduction]$ CHEERS!
Cheers!
Had to be dealing with whitespaces in pathnames, too. What I finally did was using a recursion and for item in /path/*:
function recursedir {
local item
for item in "${1%/}"/*
do
if [ -d "$item" ]
then
recursedir "$item"
else
command
fi
done
}
Convert the file list into a Bash array. This uses Matt McClure's approach for returning an array from a Bash function:
http://notes-matthewlmcclure.blogspot.com/2009/12/return-array-from-bash-function-v-2.html
The result is a way to convert any multi-line input to a Bash array.
#!/bin/bash
# This is the command where we want to convert the output to an array.
# Output is: fileSize fileNameIncludingPath
multiLineCommand="find . -mindepth 1 -printf '%s %p\\n'"
# This eval converts the multi-line output of multiLineCommand to a
# Bash array. To convert stdin, remove: < <(eval "$multiLineCommand" )
eval "declare -a myArray=`( arr=(); while read -r line; do arr[${#arr[#]}]="$line"; done; declare -p arr | sed -e 's/^declare -a arr=//' ) < <(eval "$multiLineCommand" )`"
for f in "${myArray[#]}"
do
echo "Element: $f"
done
This approach appears to work even when bad characters are present, and is a general way to convert any input to a Bash array. The disadvantage is if the input is long you could exceed Bash's command line size limits, or use up large amounts of memory.
Approaches where the loop that is eventually working on the list also have the list piped in have the disadvantage that reading stdin is not easy (such as asking the user for input), and the loop is a new process so you may be wondering why variables you set inside the loop are not available after the loop finishes.
I also dislike setting IFS, it can mess up other code.
Well, I see too many complicated answers. I don't want to pass the output of find utility or to write a loop , because find has "exec" option for this.
My problem was that I wanted to move all files with dbf extension to the current folder and some of them contained white space.
I tackled it so:
find . -name \*.dbf -print0 -exec mv '{}' . ';'
Looks much simple for me
just found out there are some similarities between my question and yours. Aparrently if you want to pass arguments into commands
test.sh "Cherry Hill" "New York City"
to print them out in order
for SOME_ARG in "$#"
do
echo "$SOME_ARG";
done;
notice the $# is surrounded by double quotes, some notes here
I needed the same concept to compress sequentially several directories or files from a certain folder. I have solved using awk to parsel the list from ls and to avoid the problem of blank space in the name.
source="/xxx/xxx"
dest="/yyy/yyy"
n_max=`ls . | wc -l`
echo "Loop over items..."
i=1
while [ $i -le $n_max ];do
item=`ls . | awk 'NR=='$i'' `
echo "File selected for compression: $item"
tar -cvzf $dest/"$item".tar.gz "$item"
i=$(( i + 1 ))
done
echo "Done!!!"
what do you think?
find Downloads -type f | while read file; do printf "%q\n" "$file"; done
For me this works, and it is pretty much "clean":
for f in "$(find ./test -type d)" ; do
echo "$f"
done
Just had a simple variant problem... Convert files of typed .flv to .mp3 (yawn).
for file in read `find . *.flv`; do ffmpeg -i ${file} -acodec copy ${file}.mp3;done
recursively find all the Macintosh user flash files and turn them into audio (copy, no transcode) ... it's like the while above, noting that read instead of just 'for file in ' will escape.