I have defined a function in bash, which checks if two files exists, compare if they are equal and delete one of them.
function remodup {
F=$1
G=${F/.mod/}
if [ -f "$F" ] && [ -f "$G" ]
then
cmp --silent "$F" "$G" && rm "$F" || echo "$G was modified"
fi
}
Then I want to call this function from a find command:
find $DIR -name "*.mod" -type f -exec remodup {} \;
I have also tried | xargs syntax. Both find and xargs tell that ``remodup` does not exist.
I can move the function into a separate bash script and call the script, but I don't want to copy that function into a path directory (yet), so I would either need to call the function script with an absolute path or allways call the calling script from the same location.
(I probably can use fdupes for this particular task, but I would like to find a way to either
call a function from find command;
call one script from a relative path of another script; or
Use a ${F/.mod/} syntax (or other bash variable manipulation) for files found with a find command.)
You need to export the function first using:
export -f remodup
then use it as:
find $DIR -name "*.mod" -type f -exec bash -c 'remodup "$1"' - {} \;
You could manually loop over find's results.
while IFS= read -rd $'\0' file; do
remodup "$file"
done < <(find "$dir" -name "*.mod" -type f -print0)
-print0 and -d $'\0' use NUL as the delimiter, allowing for newlines in the file names. IFS= ensures spaces as the beginning of file names aren't stripped. -r disables backslash escapes. The sum total of all of these options is to allow as many special characters as possible in file names without mangling.
Given that you aren't using many features of find, you can use a pure bash solution instead to iterate over the desired files.
shopt -s globstar nullglob
for fname in ./"$DIR"/**/*.mod; do
[[ -f $fname ]] || continue
f=${fname##*/}
remodup "$f"
done
To throw in a third option:
find "$dir" -name "*.mod" -type f \
-exec bash -s -c "$(declare -f remodup)"$'\n'' for arg; do remodup "$arg"; done' _ {} +
This passes the function through the argv, as opposed to through the environment, and (by virtue of using {} + rather than {} ;) uses as few shell instances as possible.
I would use John Kugelman's answer as my first choice, and this as my second.
Related
In Bash, I wish to rename many files based upon a predefined dict with many replacement strings. I have multiple files nested in a directory tree, like:
./aa
./b/aa
./b/bb
./c/aa
./c/d/ee
I have a "sed script" dict.sed whose contents is like:
s|aa|xx|g
s|ee|yy|g
Can I recursively find and rename files matching aa and ee to xx and yy, respectively, and preserving the directory structure, using said sed script?
At the moment I have:
function rename_from_sed() {
IFS='|' read -ra SEDCMD <<< "$1"
find "." -type f -name "${SEDCMD[1]}" -execdir mv {} "${SEDCMD[2]}" ';'
}
while IFS='' read -r line || [[ -n "${line}" ]]; do
rename_from_sed "$line"
done < "dict.sed"
This seems to work, but is there a better way, perhaps using sed -f dict.sed instead of parsing dict.sed line-by-line? The current approach means I need to specify the delimiter, and is also limited to exact filenames.
The closest other answer is probably https://stackoverflow.com/a/11709995/3329384 but that also has the limitation that directories may also be renamed.
That seems odd: iterating over the lines in a sed script. You want:
Can I recursively find and rename files
So iterate over files, not over the sed script.
find .... | while IFS= read -r file; do
newfile=$(sed -f dict.sed <<<"$file")
mv "$file" "$newfile"
done
Related https://mywiki.wooledge.org/BashFAQ/001 and https://mywiki.wooledge.org/BashFAQ/020
but is there a better way
Sure - do not use Bash and do use Sed. Write the whole script in Python or Perl - it will be 1000 times faster to use a tool which will do all the operations in a single thread without any fork() calls.
Anyway, you can also parse sed script line by line and run a single find call.
find "." -type f '(' -name aa -execdir mv {} xx ';' ')' -o '(' -name ee -execdir mv {} yy ';' ')'
You would have to build such call, like:
findargs=()
while IFS='|' read -r _ f t _; do
if ((${#findargs[#]})); then findargs+=('-o'); fi
findargs+=( '(' -name "$f" -execdir {} "$t" ';' ')' )
done < "dict.sed"
find . -type f "${findargs[#]}"
Assuming lines in the file dict is in the form of from|to, below is an implementation in pure bash, without using find and sed.
#!/bin/bash
shopt -s globstar nullglob
while IFS='|' read -r from to; do
for path in ./**/*"$from"*; do
[[ -f $path ]] || continue
basename=${path##*/}
echo mv "$path" "${path%/*}/${basename//"$from"/"$to"}"
done
done < dict
--
$ cat dict
aa|xx
ee|yy
Drop the echo if you are satisfied with the output.
for subj in `cat dti_list.txt`; do
echo $subj
find . -type f -iname '*306.nii' -execdir bash -c 'rename.ul "$subj" DTI_MAIN_AP.nii *.nii' \+
done
I have some trouble with a small bash script, which adds the name instead of replacing when I use the rename.ul function.
Currently, the code adds DTI_MAIN_AP.nii in front of the old name.
My goal is to replace the name from the subj list and using the find to search up any directory with a *306.nii file, and then using execdir to execute the rename.ul function to rename the file from the dti_list.txt.
Any solution, or correction to get the code working, will be appreciated.
If you just want to rename the first file matching *306.nii in each directory to DTI_MAIN_AP.nii, that might look like:
find . -type f -iname '*306.nii' \
-execdir sh -c '[[ -e DTI_MAIN_AP.nii ]] || mv "$1" DTI_MAIN_AP.nii' _ {} +
If instead of matching on *306.nii you want to iterate over names from dti_list.txt, that might instead look like:
while IFS= read -r -d '' filename <&3; do
find . -type f -name "$filename" \
-execdir sh -c '[[ -e DTI_MAIN_AP.nii ]] || mv "$1" DTI_MAIN_AP.nii' _ {} +
done <dti_list.txt
References of note:
BashFAQ #1 (on reading files line-by-line)
Using Find
I have a script that looks like this:
function main() {
for source in "$#"; do
sort_imports "${source}"
done
}
main "$#"
Right now if I pass in a file ./myFile.m the script works as expected.
I want to change it to passing in ./myClassPackage and have it find all files and call sort_imports on each of them.
I tried:
for source in $(find "$#"); do
sort_imports "${source}"
done
but when I call it I get an error that I'm passing it a directory.
Using the output of a command substitution for a for loop has pitfalls due to word splitting. A truly rock-solid solution will use null-byte delimiters to properly handle even files with newlines in their names (which is not common, but valid).
Assuming you only want regular files (and not directories), try this :
while IFS= read -r -d '' source; do
sort_imports "$source"
done < <(find "$#" -type f -print0)
The -print0 option causes find to separate entries with null bytes, and the -d '' option for read allows these to be used as record separators.
You should use find with -exec:
find "$#" -type f -exec sort_imports "{}" \;
For more information see https://www.everythingcli.org/find-exec-vs-find-xargs/
If you don't want find to enumerate directories, then exclude them:
for source in $(find "$#" -not -type d); do
sort_imports "${source}"
done
A bunch of Word & Excel documents were being moved on the server when the process terminated before it was complete. As a result, we're left with several perfectly fine files that have a .tmp extension, and we need to rename these files back to the appropriate .xlsx or .docx extension.
Here's my current code to do this in Bash:
#!/bin/sh
for i in "$(find . -type f -name *.tmp)"; do
ft="$(file "$i")"
case "$(file "$i")" in
"$i: Microsoft Word 2007+")
mv "$i" "${i%.tmp}.docx"
;;
"$i: Microsoft Excel 2007+")
mv "$i" "${i%.tmp}.xlsx"
;;
esac
done
It seems that while this does search recursively, it only does 1 file. If it finds an initial match, it doesn't go on to rename the rest of the files. How can I get this to loop correctly through the directories recursively without it doing just 1 file at a time?
Try find command like this:
while IFS= read -r -d '' i; do
ft="$(file "$i")"
case "$ft" in
"$i: Microsoft Word 2007+")
mv "$i" "${i%.tmp}.docx"
;;
"$i: Microsoft Excel 2007+")
mv "$i" "${i%.tmp}.xlsx"
;;
esac
done < <(find . -type f -name '*.tmp' -print0)
Using <(...) is called process substitution to run find command here
Quote filename pattern in find
Use -print0 to get find output delimited by a null character to allow space/newline characters in file names
Use IFS= and -d '' to read null separated filenames
I too would recommend using find. I would do this in two passes of find:
find . -type f -name \*.tmp \
-exec sh -c 'file "{}" | grep -q "Microsoft Word 2007"' \; \
-exec sh -c 'f="{}"; echo mv "$f" "${f%.tmp}.docx"' \;
find . -type f -name \*.tmp \
-exec sh -c 'file "{}" | grep -q "Microsoft Excel 2007"' \; \
-exec sh -c 'f="{}"; echo mv "$f" "${f%.tmp}.xlsx"' \;
Lines are split for readability.
Each instance of find will search for tmp files, then use -exec to test the output of find. This is similar to how you're doing it within the while loop in your shell script, only it's launched from within find itself. We're using the pipe to grep instead of your case statement.
The second -exec only gets run if the first one returned "true" (i.e. grep -q ... found something), and executes the rename in a tiny shell instance.
I haven't profiled this to see whether it would be faster or slower than a loop in a shell script. Just another way to handle things.
I have a directory that may contain multiple file types. I'm trying to use conditional expressions to perform actions only if certain file types exist in the directory.
For example, if xml files exist then move them elsewhere. The following statement is not working and I don't know why.
[ -e *.xml ] && mv *.xml ../xml
I've also tried double brackets without success. Any suggestions/explanations?
Thanks.
You can do that with find, no need to check for their existence. if present they will be moved else command will end without any error:
find . -name "*.xml" -type f -exec mv {} /path/to/dest/ \;
The -e operator expects a single file, but *.xml could expand to more than one. There isn't really a good way to handle this solely in shell; I would use find:
find -name '*.xml' -execdir mv '{}' ../xml +
One shell-only solution is to abuse a for loop:
for f in *.xml; do
[[ -f "$f" ]] && mv *.xml ../xml
break
done
This doesn't iterate, but it does allow f to be set to either the first of one or more matches, or to the literal string *.xml. Either way, the mv command is executed only if at least one file matches the pattern, then we exit the loop. A little clean is to use the nullglob option to never enter the loop if the pattern doesn't match:
shopt -s nullglob
for f in *.xml; do
mv *.xml ../xml
break
done
You might also use an array:
files=( *.xml )
[[ -f "${files[0]}" ]] && mv *.xml ../xml
or the positional parameters
set -- *.xml
[[ -f $1 ]] && mv *.xml