I cannot get the following piece of script (which is part of a larger backup script) to work correctly:
BACKUPDIR=/BACKUP/db01/physical/incremental # Backups base directory
FULLBACKUPDIR=$BACKUPDIR/full # Full backups directory
INCRBACKUPDIR=$BACKUPDIR/incr # Incremental backups directory
KEEP=5 # Number of full backups (and its incrementals) to keep
...
FIRST_DELETE=`expr $KEEP + 1` # add one to the number of backups to keep, this will be the first deleted
FILE0=`ls -ltr $FULLBACKUPDIR | awk '{print $9}' | tail -$FIRST_DELETE | head -1` # search for the first backup to be deleted
...
find $FULLBACKUPDIR -maxdepth 1 -type d ! -newer $FULLBACKUPDIR/$FILE0 -execdir echo "removing: "$FULLBACKUPDIR/$(basename {}) \; -execdir bash -c 'rm -rf $FULLBACKUPDIR/$(basename {})' \; -execdir echo "removing: "$INCRBACKUPDIR/$(basename {}) \; -execdir bash -c 'rm -rf $INCRBACKUPDIR/$(basename {})' \;
So the find works correctly which on its own will output something like this:
/BACKUPS/db01/physical/incremental/full/2013-08-12_17-51-28
/BACKUPS/db01/physical/incremental/full/2013-08-12_17-51-28
/BACKUPS/db01/physical/incremental/full/2013-08-12_17-25-07
What I want is the -exec to echo a line showing what is being removed and then remove the folder from both directories.
I've tried various ways to get just the basename but nothing seems to be working. I get this:
removing: /BACKUPS/mysql/physical/incremental/full/"/BACKUPS/mysql/physical/incremental/full/2013-08-12_17-51-28"
removing: /BACKUPS/mysql/physical/incremental/incr/"/BACKUPS/mysql/physical/incremental/full/2013-08-12_17-51-28"
removing: /BACKUPS/mysql/physical/incremental/full/"/BACKUPS/mysql/physical/incremental/full/2013-08-12_17-25-07"
And of course the folders arn't deleted because they don't exist, just fail silently because of the -f option. If I remove the -f I get the 'cannot be found' error on each rm.
How do I accomplish this? Because backups and parts of backups may be stored across different storage systems I really need the ability to just get the folder name for use in any known path.
the $(basename {}) is run first, making removing: "$INCRBACKUPDIR/$(basename {}) to removing: "$INCRBACKUPDIR/{} then the replacement is done of {}.
a way around it may be to pipe it to bash:
-exec echo "echo \"removing: \\\"$INCRBACKUPDIR/\$(basename {})\\\"\" | bash" \;
Lots of broken here.
All caps variables are by convention env vars and should not be used in scripts.
Using legacy backticks instead of $()
Parsing the output of ls (!)
Parsing the output of ls -l (!!!)
Expanding variables known to contain paths without full quotes.
All you absolutely need in order to improve this is to -exec bash properly, e.g.
-execdir bash -c 'filepath="$1" ; base=$(basename "$filepath") ; echo use $filepath and $base here' -- {} \;
But how about this instead:
#!/usr/bin/env bash
backup_base=/BACKUP/db01/physical/incremental
full_backup="$backup_base"/full
incremental_backup="$backup_base"/incr
keep=5
rm=echo
let n=0
while IFS= read -r -d $'\0' line ; do
file="${line#* }"
if [[ $n -lt $keep ]] ; then
let n=n+1
continue
fi
base=$(basename "$file")
echo "removing: $full_backup/$base"
"$rm" -rf -- "$full_backup"/"$base"
echo "removing: $incremental_backup/$base"
"$rm" -rf -- "$incremental_backup"/"$base"
done < <(find "$full_backup" -maxdepth 1 -printf '%T#.%p\0' 2>/dev/null | sort -z -r -n -t. -k1,2)
Iterate over files and directories immediately under the backup dir and skip the first 5 newest. Delete from the full and incremental dirs files matching the names of the rest.
This is an essentially safe version, except of course for timing attacks.
I have defined rm as being echo to avoid accidental deletes; swap it back to rm for actual deletion once you're sure it's correct.
Related
Update 01/12/2022
With triplee's helpful suggestions, I resolved it to take both files & directories by adding a comma in between f and d, the final code now looks like this:
while read -r old new;
do echo "replacing ${old} by ${new}" >&2
find '/path/to/dir' -depth -type d,f -name "$old" -exec rename
"s/${old}/${new}/" {} ';'
done <input.txt
Thank you!
Original request:
I am trying to rename a list of files (from $old to $new), all present in $homedir or in subdirectories in $homedir.
In the command line this line works to rename files in the subfolders:
find ${homedir}/ -name ${old} -exec rename "s/${old}/${new}/" */${old} ';'
However, when I want to implement this line in a simple bash script getting the $old and $new filenames from input.txt, it doesn't work anymore...
input.txt looks like this:
name_old name_new
name_old2 name_new2
etc...
the script looks like this:
#!/bin/bash
homedir='/path/to/dir'
cat input.txt | while read old new;
do
echo 'replacing' ${old} 'by' ${new}
find ${homedir}/ -name ${old} -exec rename "s/${old}/${new}/" */${old} ';'
done
After running the script, the text line from echo with $old and $new filenames being replaced is printed for the entire loop, but no files are renamed. No error is printed either. What am I missing? Your help would be greatly appreaciated!
I checked whether the $old and $new variables were correctly passed to the find -exec rename command, but because they are printed by echo that doesn't seem to be the issue.
If you add an echo, like -exec echo rename ..., you'll see what actually gets executed. I'd say that both the path to $old is wrong (you're not using the result of find in the -exec clause), and */$old isn't quoted and might be expanded by the shell before find ever gets to see it.
You're also having most other expansions unquoted, which can lead to all sorts of trouble.
You could do it in pure Bash (drop echo when output looks good):
shopt -s globstar
for f in **/"$old"; do echo mv "$f" "${f/%*/$new}"; done
Or with rename directly, though this would run into trouble if too many files match (drop -n when output looks good):
rename -n "s/$old\$/$new/" **/"$old"
Or with GNU find, using -execdir to run in the same directory as the matching file (drop echo when output looks good):
find -type f -name "$old" -execdir echo mv "$old" "$new" \;
And finally, a version with find that spawns just a single subshell (drop echo when output looks right):
find -type f -name "$old" -exec bash -c '
new=$1
shift
for f; do
echo mv "$f" "${f/%*/$new}"
done
' bash "$new" {} +
The argument to rename should be the file itself, not */${old}. You also have a number of quoting errors, and a useless cat).
#!/bin/bash
while read -r old new;
do
echo "replacing ${old} by ${new}" >&2
find /path/to/dir -name "$old" -exec rename "s/${old}/${new}/" {} ';'
done <input.txt
Running find multiple times on the same directory is hugely inefficient, though. Probably a better solution is to find all files in one go, and abort if it's not one of the files on the list.
find /path/to/dir -type f -exec sh -c '
for f in "$#"; do
awk -v f="$f" "f==\$1 { print \"s/\" \$1 \"/\" \$2 \"/\" }" "$0" |
xargs -I _ -r rename _ "$f"
done' input.txt {} +
(Untested; probably try with echo before you run this live.)
I'm trying to create a script that will remove a list of applications in OS X.
My general thinking:
Applications are just directories in OS X
Those directories have a Contents sub-directory that contains an Info.plist that can be used to identify the app.
So the core logic would be
Walk the drive
IF the current folder has a subfolder named Contents that contains a file called Info.plist that contains certain text, delete the current folder.
I've been playing around with find -exec, but am open to other approaches.
This is what I have for find -exec that isn't quite working
find /Applications -maxdepth 3 -type f -name Info.plist -exec sh -c "grep '<string>Example</string>' | xargs dirname | xargs echo rm --" 2>/dev/null
I realize
defaults read $dir/Contents/Info CFBundleExecutable
is probably better than grep to extract the package name, but don't think that is why the above isn't working (no output from the test "echo rm" at all, and inserting a tee command to output to a file didn't do anything either).
The above line, if it worked, would go in a loop running for each app to be removed with a list of app names in a variable.
I am totally open to other approaches, especially if there's a more efficient way to do this.
It's saner to keep the logic in your parent shell, rather than shuffling it off to a subprocess:
#!/usr/bin/env bash
# works correctly with names echo doesn't handle -- ones containing spaces, backslashes, etc
log_command() { printf '%q ' "$#" && echo; }
while IFS= read -r -d '' plist; do
if grep -e '<string>Example</string>' "$plist"; then
log_command rm -r -- "${plist%/*}"
fi
done < <(find /Applications -maxdepth 3 -type f -name Info.plist -print0)
Note that <(...) -- process substitution -- is a bash-only feature, so it isn't guaranteed to work if your script is started with sh instead of bash.
This also can be easily modified to use better practices, so it's easy to modify to:
#!/usr/bin/env bash
log_command() { printf '%q ' "$#" && echo; }
while IFS= read -r -d '' plist; do
dir=${plist%/*}
if [[ "$(defaults read "$dir"/Contents/Info CFBundleExecutable)" = example ]]; then
log_command rm -r -- "$dir"
fi
done < <(find /Applications -maxdepth 3 -type f -name Info.plist -print0)
However, if you really want find to be the parent process, you can do that:
find /Applications -maxdepth 3 -type f -name Info.plist -exec bash -c '
log_command() { printf '%s\n' "$#" && echo; }
for plist do
if grep -e "<string>Example</string>" "$plist"; then
log_command rm -r -- "${plist%/*}"
fi
done
' _ {} +
The use of -exec ... {} + passes as many results as possible to each copy of bash. The _ fills in $0, so the filenames that were found and placed on the command line are put in $1, $2, etc; this is what for loops over when not given an explicit list.
Since you're limiting it to a depth of 3, and that's also practically the minimum depth, and the rest of the path is going to be highly constrained (it must be *.app/Contents/) you don't really need find. A simple glob pattern should suffice, and should also be more efficient (since it doesn't have to search e.g. Contents/Resources):
for plist in /Applications/*.app/Contents/Info.plist; do
if [ "$(defaults read "${plist%.plist}" CFBundleExecutable)" = Example ]; then
echo rm -R -- "${plist%/Contents/Info.plist}" # remove "echo" to actually do it
fi
done
I have a lot of files that are named as: MM-DD-YYYY.pdf. I want to rename them as YYYY-MM-DD.pdf I’m sure there is some bash magic to do this. What is it?
For files in the current directory:
for name in ./??-??-????.pdf; do
if [[ "$name" =~ (.*)/([0-9]{2})-([0-9]{2})-([0-9]{4})\.pdf ]]; then
echo mv "$name" "${BASH_REMATCH[1]}/${BASH_REMATCH[4]}-${BASH_REMATCH[3]}-${BASH_REMATCH[2]}.pdf"
fi
done
Recursively, in or under the current directory:
find . -type f -name '??-??-????.pdf' -exec bash -c '
for name do
if [[ "$name" =~ (.*)/([0-9]{2})-([0-9]{2})-([0-9]{4})\.pdf ]]; then
echo mv "$name" "${BASH_REMATCH[1]}/${BASH_REMATCH[4]}-${BASH_REMATCH[3]}-${BASH_REMATCH[2]}.pdf"
fi
done' bash {} +
Enabling the globstar shell option in bash lets us do the following (will also, like the above solution, handle all files in or below the current directory):
shopt -s globstar
for name in **/??-??-????.pdf; do
if [[ "$name" =~ (.*)/([0-9]{2})-([0-9]{2})-([0-9]{4})\.pdf ]]; then
echo mv "$name" "${BASH_REMATCH[1]}/${BASH_REMATCH[4]}-${BASH_REMATCH[3]}-${BASH_REMATCH[2]}.pdf"
fi
done
All three of these solutions uses a regular expression to pick out the relevant parts of the filenames, and then rearranges these parts into the new name. The only difference between them is how the list of pathnames is generated.
The code prefixes mv with echo for safety. To actually rename files, remove the echo (but run at least once with echo to see that it does what you want).
A direct approach example from the command line:
$ ls
10-01-2018.pdf 11-01-2018.pdf 12-01-2018.pdf
$ ls [0-9]*-[0-9]*-[0-9]*.pdf|sed -r 'p;s/([0-9]{2})-([0-9]{2})-([0-9]{4})/\3-\1-\2/'|xargs -n2 mv
$ ls
2018-10-01.pdf 2018-11-01.pdf 2018-12-01.pdf
The ls output is piped to sed , then we use the p flag to print the argument without modifications, in other words, the original name of the file, and s to perform and output the conversion.
The ls + sed result is a combined output that consist of a sequence of old_file_name and new_file_name.
Finally we pipe the resulting feed through xargs to get the effective rename of the files.
From xargs man:
-n number Execute command using as many standard input arguments as possible, up to number arguments maximum.
You can use the following command very close to the one of klashxx:
for f in *.pdf; do echo "$f"; mv "$f" "$(echo "$f" | sed 's#\(..\)-\(..\)-\(....\)#\3-\2-\1#')"; done
before:
ls *.pdf
12-01-1998.pdf 12-03-2018.pdf
after:
ls *.pdf
1998-01-12.pdf 2018-03-12.pdf
Also if you have other pdf files that does not respect this format in your folder, what you can do is to select only the files that respect the format: MM-DD-YYYY.pdf to do so use the following command:
for f in `find . -maxdepth 1 -type f -regextype sed -regex './[0-9]\{2\}-[0-9]\{2\}-[0-9]\{4\}.pdf' | xargs -n1 basename`; do echo "$f"; mv "$f" "$(echo "$f" | sed 's#\(..\)-\(..\)-\(....\)#\3-\2-\1#')"; done
Explanations:
find . -maxdepth 1 -type f -regextype sed -regex './[0-9]\{2\}-[0-9]\{2\}-[0-9]\{4\}.pdf this find command will look only for files in the current working directory that respect your syntax and extract their basename (remove the ./ at the beginning, folders and other type of files that would have the same name are not taken into account, other *.pdf files are also ignored.
for each file you do a move and the resulting file name is computed using sed and back reference to the 3 groups for MM,DD and YYYY
For these simple filenames, using a more verbose pattern, you can simplify the body of the loop a bit:
twodigit=[[:digit:]][[:digit:]]
fourdigit="$twodigit$twodigit"
for f in $twodigit-$twodigit-$fourdigit.pdf; do
IFS=- read month day year <<< "${f%.pdf}"
mv "$f" "$year-$month-$day.pdf"
done
This is basically #Kusalananda's answer, but without the verbosity of regular-expression matching.
I have a shell script which basically searches all folders inside a location and I use grep to find the exact folder I want to target.
for dir in /root/*; do
grep "Apples" "${dir}"/*.* || continue
While grep successfully finds my target directory, I'm stuck on how I can move the folders I want to move in my target directory. An idea I had was to cd into grep output but that's where I got stuck. Tried some Google results, none helped with my case.
Example grep output: Binary file /root/ant/containers/secret/Documents/2FD412E0/file.extension matches
I want to cd into 2FD412E0and move two folders inside that directory.
dirname is the key to that:
cd $(dirname $(grep "...." ...))
will let you enter the directory.
As people mentioned, dirname is the right tool to strip off the file name from the path.
I would use find for such kind of task:
while read -r file
do
target_dir=`dirname $file`
# do something with "$target_dir"
done < <(find /root/ -type f \
-exec grep "Apples" --files-with-matches {} \;)
Consider using find's -maxdepth option. See the man page for find.
Well, there is actually simpler solution :) I just like to write bash scripts. You might simply use single find command like this:
find /root/ -type f -exec grep Apples {} ';' -exec ls -l {} ';'
Note the second -exec. It will be executed, if the previous -exec command exited with status 0 (success). From the man page:
-exec command ;
Execute command; true if 0 status is returned. All following arguments to find are taken to be arguments to the command until an argument consisting of ; is encountered. The string {} is replaced by the current file name being processed everywhere it occurs in the arguments to the command, not just in arguments where it is alone, as in some versions of find.
Replace the ls -l command with your stuff.
And if you want to execute dirname within the -exec command, you may do the following trick:
find /root/ -type f -exec grep -q Apples {} ';' \
-exec sh -c 'cd `dirname $0`; pwd' {} ';'
Replace pwd with your stuff.
When find is not available
In the comments you write that find is not available on your system. The following solution works without find:
grep -R --files-with-matches Apples "${dir}" | while read -r file
do
target_dir=`dirname $file`
# do something with "$target_dir"
echo $target_dir
done
I know there are a lot of things like this around, but either they don't work recursively or they are huge.
This is what I got:
find . -name "*.so" -exec mv {} `echo {} | sed s/.so/.dylib/` \;
When I just run the find part it gives me a list of files. When I run the sed part it replaces any .so with .dylib. When I run them together they don't work.
I replaced mv with echo to see what happened:
./AI/Interfaces/C/0.1/libAIInterface.so ./AI/Interfaces/C/0.1/libAIInterface.so
Nothing is replaced at all!
What is wrong?
This will do everything correctly:
find -L . -type f -name "*.so" -print0 | while IFS= read -r -d '' FNAME; do
mv -- "$FNAME" "${FNAME%.so}.dylib"
done
By correctly, we mean:
1) It will rename just the file extension (due to use of ${FNAME%.so}.dylib). All the other solutions using ${X/.so/.dylib} are incorrect as they wrongly rename the first occurrence of .so in the filename (e.g. x.so.so is renamed to x.dylib.so, or worse, ./libraries/libTemp.so-1.9.3/libTemp.so is renamed to ./libraries/libTemp.dylib-1.9.3/libTemp.so - an error).
2) It will handle spaces and any other special characters in filenames (except double quotes).
3) It will not change directories or other special files.
4) It will follow symbolic links into subdirectories and links to target files and rename the target file, not the link itself (the default behaviour of find is to process the symbolic link itself, not the file pointed to by the link).
for X in `find . -name "*.so"`
do
mv $X ${X/.so/.dylib}
done
A bash script to rename file extensions generally
#/bin/bash
find -L . -type f -name '*.'$1 -print0 | while IFS= read -r -d '' file; do
echo "renaming $file to $(basename ${file%.$1}.$2)";
mv -- "$file" "${file%.$1}.$2";
done
Credits to aps2012.
Usage
Create a file e.g. called ext-rename (no extension, so you can run it like a command) in e.g. /usr/bin (make sure /usr/bin is added to your $PATH)
run ext-rename [ext1] [ext2] anywhere in terminal, where [ext1] is renaming from and [ext2] is renaming to. An example use would be: ext-rename so dylib, which will rename any file with extension .so to same name but with extension .dylib.
What is wrong is that
echo {} | sed s/.so/.dylib/
is only executed once, before the find is launched, sed is given {} on its input, which doesn't match /.so/ and is left unchanged, so your resulting command line is
find . -name "*.so" -exec mv {} {}
if you have Bash 4
#!/bin/bash
shopt -s globstar
shopt -s nullglob
for file in /path/**/*.so
do
echo mv "$file" "${file/%.so}.dylib"
done
He needs recursion:
#!/bin/bash
function walk_tree {
local directory="$1"
local i
for i in "$directory"/*;
do
if [ "$i" = . -o "$i" = .. ]; then
continue
elif [ -d "$i" ]; then
walk_tree "$i"
elif [ "${i##*.}" = "so" ]; then
echo mv $i ${i%.*}.dylib
else
continue
fi
done
}
walk_tree "."