How do I copy directory structure containing placeholders - bash

I have the situation, where a template directory - containing files and links (!) - needs to be copied recursively to a destination directory, preserving all attributes. The template directory contains any number of placeholders (__NOTATION__), that need to be renamed to certain values.
For example template looks like this:
./template/__PLACEHOLDER__/name/__PLACEHOLDER__/prog/prefix___FILENAME___blah.txt
Destination becomes like this:
./destination/project1/name/project1/prog/prefix_customer_blah.txt
What I tried so far is this:
# first create dest directory structure
while read line; do
dest="$(echo "$line" | sed -e 's#__PLACEHOLDER__#project1#g' -e 's#__FILENAME__#customer#g' -e 's#template#destination#')"
if ! [ -d "$dest" ]; then
mkdir -p "$dest"
fi
done < <(find ./template -type d)
# now copy files
while read line; do
dest="$(echo "$line" | sed -e 's#__PLACEHOLDER__#project1#g' -e 's#__FILENAME__#customer#g' -e 's#template#destination#')"
cp -a "$line" "$dest"
done < <(find ./template -type f)
However, I realized that if I want to take care about permissions and links, this is going to be endless and very complicated. Is there a better way to replace __PLACEHOLDER__ with "value", maybe using cp, find or rsync?

I suspect that your script will already do what you want, if only you replace
find ./template -type f
with
find ./template ! -type d
Otherwise, the obvious solution is to use cp -a to make an "archive" copy of the template, complete with all links, permissions, etc, and then rename the placeholders in the copy.
cp -a ./template ./destination
while read path; do
dir=`dirname "$path"`
file=`basename "$path"`
mv -v "$path" "$dir/${file//__PLACEHOLDER__/project1}"
done < <(`find ./destination -depth -name '*__PLACEHOLDER__*'`)
Note that you'll want to use -depth or else renaming files inside renamed directories will break.
If it's very important to you that the directory tree is created with the names already changed (i.e. you must never see placeholders in the destination), then I'd recommend simply using an intermediate location.

First copy with rsync, preserving all the properties and links etc.
Then change the placeholder strings in the destination filenames:
#!/bin/bash
TEMPL="$PWD/template" # somewhere else
DEST="$PWD/dest" # wherever it is
mkdir "$DEST"
(cd "$TEMPL"; rsync -Hra . "$DEST") #
MyRen=$(mktemp)
trap "rm -f $MyRen" 0 1 2 3 13 15
cat >$MyRen <<'EOF'
#!/bin/bash
fn="$1"
newfn="$(echo "$fn" | sed -e 's#__PLACEHOLDER__#project1#g' -e s#__FILENAME__#customer#g' -e 's#template#destination#')"
test "$fn" != "$newfn" && mv "$fn" "$newfn"
EOF
chmod +x $MyRen
find "$DEST" -depth -execdir $MyRen {} \;

Related

Move files with whitespaces in their names to folders named like the files' modification dates

I created a bunch of folders from modification dates, like these:
..
2012-11-29
2012-11-20
..
Now I want to move files into these folders, in case the have a modification date that equals the folders name. The files contain whitespace in their names.
If I run this I get a list that looks like the folder names:
find . -iname "*.pdf" -print0 | while read -d $'\0' file; do stat -c "%.10y" "$file"; done
How do I take this output and use it in a script that moves these files like (pseudocode):
find . -iname "*.pdf" -print0 | while read -d $'\0' file; do mv $<FILEWITHWHITESPACEINNAME> <FOLDERNAMEDLIKE $file stat -c "%.10y" "$file" > ; done
find . -iname "*.pdf" -print0 |
while IFS= read -r -d '' file; do
folder=$(stat -c "%.10y" -- "$file") || continue # store date in variable
[[ $file = ./$folder/$file ]] && continue # skip if already there
mkdir -p -- "$folder" || continue # ensure directory exists
mv -- "$file" "$folder/" # actually do the move
done
To read a NUL-delimited name correctly, use IFS= to avoid losing leading or trailing spaces, and -r to avoid losing backslashes in the filename. See BashFAQ #1.
Inside your shell loop, you have shell variables -- so you can use one of those to store the output of your stat command. See How do I set a variable to the output of a command in bash?
Using -- signifies end-of-arguments, so even if your find were replaced with something that could emit a path starting with a - instead of a ./, we wouldn't try to treat values in file as anything but positional arguments (filenames, in the context of stat, mkdir and mv).
Using || continue inside the loop means that if any step fails, we don't do the subsequent ones -- so we don't try to do the mkdir if the stat fails, and we don't try to do the mv if the mkdir fails.

Copy files of specific type but prevent overwriting files

I need to copy all files of a specific type to a folder, which I'm doing like this:
find ./ -name '*.gql' -exec cp -prv '{}' '/path/to/dir/' ';'
But if there are two files with a identical name, although located in different subfolders, some files would be overwritten.
Is it possible to keep all files, which are copied? Maybe renaming the copied file or is it possible to keep the folder structure in the target directory?
c.f. https://ss64.com/osx/cp.html : Historic versions of the cp utility had a -r option...its use is strongly discouraged, as it does not correctly copy special files, symbolic links, or fifo's. You can use -n to prevent overwrites, but more complex logic will likely require custom code.
dest=/path/to/dir
while IFS= read -r -d '' filename # read null-delimited list from find
do basename="${filename##*/}"
if [[ -e "$dest/$basename" ]]
then ctr=0
while [[ -e "$dest/$basename.$((++ctr))" ]]; do : ; done # find available name
mv "$dest/$basename" "$dest/$basename.$ctr"
fi
cp -pv "$filename" "$dest/$basename"
done < <( find ./ -name '*.gql' -type f -print0 ) # null-delimit

Remove OS X Apps via Shell Script

I'm trying to create a script that will remove a list of applications in OS X.
My general thinking:
Applications are just directories in OS X
Those directories have a Contents sub-directory that contains an Info.plist that can be used to identify the app.
So the core logic would be
Walk the drive
IF the current folder has a subfolder named Contents that contains a file called Info.plist that contains certain text, delete the current folder.
I've been playing around with find -exec, but am open to other approaches.
This is what I have for find -exec that isn't quite working
find /Applications -maxdepth 3 -type f -name Info.plist -exec sh -c "grep '<string>Example</string>' | xargs dirname | xargs echo rm --" 2>/dev/null
I realize
defaults read $dir/Contents/Info CFBundleExecutable
is probably better than grep to extract the package name, but don't think that is why the above isn't working (no output from the test "echo rm" at all, and inserting a tee command to output to a file didn't do anything either).
The above line, if it worked, would go in a loop running for each app to be removed with a list of app names in a variable.
I am totally open to other approaches, especially if there's a more efficient way to do this.
It's saner to keep the logic in your parent shell, rather than shuffling it off to a subprocess:
#!/usr/bin/env bash
# works correctly with names echo doesn't handle -- ones containing spaces, backslashes, etc
log_command() { printf '%q ' "$#" && echo; }
while IFS= read -r -d '' plist; do
if grep -e '<string>Example</string>' "$plist"; then
log_command rm -r -- "${plist%/*}"
fi
done < <(find /Applications -maxdepth 3 -type f -name Info.plist -print0)
Note that <(...) -- process substitution -- is a bash-only feature, so it isn't guaranteed to work if your script is started with sh instead of bash.
This also can be easily modified to use better practices, so it's easy to modify to:
#!/usr/bin/env bash
log_command() { printf '%q ' "$#" && echo; }
while IFS= read -r -d '' plist; do
dir=${plist%/*}
if [[ "$(defaults read "$dir"/Contents/Info CFBundleExecutable)" = example ]]; then
log_command rm -r -- "$dir"
fi
done < <(find /Applications -maxdepth 3 -type f -name Info.plist -print0)
However, if you really want find to be the parent process, you can do that:
find /Applications -maxdepth 3 -type f -name Info.plist -exec bash -c '
log_command() { printf '%s\n' "$#" && echo; }
for plist do
if grep -e "<string>Example</string>" "$plist"; then
log_command rm -r -- "${plist%/*}"
fi
done
' _ {} +
The use of -exec ... {} + passes as many results as possible to each copy of bash. The _ fills in $0, so the filenames that were found and placed on the command line are put in $1, $2, etc; this is what for loops over when not given an explicit list.
Since you're limiting it to a depth of 3, and that's also practically the minimum depth, and the rest of the path is going to be highly constrained (it must be *.app/Contents/) you don't really need find. A simple glob pattern should suffice, and should also be more efficient (since it doesn't have to search e.g. Contents/Resources):
for plist in /Applications/*.app/Contents/Info.plist; do
if [ "$(defaults read "${plist%.plist}" CFBundleExecutable)" = Example ]; then
echo rm -R -- "${plist%/Contents/Info.plist}" # remove "echo" to actually do it
fi
done

Shell Script: How to copy files with specific string from big corpus

I have a small bug and don't know how to solve it. I want to copy files from a big folder with many files, where the files contain a specific string. For this I use grep, ack or (in this example) ag. When I'm inside the folder it matches without problem, but when I want to do it with a loop over the files in the following script it doesn't loop over the matches. Here my script:
ag -l "${SEARCH_QUERY}" "${INPUT_DIR}" | while read -d $'\0' file; do
echo "$file"
cp "${file}" "${OUTPUT_DIR}/${file}"
done
SEARCH_QUERY holds the String I want to find inside the files, INPUT_DIR is the folder where the files are located, OUTPUT_DIR is the folder where the found files should be copied to. Is there something wrong with the while do?
EDIT:
Thanks for the suggestions! I took this one now, because it also looks for files in subfolders and saves a list with all the files.
ag -l "${SEARCH_QUERY}" "${INPUT_DIR}" > "output_list.txt"
while read file
do
echo "${file##*/}"
cp "${file}" "${OUTPUT_DIR}/${file##*/}"
done < "output_list.txt"
Better implement it like below with a find command:
find "${INPUT_DIR}" -name "*.*" | xargs grep -l "${SEARCH_QUERY}" > /tmp/file_list.txt
while read file
do
echo "$file"
cp "${file}" "${OUTPUT_DIR}/${file}"
done < /tmp/file_list.txt
rm /tmp/file_list.txt
or another option:
grep -l "${SEARCH_QUERY}" "${INPUT_DIR}/*.*" > /tmp/file_list.txt
while read file
do
echo "$file"
cp "${file}" "${OUTPUT_DIR}/${file}"
done < /tmp/file_list.txt
rm /tmp/file_list.txt
if you do not mind doing it in just one line, then
grep -lr 'ONE\|TWO\|THREE' | xargs -I xxx -P 0 cp xxx dist/
guide:
-l just print file name and nothing else
-r search recursively the CWD and all sub-directories
match these works alternatively: 'ONE' or 'TWO' or 'THREE'
| pipe the output of grep to xargs
-I xxx name of the files is saved in xxx it is just an alias
-P 0 run all the command (= cp) in parallel (= as fast as possible)
cp each file xxx to the dist directory
If i understand the behavior of ag correctly, then you have to
adjust the read delimiter to '\n' or
use ag -0 -l to force delimiting by '\0'
to solve the problem in your loop.
Alternatively, you can use the following script, that is based on find instead of ag.
while read file; do
echo "$file"
cp "$file" "$OUTPUT_DIR/$file"
done < <(find "$INPUT_DIR" -name "*$SEARCH_QUERY*" -print)

Bash rename extension recursive

I know there are a lot of things like this around, but either they don't work recursively or they are huge.
This is what I got:
find . -name "*.so" -exec mv {} `echo {} | sed s/.so/.dylib/` \;
When I just run the find part it gives me a list of files. When I run the sed part it replaces any .so with .dylib. When I run them together they don't work.
I replaced mv with echo to see what happened:
./AI/Interfaces/C/0.1/libAIInterface.so ./AI/Interfaces/C/0.1/libAIInterface.so
Nothing is replaced at all!
What is wrong?
This will do everything correctly:
find -L . -type f -name "*.so" -print0 | while IFS= read -r -d '' FNAME; do
mv -- "$FNAME" "${FNAME%.so}.dylib"
done
By correctly, we mean:
1) It will rename just the file extension (due to use of ${FNAME%.so}.dylib). All the other solutions using ${X/.so/.dylib} are incorrect as they wrongly rename the first occurrence of .so in the filename (e.g. x.so.so is renamed to x.dylib.so, or worse, ./libraries/libTemp.so-1.9.3/libTemp.so is renamed to ./libraries/libTemp.dylib-1.9.3/libTemp.so - an error).
2) It will handle spaces and any other special characters in filenames (except double quotes).
3) It will not change directories or other special files.
4) It will follow symbolic links into subdirectories and links to target files and rename the target file, not the link itself (the default behaviour of find is to process the symbolic link itself, not the file pointed to by the link).
for X in `find . -name "*.so"`
do
mv $X ${X/.so/.dylib}
done
A bash script to rename file extensions generally
#/bin/bash
find -L . -type f -name '*.'$1 -print0 | while IFS= read -r -d '' file; do
echo "renaming $file to $(basename ${file%.$1}.$2)";
mv -- "$file" "${file%.$1}.$2";
done
Credits to aps2012.
Usage
Create a file e.g. called ext-rename (no extension, so you can run it like a command) in e.g. /usr/bin (make sure /usr/bin is added to your $PATH)
run ext-rename [ext1] [ext2] anywhere in terminal, where [ext1] is renaming from and [ext2] is renaming to. An example use would be: ext-rename so dylib, which will rename any file with extension .so to same name but with extension .dylib.
What is wrong is that
echo {} | sed s/.so/.dylib/
is only executed once, before the find is launched, sed is given {} on its input, which doesn't match /.so/ and is left unchanged, so your resulting command line is
find . -name "*.so" -exec mv {} {}
if you have Bash 4
#!/bin/bash
shopt -s globstar
shopt -s nullglob
for file in /path/**/*.so
do
echo mv "$file" "${file/%.so}.dylib"
done
He needs recursion:
#!/bin/bash
function walk_tree {
local directory="$1"
local i
for i in "$directory"/*;
do
if [ "$i" = . -o "$i" = .. ]; then
continue
elif [ -d "$i" ]; then
walk_tree "$i"
elif [ "${i##*.}" = "so" ]; then
echo mv $i ${i%.*}.dylib
else
continue
fi
done
}
walk_tree "."

Resources