I need to find all files recursively with the name 'config.xml' and set them aside for analysis. The paths have spaces in them just to keep it interesting. However, I need them to be unique or they will collide in the same folder. What I would like to do is basically copy them off but using the name of the directory they were found in. The command I want is something like from this question except I need it to do something like $(dirname {}). When I do that, nothing gets moved (but I get no error)
Sample, but non-functional command:
find . -name 'config.xml' -exec sh -c 'cp "$1" "$2.xml"' -- {} "$HOME/data/$(dirname {})" \;
To do this with just one shell, not one per file found (as used by prior answers):
while IFS= read -r -d '' filename; do
outFile="$HOME/data/${filename%/*}.xml"
mkdir -p -- "${outFile%/*}"
cp -- "$filename" "$outFile"
done < <(find . -name 'config.xml' -print0)
This way your find emits a NUL-delimited stream of filenames, consumed one-by-one by the while read loop in the parent shell.
(You could use "$HOME/data/$(dirname "$filename").xml", but from a performance perspective that's really silly: $() fork()s off a subshell, and dirname is an external executable that needs to be exec'd, linked and loaded; no point to all that overhead when you can just do the string manipulation internal to the shell itself).
You may use it like this:
find . -name 'config.xml' -exec bash -c \
'd="$HOME/data/${1%/*}/"; mkdir -p "$d"; command cp -p "$1" "$d"' - {} \;
-exec sh is a little hard to handle, but not impossible. The $(dirname ...) is expanded prior sh is run, so it's equal dirname {} - the dirname of file {}. Do something like -exec sh -c ' .... ' -- {} and put the $(dirname ... ) inside sh script using $1.
find . -name 'config.xml' -exec sh -c 'cp "$1" "$2/data/$(dirname "$1").xml"' -- {} "$HOME" \;
Related
Whole story: I am writing the script that will link all files from one directory to another. New file name will contain an original directory name. I use find at this moment with -execdir option.
This is how I want to use it:
./linkPictures.sh 2017_wien 2017/10
And it will create a symbolic link 2017_wien_picture.jpg in 2017/10 pointing to a file 2017_wien/picture.jpg.
This is my current script:
#!/bin/bash
UPLOAD="/var/www/wordpress/wp-content/uploads"
SOURCE="$UPLOAD/photo-gallery/$1/"
DEST="$UPLOAD/$2/"
find $SOURCE -type f -execdir echo ln -s {} $DEST/"$1"_{} ";"
It prints:
ln -s ./DSC03278.JPG /var/www/wordpress/wp-content/uploads/2017/10/pokus_./DSC03278.JPG
This is what I want:
ln -s ./DSC03278.JPG /var/www/wordpress/wp-content/uploads/2017/10/pokus_DSC03278.JPG
How to implement it? I do not know how to incorporate basename into to strip ./.
To run basename on {} you would need to execute a command through sh:
find "$SOURCE" -type f -execdir sh -c "echo ln -s '{}' \"$DEST/${1}_\$(basename \"{}\")\"" ";"
This won't win any speed contests (because of the sh for every file), but it will work.
All the quoting may look a bit crazy, but it's necessary to make it safe for files that may contain spaces.
You can use this find with bash -c:
find $SOURCE -type f -execdir bash -c 'echo ln -s "$2" "/$DEST/$1"_${2#./}' - "$1" '{}' \;
${2#./} will strip starting ./ from each entry of find command's output.
$1 will be passed as is to bash -c command line.
If you have large number of files to process I suggest using this while loop using a process substitution for faster execution since it doesn't spawn a new bash for every file. Moreover it will also handle filenames with white-spaces and other special characters:
while IFS= read -r file; do
echo ln -s "$file" "/$DEST/${1}_${file#./}"
done < <(find "$SOURCE" -type f -print0)
I'm trying to write a simple scripts that can mv every file within a folder to a folder generated from the current date.
This is my initiatives.
#!/bin/bash
storage_folder=`date +%F` # date is generated to name the folder
mkdir "$storage_folder" #createing a folder to store data
find "$PWD" | xargs -E mv "$storage_folder" # mv everyfile to the folder
xargs is not needed. Try:
find . -exec mv -t "$storage_folder" {} +
Notes:
Find's -exec feature eliminates most needs for xargs.
Because . refers to the current working directoy, find "$PWD" is the same as the simpler find ..
The -t target option to mv tells mv to move all files to the target directory. This is handy here because it allows us to fit the mv command into the natural format for a find -exec command.
POSIX
If you do not have GNU tools, then your mv may not have the -t option. In that case:
find . -exec sh -c 'mv -- "$1" "$storage_folder"' Move {} \;
The above creates one shell process for each move. A more efficient approach, as suggested by Charles Duffy in the comments, passes in the target directory using $0:
find . -exec sh -c 'mv -- "$#" "$0"' "$storage_folder" {} +
Safety
As Gordon Davisson points out in the comments, for safety, you may want to use the -i or -n options to mv so that files at the destination are not overwritten without your explicit approval.
I have a directory named as assets, which in further has a set of directories.
user_images
|-content_images
|-original
|-cropped
|-resize
|-gallery_images
|-slider_images
|-logo
These folders can have folders like original, cropped, resize. And these folders further will have images. These images are named something like this – 14562345+Image.jpeg. I need to replace all the images/files that have + to _.
for f in ls -a;
do
if [[ $f == + ]]
then
cp "$f" "${f//+/_}"
fi
done
I was able to do this in the current directory. But I need to iterate this to other many other directories. How can I do that?
You can use this loop using find in a process substitution:
cd user_images
while IFS= read -r -d '' f; do
echo "$f"
mv "$f" "${f//+/_}"
done < <(find . -name '*+*' -type f -print0)
With find -exec:
find user_images -type f \
-exec bash -c '[[ $0 == *+* ]] && mv "$0" "${0//+/_}"' {} \;
Notice that this uses mv and not cp as the question states "rename", but simply replace by cp if you want to keep the original files.
The bash -c is required to be able to manipulate the file names, otherwise we could use {} directly in the -exec action.
The following will run recursively and will rename all files replacing + with a _ :
find . -name '*+*' -type f -execdir bash -c 'for f; do mv "$f" "${f//+/_}"' _ {} +
Notice the use of -execdir :
Like -exec, but the specified command is run from the subdirectory containing the matched file, which is not normally the directory in which you started find -- Quoted from man find.
Which will protect us in case of directory names matching the pattern *+* which you do not want to rename.
I am trying to process DVD files that are in many different locations on a disk. The thing they have in common is that they (each set of input files) are in a directory named VIDEO_TS. The output in each case will be a single file named for the parent of this directory.
I know I can get a fully qualified path to each directory with:
find /Volumes/VolumeName -type d -name "VIDEO_TS" -print0
and I can get the parent directory by piping to xargs:
find /Volumes/VolumeName -type d -name "VIDEO_TS" -print0 | xargs -0 -I{} dirname {}
and I also know that I can get the parent directory name on its own by appending:
| xargs -o I{} basename {}
What I can't figure out is how do I then pass these parameters to, e.g. HandBrakeCLI:
./HandBrakeCLI -i /path/to/filename/VIDEO_TS -o /path/to/convertedfiles/filename.m4v
I have read here about expansion capability of the shell and suspect that's going to help here (not using dirname or basename for a start), but the more I read the more confused I am getting!
You don't actually need xargs for this at all: You can read a NUL-delimited stream into a shell loop, and run the commands you want directly from there.
#!/bin/bash
source_dir=/Volumes/VolumeName
dest_dir=/Volumes/OtherName
while IFS= read -r -d '' dir; do
name=${dir%/VIDEO_TS} # trim /VIDEO_TS off the end of dir, assign to name
name=${name##*/} # remove everything before last remaining / from name
./HandBrakeCLI -i "$dir" -o "$dest_dir/$name.m4v"
done < <(find "$source_dir" -type d -name "VIDEO_TS" -print0)
See the article Using Find on Greg's wiki, or BashFAQ #001 for general information on processing input streams in bash, or BashFAQ #24 to understand the value of using process substitution (the <(...) construct here) rather than piping from find into the loop.
Also, find contains an -exec action which can be used as follows:
source_dir=/Volumes/VolumeName
dest_dir=/Volumes/OtherName
export dest_dir # export allows use by subprocesses!
find "$source_dir" -type d -name "VIDEO_TS" -exec bash -c '
for dir; do
name=${dir%/VIDEO_TS}
name=${name##*/}
./HandBrakeCLI -i "$dir" -o "$dest_dir/$name.m4v"
done
' _ {} +
This passes the found directory names directly on the argument list to the shell invoked with bash -c. Since the default object for for loop to iterate over is "$#", the argument list, this implicitly iterates over directories found by find.
If I understand what you are trying to do, the simplest solution would be to create a little wrapper which takes a path and invokes your CLI:
File: CLIWrapper
#!/bin/bash
for dir in "$#"; do
./HandBrakeCLI -i "${dir%/*}" -o "/path/to/convertedfiles/${dir##*/}.m4v"
done
Edit: I think I misunderstood the question. It's possible that the above script should read:
./HandBrakeCLI -i "$dir" -o "/path/to/convertedfiles/${dir##*/}.m4v"
or perhaps something slightly different. But the theory is valid. :)
Then you can invoke that script using the -exec option to find. The script loops over its arguments, making it possible for find to send multiple arguments to a single invocation using the + terminator:
find /Volumes/VolumeName -type d -name "VIDEO_TS" -exec ./CLIWrapper {} +
I have read many times that if I want to execute something over all subdirectories I should run something like one of these:
find . -name '*' -exec command arguments {} \;
find . -type f -print0 | xargs -0 command arguments
find . -type f | xargs -I {} command arguments {} arguments
The problem is that it works well with corefunctions, but not as expected when the command is a user-defined function or a script. How to fix it?
So what I am looking for is a line of code or a script in which I can replace command for myfunction or myscript.sh and it goes to every single subdirectory from current directory and executes such function or script there, with whatever arguments I supply.
Explaining in another way, I want something to work over all subdirectories as nicely as for file in *; do command_myfunction_or_script.sh arguments $file; done works over current directory.
Instead of -exec, try -execdir.
It may be that in some cases you need to use bash:
foo () { echo $1; }
export -f foo
find . -type f -name '*.txt' -exec bash -c 'foo arg arg' \;
The last line could be:
find . -type f -name '*.txt' -exec bash -c 'foo "$#"' _ arg arg \;
Depending on what args might need expanding and when. The underscore represents $0.
You could use -execdir where I have -exec if that's needed.
The examples that you give, such as:
find . -name '*' -exec command arguments {} \;
Don't go to every single subdirectory from current directory and execute command there, but rather execute command from the current directory with the path to each file listed by the find as an argument.
If what you want is to actually change directory and execute a script, you could try something like this:
STDIR=$PWD; IFS=$'\n'; for dir in $(find . -type d); do cd $dir; /path/to/command; cd $STDIR; done; unset IFS
Here the current directory is saved to STDIR and the bash Internal Field Separator is set to a newline so names won't split on spaces. Then for each directory (-type d) that find returns, we cd to that directory, execute the command (using the full path here as changing directories will break a relative path) and then cd back to the starting directory.
There may be some way to use find with a function, but it won't be terribly elegant. If you have bash 4, what you probably want to do is use globstar:
shopt -s globstar
for file in **/*; do
myfunction "$file"
done
If you're looking for compatibility with POSIX or older versions of bash, you will be forced to source the file defining your function when you invoke bash. So something like this:
find <args> -exec bash -c '. funcfile;
for file; do
myfunction "$file"
done' _ {} +
But that's just ugly. When I get to this point, I usually just put my function in a script on my PATH and live with it.
If you want to use a bash function, this is one way.
work ()
{
local file="$1"
local dir=$(dirname $file)
pushd "$dir"
echo "in directory $(pwd) working with file $(basename $file)"
popd
}
find . -name '*' | while read line;
do
work "$line"
done