I've been trying:
find dev-other -name '*.flac' -type f -exec echo $(echo {} | sed 's,^[^/]*/,,') \;
I expect to see a list of paths to .flac files within dev-other, but without a prepended dev-other/, e.g.:
4515/11057/4515-11057-0095.flac
4515/11057/4515-11057-0083.flac
4515/11057/4515-11057-0040.flac
4515/11057/4515-11057-0105.flac
4515/11057/4515-11057-0017.flac
4515/11057/4515-11057-0001.flac
Instead I see
dev-other/4515/11057/4515-11057-0095.flac
dev-other/4515/11057/4515-11057-0083.flac
dev-other/4515/11057/4515-11057-0040.flac
dev-other/4515/11057/4515-11057-0105.flac
dev-other/4515/11057/4515-11057-0017.flac
Why isn't the sed replace working here even though it works on its own
$ echo $(echo dev-other/4515/11057/4515-11057-0047.flac | sed 's,^[^/]*/,,')
4515/11057/4515-11057-0047.flac
I first tried with expansions:
find dev-other -name '*.flac' -type f -exec a={} echo ${a#*/} \;
But got the errors:
find: a=dev-other/700/122866/700-122866-0001.flac: No such file or directory
find: a=dev-other/700/122866/700-122866-0030.flac: No such file or directory
find: a=dev-other/700/122866/700-122866-0026.flac: No such file or directory
find: a=dev-other/700/122866/700-122866-0006.flac: No such file or directory
find: a=dev-other/700/122866/700-122866-0010.flac: No such file or directory
You can just use parameter expansion for your use-case when using find with the -exec option,
find dev-other -name '*.flac' -type f -exec bash -c 'x=$1; y="${x#*/}"; echo "$y"' bash {} \;
I used a separate shell (use bash or sh) using bash -c because to involve separate string operations involving parameter expansion. Think of each output of find result to be passed as argument to this sub-shell where this manipulation takes place.
When bash -c executes a command, the next argument after the command is used as $0 (the script's "name" in the process listing), and subsequent arguments become the positional parameters ($1, $2, etc.). This means that the filename passed by find (in place of the {}) becomes the first parameter of the script -- and is referenced by $1 inside the mini-script
If you don't want to use an extra bash, use _ in-place
find dev-other -name '*.flac' -type f -exec bash -c 'x=$1; y="${x#*/}"; echo "$y"' _ {} \;
where _ i is a bash predefined variable (not defined in dash for instance): "At shell startup, set to the absolute path-name used to invoke the shell or shell script being executed as passed in the environment or argument list" ( See man bash - Special Parameters section)
Worth looking at Using Find - Complex Actions
Related
I have some wav files. For each of those files I would like to create a new text file with the same name (obviously with the wav extension being replaced with txt).
I first tried this:
find . -name *.wav -exec 'touch $(echo '{}" | sed -r 's/[^.]+\$/txt/')" \;
which outputted
< touch $(echo {} | sed -r 's/[^.]+$/txt/') ... ./The_stranglers-Golden_brown.wav > ?
Then find complained after I hit y key with:
find: ‘touch $(echo ./music.wav | sed -r 's/[^.]+$/txt/')’: No such file or directory
I figured out I was using a pipe and actually needed a shell. I then ran:
find . -name *.wav -exec sh -c 'touch $(echo "'{}"\" | sed -r 's/[^.]+\$/txt/')" \;
Which did the job.
Actually, I do not really get what is being done internally, but I guess a shell is spawned on every file right ? I fear this is memory costly.
Then, what if I need to run this command on a large bunch of files and directories !?
Now is there a way to do this in a more efficient way ?
Basically I need to transform the current file's name and to feed touch command.
Thank you.
This find with bash parameter-expansion will do the trick for you. You don't need sed at all.
find . -type f -name "*.wav" -exec sh -c 'x=$1; file="${x##*/}"; woe="${file%.*}"; touch "${woe}.txt"; ' sh {} \;
The idea is the part
x=$1 represents each of the entry returned from the output of find
file="${x##*/}" strips the path of the file leaving only the last file name part (only filename.ext)
The part woe="${file%.*}" stores the name without extension, and the new file is created with an extension .txt from the name found.
EDIT
Parameter expansion sets us free from using Command substitution $() sub-process and sed.
After looking at sh man page, I figured out that the command up above could be simplified.
Synopsis -c [-aCefnuvxIimqVEbp] [+aCefnuvxIimqVEbp] [-o option_name] [+o option_name] command_string [command_name [argument ...]]
...
-c Read commands from the command_string operand instead of from the stan‐dard input. Special parameter 0 will be set from the command_name oper‐and and the positional parameters ($1, $2, etc.) set from the remaining argument operands.
We can directly pass the file path, skipping the shell's name (which is useless inside the script anyway). So {} is passed as the command_name $0 which can be expanded right away.
We end up with a cleaner command.
find . -name *.wav -exec sh -c 'touch "${0%.*}".txt ;' {} \;
I am trying to process DVD files that are in many different locations on a disk. The thing they have in common is that they (each set of input files) are in a directory named VIDEO_TS. The output in each case will be a single file named for the parent of this directory.
I know I can get a fully qualified path to each directory with:
find /Volumes/VolumeName -type d -name "VIDEO_TS" -print0
and I can get the parent directory by piping to xargs:
find /Volumes/VolumeName -type d -name "VIDEO_TS" -print0 | xargs -0 -I{} dirname {}
and I also know that I can get the parent directory name on its own by appending:
| xargs -o I{} basename {}
What I can't figure out is how do I then pass these parameters to, e.g. HandBrakeCLI:
./HandBrakeCLI -i /path/to/filename/VIDEO_TS -o /path/to/convertedfiles/filename.m4v
I have read here about expansion capability of the shell and suspect that's going to help here (not using dirname or basename for a start), but the more I read the more confused I am getting!
You don't actually need xargs for this at all: You can read a NUL-delimited stream into a shell loop, and run the commands you want directly from there.
#!/bin/bash
source_dir=/Volumes/VolumeName
dest_dir=/Volumes/OtherName
while IFS= read -r -d '' dir; do
name=${dir%/VIDEO_TS} # trim /VIDEO_TS off the end of dir, assign to name
name=${name##*/} # remove everything before last remaining / from name
./HandBrakeCLI -i "$dir" -o "$dest_dir/$name.m4v"
done < <(find "$source_dir" -type d -name "VIDEO_TS" -print0)
See the article Using Find on Greg's wiki, or BashFAQ #001 for general information on processing input streams in bash, or BashFAQ #24 to understand the value of using process substitution (the <(...) construct here) rather than piping from find into the loop.
Also, find contains an -exec action which can be used as follows:
source_dir=/Volumes/VolumeName
dest_dir=/Volumes/OtherName
export dest_dir # export allows use by subprocesses!
find "$source_dir" -type d -name "VIDEO_TS" -exec bash -c '
for dir; do
name=${dir%/VIDEO_TS}
name=${name##*/}
./HandBrakeCLI -i "$dir" -o "$dest_dir/$name.m4v"
done
' _ {} +
This passes the found directory names directly on the argument list to the shell invoked with bash -c. Since the default object for for loop to iterate over is "$#", the argument list, this implicitly iterates over directories found by find.
If I understand what you are trying to do, the simplest solution would be to create a little wrapper which takes a path and invokes your CLI:
File: CLIWrapper
#!/bin/bash
for dir in "$#"; do
./HandBrakeCLI -i "${dir%/*}" -o "/path/to/convertedfiles/${dir##*/}.m4v"
done
Edit: I think I misunderstood the question. It's possible that the above script should read:
./HandBrakeCLI -i "$dir" -o "/path/to/convertedfiles/${dir##*/}.m4v"
or perhaps something slightly different. But the theory is valid. :)
Then you can invoke that script using the -exec option to find. The script loops over its arguments, making it possible for find to send multiple arguments to a single invocation using the + terminator:
find /Volumes/VolumeName -type d -name "VIDEO_TS" -exec ./CLIWrapper {} +
Please find the code below for displaying the directories in the current folder.
What is the meaning of ${arg##/*/} in shell scripting. (Both arg#* and arg##/*/ gives the same output. )
What is the meaning of {} \; in for loop statement.
for arg in `find . -type d -exec ls -d {} \;`
do
echo "Output 1" ${arg##/*/}
echo "Output 2" ${arg#*}
done
Adding to #JoSo's helpful answer:
${arg#*} is a fundamentally pointless expansion, as its result is always identical to $arg itself, since it strips the shortest prefix matching any character (*) and the shortest prefix matching any character is the empty string.
${arg##/*/} - stripping the longest prefix matching pattern /*/ - is useless in this context, because the output paths will be ./-prefixed due to use of find ., so there will be no prefix starting with /. By contrast, ${arg##*/} will work and strip the parent path (leaving the folder-name component only).
Aside from it being ill-advised to parse command output in a for loop, as #JoSo points out,
the find command in the OP is overly complicated and inefficient
(as an aside, just to clarify, the find command lists all folders in the current folder's subtree, not just immediate subfolders):
find . -type d -exec ls -d {} \;
can be simplified to:
find . -type d
The two commands do the same: -exec ls -d {} \; simply does what find does by default anyway (an implied -print).
If we put it all together, we get:
find . -mindepth 1 -type d | while read -r arg
do
echo "Folder name: ${arg##*/}"
echo "Parent path: ${arg%/*}"
done
Note that I've used ${arg%/*} as the second output item, which strips the shortest suffix matching /* and thus returns the parent path; furthermore, I've added -mindepth 1 so that find doesn't also match .
#JoSo, in a comment, demonstrates a solution that's both simpler and more efficient; it uses -exec to process a shell command in-line and + to pass as many paths as possible at once:
find . -mindepth 1 -type d -exec /bin/sh -c \
'for arg; do echo "Folder name: ${arg##*/}"; echo "Parent: ${arg%/*}"; done' \
-- {} +
Finally, if you have GNU find, things get even easier, as you can take advantage of the -printf primary, which supports placeholders for things like filenames and parent paths:
find . -type d -printf 'Folder name: %f\nParen path: %h\n'
Here's a bash-only solution based on globbing (pathname expansion), courtesy of #Adrian Frühwirth:
Caveat: This requires bash 4+, with the shell option globstar turned ON (shopt -s globstar) - it is OFF by default.
shopt -s globstar # bash 4+ only: turn on support for **
for arg in **/ # process all directories in the entire subtree
do
echo "Folder name: $(basename "$arg")"
echo "Parent path: $(dirname "$arg")"
done
Note that I'm using basename and dirname here for parsing, as they conveniently ignore the terminating / that the glob **/ invariably adds to its matches.
Afterthought re processing find's output in a while loop: on the off chance that your filenames contain embedded \n chars, you can parse as follows, using a null char. to separate items (see comments for why -d $'\0' rather than -d '' is used):
find . -type d -print0 | while read -d $'\0' -r arg; ...
${arg##/*/} is an application of "parameter expansion". (Search for this term in your shell's manual, e.g. type man bash in a linux shell). It expands to arg without the longest prefix of arg that matches /*/ as a glob pattern. E.g. if arg is /foo/bar/doo, it expands to doo.
That's bad shell code (similar to item #1 on Bash Pitfalls). The {} \; has not so much to do with shell, but more with the arguments that the find command expects to an -exec subcommand. The {} is replaced with the current filename, e.g. this results in find executing the command ls -d FILENAME with FILENAME replaced by each file it found. The \; serves as a terminator of the -exec argument. See the manual page of find, e.g. type man find on a linux shell, and look for the string -exec there to find the description.
I have found several similar questions that have solutions, except they don't involve variables.
I have a particular pattern in a tree of files and directories - the pattern is the word TEMPLATE. I want a script file to rename all of the files and directories by replacing the word TEMPLATE with some other name that is contained in the variable ${newName}
If I knew that the value of ${newName} was say "Fred lives here", then the command
find . -name '*TEMPLATE*' -exec bash -c 'mv "$0" "${0/TEMPLATE/Fred lives here}"' {} \;
will do the job
However, if my script is:
newName="Fred lives here"
find . -name '*TEMPLATE*' -exec bash -c 'mv "$0" "${0/TEMPLATE/${newName}}"' {} \;
then the word TEMPLATE is replaced by null rather than "Fred lives here"
I need the "" around $0 because there are spaces in the path name, so I can't do something like:
find . -name '*TEMPLATE*' -exec bash -c 'mv "$0" "${0/TEMPLATE/"${newName}"}"' {} \;
Can anyone help me get this script to work so that all files and directories that contain the word TEMPLATE have TEMPLATE replaced by whatever the value of ${newName} is
eg, if newName="A different name" and a I had directory of
/foo/bar/some TEMPLATE directory/with files then the directory would be renamed to
/foo/bar/some A different name directory/with files
and a file called some TEMPLATE file would be renamed to
some A different name file
You have two options.
1) The easiest solution is export newName. If you don't export the variable, then it's not available in subshells, and bash -c is a subshell. That's why you're getting TEMPLATE replaced by nothing.
2) Alternatively, you can try to construct a correctly quoted command line containing the replacement of $newName. If you knew that $newName were reasonably well-behaved (no double quotes or dollar signs, for example), then it's easy:
find . -name '*TEMPLATE*' \
-exec bash -c 'mv "$0" "${0/TEMPLATE/'"${newName}"'}"' {} \;
(Note: bash quoting is full of subtleties. The following has been edited several times, but I think it is now correct.)
But since you can't count on that, probably, you need to construct the command line by substituting both the filename and the substitution as command line parameters. But before we do that, let's fix the $0. You shouldn't be using $0 as a parameter. The correct syntax is:
bash -c '...$1...$1...' bash "argument"
Note the extra bash (many people prefer to use _); it's there to provide a sensible name for the subprocess.
So with that in mind:
find . -name '*TEMPLATE*' \
-exec bash -c 'mv "$1" "${1/TEMPLATE/$2}"' bash {} "$newName" \;
You an get around having to use quotes with IFS=$'\n' and since bash -c is a subshell an export of any variable is required. This works:
#!/bin/bash
IFS=$'\n'
export newName="Fred lives here"
find . -name '*TEMPLATE*' -exec bash -c 'mv "$0" "${0/TEMPLATE/${newName}}"' {} \;
If you do not mind two more lines and would like a script that is easier to read (no export required):
#!/bin/bash
IFS=$'\n'
newName="Fred lives here"
for file in $(find . -name '*TEMPLATE*'); do
mv ${file} ${file/TEMPLATE/${newName}}
done
I have read many times that if I want to execute something over all subdirectories I should run something like one of these:
find . -name '*' -exec command arguments {} \;
find . -type f -print0 | xargs -0 command arguments
find . -type f | xargs -I {} command arguments {} arguments
The problem is that it works well with corefunctions, but not as expected when the command is a user-defined function or a script. How to fix it?
So what I am looking for is a line of code or a script in which I can replace command for myfunction or myscript.sh and it goes to every single subdirectory from current directory and executes such function or script there, with whatever arguments I supply.
Explaining in another way, I want something to work over all subdirectories as nicely as for file in *; do command_myfunction_or_script.sh arguments $file; done works over current directory.
Instead of -exec, try -execdir.
It may be that in some cases you need to use bash:
foo () { echo $1; }
export -f foo
find . -type f -name '*.txt' -exec bash -c 'foo arg arg' \;
The last line could be:
find . -type f -name '*.txt' -exec bash -c 'foo "$#"' _ arg arg \;
Depending on what args might need expanding and when. The underscore represents $0.
You could use -execdir where I have -exec if that's needed.
The examples that you give, such as:
find . -name '*' -exec command arguments {} \;
Don't go to every single subdirectory from current directory and execute command there, but rather execute command from the current directory with the path to each file listed by the find as an argument.
If what you want is to actually change directory and execute a script, you could try something like this:
STDIR=$PWD; IFS=$'\n'; for dir in $(find . -type d); do cd $dir; /path/to/command; cd $STDIR; done; unset IFS
Here the current directory is saved to STDIR and the bash Internal Field Separator is set to a newline so names won't split on spaces. Then for each directory (-type d) that find returns, we cd to that directory, execute the command (using the full path here as changing directories will break a relative path) and then cd back to the starting directory.
There may be some way to use find with a function, but it won't be terribly elegant. If you have bash 4, what you probably want to do is use globstar:
shopt -s globstar
for file in **/*; do
myfunction "$file"
done
If you're looking for compatibility with POSIX or older versions of bash, you will be forced to source the file defining your function when you invoke bash. So something like this:
find <args> -exec bash -c '. funcfile;
for file; do
myfunction "$file"
done' _ {} +
But that's just ugly. When I get to this point, I usually just put my function in a script on my PATH and live with it.
If you want to use a bash function, this is one way.
work ()
{
local file="$1"
local dir=$(dirname $file)
pushd "$dir"
echo "in directory $(pwd) working with file $(basename $file)"
popd
}
find . -name '*' | while read line;
do
work "$line"
done