I wanted to write a short script with the following structure:
find the right folders
cd into them
replace an item
So my problem is that I get the right folders from findbut I don't know how to do the action for every line findis giving me. I tried it with a for loop like this:
for item in $(find command)
do magic for item
done
but the problem is that this command will print the relative pathnames, and if there is a space within my path it will split the path at this point.
I hope you understood my problem and can give me a hint.
You can run commands with -exec option of find directly:
find . -name some_name -exec your_command {} \;
One way to do it is:
find command -print0 |
while IFS= read -r -d '' item ; do
... "$item" ...
done
-print0 and read ... -d '' cause the NUL character to be used to separate paths, and ensure that the code works for all paths, including ones that contain spaces and newlines. Setting IFS to empty and using the -r option to read prevents the paths from being modified by read.
Note that the while loop runs in a subshell, so variables set within it will not be visible after the loop completes. If that is a problem, one way to solve it is to use process substitution instead of a pipe:
while IFS= ...
...
done < <(find command -print0)
Another option, if you have got Bash 4.2 or later, is to use the lastpipe option (shopt -s lastpipe) to cause the last command in pipelines to be run in the current shell.
If the pattern you want to find is simple enough and you have bash 4 you may not need find. In that case, you could use globstar instead for recursive globbing:
#!/bin/bash
shopt -s globstar
for directory in **/*pattern*/; do
(
cd "$directory"
do stuff
)
done
The parentheses make each operation happen in a subshell. That may have performance cost, but usually doesn't, and means you don't have to remember to cd back each time.
If globstar isn't an option (because your find instructions are not a simple pattern, or because you don't have a shell that supports it) you can use find in a similar way:
find . -whatever -exec bash -c 'cd "$1" && do stuff' _ {} \;
You could use + instead of ; to pass multiple arguments to bash each time, but doing one directory per shell (which is what ; would do) has similar benefits and costs to using the subshell expression above.
Related
I've been trying to rename a bunch of files in a proper order using xargs but to no avail. While digging around on piles of similar question, I found answers with the use of sed alongside xargs. Novice me wants to avoid the use of sed. I presume there must be some easier way around.
To be more specific, I've got some files as follows:
Abc.jpg
Def.jpg
Ghi.jpg
Jkl.jpg
and I want these to be renamed in an ordered way, like:
Something1.jpg
Something2.jpg
Something3.jpg
Something4.jpg
Could xargs command along with seq achieve this? If so, how do I implement it?
I don't know why anyone would try to engage sed for this. Probably not xargs or seq, either. Here's a pure-Bash one-liner:
(x=1; for f in *.jpg; do mv "$f" "Something$((x++)).jpg"; done)
At its core, that's a for loop over the files you want to rename, performing a mv command on each one. The files to operate on are expressed via a single glob expression, but you could also name them individually, use multiple globs, or use one of a variety of other techniques. Variable x is used as a simple counter, initialized to 1 before entering the loop. $((x++)) expands to the current value of x, with the side effect of incrementing x by 1. The whole thing is wrapped in parentheses to run it in a subshell, so that nothing in it affects the host shell environment. (In this case, that means it does not create or modify any variable x in the invoking shell.)
If you were putting that in a script instead of typing it on the command line then it would be more readable to split it over several lines:
(
x=1
for f in *.jpg; do
mv "$f" "Something$((x++)).jpg"
done
)
You can type it that way, too, if you wish.
This is an example of how to find, number and rename jpgs.
Regardless of how you use the find (what options you need. recursive, mindepth, maxdepth, regex, ...).
You can add numbers to find ouput with nl and use number and file as 2 arguments for xargs $1, $2
$ find . -type f -name "*.jpg" |nl| xargs -n 2 bash -c 'echo mv "$2" Something"$1".jpg' argv0
the echo echo mv ... will show this
mv ./Jkl.jpg Something1.jpg
mv ./Abc.jpg Something2.jpg
mv ./Def.jpg Something3.jpg
Using sort and testing the number of arguments
$ find . -type f -name "*.jpg" |sort|nl| xargs -n 2 bash -c '[ "$#" -eq 2 ] && echo mv "$2" Something"$1".jpg' argv0
mv ./Abc.jpg Something1.jpg
mv ./Def.jpg Something2.jpg
mv ./Jkl.jpg Something3.jpg
I know you can do this to rename all filenames in a single folder with something like this:
for file in 1_*; do
mv "$file" "${file/1_/}"
done
However, is there a way to do this across multiple folders? For example, it will search through all the folders in the current directory and change them.
I have bash version 4.3
A robust solution, assuming you have GNU or BSD/OSX find:
find . -type f -name '1_*' -execdir sh -c 'echo mv -- "$1" "${1#1_}"' _ {} \;
Note:
- This will only echo the mv commands, to be safe; remove the echo to perform actual renaming.
- The OP's substitution, "${file/1_/}" was changed to the POSIX-compliant "${file#1_}", which is actually closer to the intent.
- If you truly need a substitution such as "${file/1_/}", which the sh on your system may or may not support, it is better to explicitly invoke a shell known to support it, such as bash.
- Symlinks are ignored (both files and directories); use find -L ... to include them (both as potential files to be renamed and to make find descend into symlinks to directories).
find . -type f -name '1_*' finds all files (-type f) with names matching 1_* (-name '1_*') in the current dir.'s (.) subtree.
-execdir executes the command passed to it in the subdirectory in which the file at hand is located.
sh -c 'echo mv -- "$1" "${1#1_}"' _ {} \; invokes the default shell (sh):
with a command string (passed to -c)
mv -- "$1" "${1#1_}" effectively removes prefix 1_ from the filename represented by the first positional parameter ($1).
and dummy parameter _ (which sh will assign to $0, which is not of interest here)
and the path of the file at hand, {}, which the shell will bind to $1;
\; simply terminates -execdir's argument.
Note that -- ensures that any filename that happens to start with - isn't mistaken for an option by mv (applies analogously below).
-execdir is not POSIX-compliant; if a POSIX-compliant variant is therefore more cumbersome:
find . -type f -name '1_*' -exec sh -c \
'cd -- "${1%/*}" && f=${1##*/} && echo mv -- "$f" "${f#1_}"' _ {} \;
cd -- "${1%/*}" changes to the directory in which the file at hand is located.
Note: cd -- "$(dirname -- "$1")" is generally more robust, but less efficient; since we know that $1 is always a path rather than a mere filename in this scenario, we can use the more efficient cd -- "${1%/*}".
f=${1##*/} extracts the mere filename from the file path at hand.
The remainder of the command line then works as above, analogously.
Performance note:
The above solutions are concise, but inefficient, because a shell instance has to be spawned for each matching file.
A potential speed-up is to use a variant of peak's approach, but only if you avoid calling external utilities in the while loop (except for mv):
find . -type f -name '1_*' | while IFS= read -r f; do
new_name=${f##*/} # extract filename
new_name=${new_name#1_} # perform substitution
d=${f%/*} # extract dir
echo mv -- "$f" "$d/$new_name" # call mv to rename
done
The above bears the hypothetical risk of breaking with filenames with embedded newlines (very rare); with GNU or BSD find, this problem could be solved.
With this approach, only a single shell instance is spawned, which processes all matching filenames in a loop - as long as the only external utility that is called in the loop is mv, this will generally be faster than the find-only solutions with -exec[dir].
If you don't want to depend on too many subtleties, consider this very pedestrian approach, which assumes a bash-like shell and that all the usual suspects (find, sed, ....) are directly available:
find . -type f -name "1_*" | while read -r file ; do
x=$(basename "$file")
y=$(sed 's/1_//' <<< "$x")
d=$(dirname "$file")
mv "$file" "$d/$y"
done
(You might want to try this using "mv -i" or "echo mv ....". You might also want to use find with the -follow option.)
Part of my Bash script's intended function is to accept a directory name and then iterate through every file.
Here is part of my code:
#! /bin/bash
# sameln --- remove duplicate copies of files in specified directory
D=$1
cd $D #go to directory specified as default input
fileNum=0 #save file numbers
DIR=".*|*"
for f in $DIR #for every file in the directory
do
files[$fileNum]=$f #save that file into the array
fileNum=$((fileNum+1)) #increment the fileNum
echo aFile
done
The echo statement is for testing purposes. I passed as an argument the name of a directory with four regular files, and I expected my output to look like:
aFile
aFile
aFile
aFile
but the echo statement only shows up once.
A single operation
Use find for this, it's perfect for it.
find <dirname> -maxdepth 1 -type f -exec echo "{}" \;
The flags explained: maxdepth defines how deep int he hierarchy you want to look (dirs in dirs in dirs), type f defines files, as opposed to type d for dirs. And exec allows you to process the found file/dir, which is can be accessed through {}. You can alternatively pass it to a bash function to perform more tasks.
This simple bash script takes a dir as argument and lists all it's files:
#!/bin/bash
find "$1" -maxdepth 1 -type f -exec echo "{}" \;
Note that the last line is identical to find "$1" -maxdepth 1 -type f -print0.
Performing multiple tasks
Using find one can also perform multiple tasks by either piping to xargs or while read, but I prefer to use a function. An example:
#!/bin/bash
function dostuff {
# echo filename
echo "filename: $1"
# remove extension from file
mv "$1" "${1%.*}"
# get containing dir of file
dir="${1%/*}"
# get filename without containing dirs
file="${1##*/}"
# do more stuff like echoing results
echo "containing dir = $dir and file was called $file"
}; export -f dostuff
# export the function so you can call it in a subshell (important!!!)
find . -maxdepth 1 -type f -exec bash -c 'dostuff "{}"' \;
Note that the function needs to be exported, as you can see. This so you can call it in a subshell, which will be opened by executing bash -c 'dostuff'. To test it out, I suggest your comment to mv command in dostuff otherwise you will remove all your extensions haha.
Also note that this is safe for weird characters like spaces in filenames so no worries there.
Closing note
If you decide to go with the find command, which is a great choice, I advise you read up on it because it is a very powerful tool. A simple man find will teach you a lot and you will learn a lot of useful options to find. You can for instance quit from find once it has found a result, this can be handy to check if dirs contain videos or not for example in a rapid way. It's truly an amazing tool that can be used on various occasions and often you'll be done with a one liner (kinda like awk).
You can directly read the files into the array, then iterate through them:
#! /bin/bash
cd $1
files=(*)
for f in "${files[#]}"
do
echo $f
done
If you are iterating only files below a single directory, you are better off using simple filename/path expansion to avoid certain uncommon filename issues. The following will iterate through all files in a given directory passed as the first argument (default ./):
#!/bin/bash
srchdir="${1:-.}"
for i in "$srchdir"/*; do
printf " %s\n" "$i"
done
If you must iterate below an entire subtree that includes numerous branches, then find will likely be your only choice. However, be aware that using find or ls to populate a for loop brings with it the potential for problems with embedded characters such as a \n within a filename, etc. See Why for i in $(find . -type f) # is wrong even though unavoidable at times.
this question is somewhat unique among others asked.
i have a dir with a bunch of folders and they are named using periods to separate every word.
such as: foo.bar.2011.useless.words
the last two words are always the useless ones, so i would like to truncate starting with the second to last period.
not sure of the wording...
many thanks
for file in *.*.*
do
mv "$file" "${file%.*.*}"
done
If your folders with dots are only one level deep, go with Ignacio's answer. However, if you have folders that you want to rename that exist in subdirs such as /toplevel/subdir1/foo.bar.baz.blah/ then you'll need to use my find command below. Unless you have Bash 4.x in which case you can use the shopt -s globstar option.
find /top/level/dir -type d -name '*.*.*' -exec sh -c 'for arg; do echo mv "$arg" "${arg%.*.*}"; done' _ {} +
I added an echo in there so you can do a dry-run without making any changes. Remove the echo if you are satisfied with the output and run it again to make the changes permanent.
Edit
Removed my $tmp var by shamelessly stealing Ignacio's PE
I'm trying to do something like the following:
for file in `find . *.foo`
do
somecommand $file
done
But the command isn't working because $file is very odd. Because my directory tree has crappy file names (including spaces), I need to escape the find command. But none of the obvious escapes seem to work:
-ls gives me the space-delimited filename fragments
-fprint doesn't do any better.
I also tried: for file in "find . *.foo -ls"; do echo $file; done
- but that gives all of the responses from find in one long line.
Any hints? I'm happy for any workaround, but am frustrated that I can't figure this out.
Thanks,
Alex
(Hi Matt!)
You have plenty of answers that explain well how to do it; but for the sake of completion I'll repeat and add to it:
xargs is only ever useful for interactive use (when you know all your filenames are plain - no spaces or quotes) or when used with the -0 option. Otherwise, it'll break everything.
find is a very useful tool; put using it to pipe filenames into xargs (even with -0) is rather convoluted as find can do it all itself with either -exec command {} \; or -exec command {} + depending on what you want:
find /path -name 'pattern' -exec somecommand {} \;
find /path -name 'pattern' -exec somecommand {} +
The former runs somecommand with one argument for each file recursively in /path that matches pattern.
The latter runs somecommand with as many arguments as fit on the command line at once for files recursively in /path that match pattern.
Which one to use depends on somecommand. If it can take multiple filename arguments (like rm, grep, etc.) then the latter option is faster (since you run somecommand far less often). If somecommand takes only one argument then you need the former solution. So look at somecommand's man page.
More on find: http://mywiki.wooledge.org/UsingFind
In bash, for is a statement that iterates over arguments. If you do something like this:
for foo in "$bar"
you're giving for one argument to iterate over (note the quotes!). If you do something like this:
for foo in $bar
you're asking bash to take the contents of bar and tear it apart wherever there are spaces, tabs or newlines (technically, whatever characters are in IFS) and use the pieces of that operation as arguments to for. That is NOT filenames. Assuming that the result of a tearing long string that contains filenames apart wherever there is whitespace yields in a pile of filenames is just wrong. As you have just noticed.
The answer is: Don't use for, it's obviously the wrong tool. The above find commands all assume that somecommand is an executable in PATH. If it's a bash statement, you'll need this construct instead (iterates over find's output, like you tried, but safely):
while read -r -d ''; do
somebashstatement "$REPLY"
done < <(find /path -name 'pattern' -print0)
This uses a while-read loop that reads parts of the string find outputs until it reaches a NULL byte (which is what -print0 uses to separate the filenames). Since NULL bytes can't be part of filenames (unlike spaces, tabs and newlines) this is a safe operation.
If you don't need somebashstatement to be part of your script (eg. it doesn't change the script environment by keeping a counter or setting a variable or some such) then you can still use find's -exec to run your bash statement:
find /path -name 'pattern' -exec bash -c 'somebashstatement "$1"' -- {} \;
find /path -name 'pattern' -exec bash -c 'for file; do somebashstatement "$file"; done' -- {} +
Here, the -exec executes a bash command with three or more arguments.
The bash statement to execute.
A --. bash will put this in $0, you can put anything you like here, really.
Your filename or filenames (depending on whether you used {} \; or {} + respectively). The filename(s) end(s) up in $1 (and $2, $3, ... if there's more than one, of course).
The bash statement in the first find command here runs somebashstatement with the filename as argument.
The bash statement in the second find command here runs a for(!) loop that iterates over each positional parameter (that's what the reduced for syntax - for foo; do - does) and runs a somebashstatement with the filename as argument. The difference here between the very first find statement I showed with -exec {} + is that we run only one bash process for lots of filenames but still one somebashstatement for each of those filenames.
All this is also well explained in the UsingFind page linked above.
Instead of relying on the shell to do that work, rely on find to do it:
find . -name "*.foo" -exec somecommand "{}" \;
Then the file name will be properly escaped, and never interpreted by the shell.
find . -name '*.foo' -print0 | xargs -0 -n 1 somecommand
It does get messy if you need to run a number of shell commands on each item, though.
xargs is your friend. You will also want to investigate the -0 (zero) option with it. find (with -print0) will help to produce the list. The Wikipedia page has some good examples.
Another useful reason to use xargs, is that if you have many files (dozens or more), xargs will split them up into individual calls to whatever xargs is then called upon to run (in the first wikipedia example, rm)
find . -name '*.foo' -print0 | xargs -0 sh -c 'for F in "${#}"; do ...; done' "${0}"
I had to do something similar some time ago, renaming files to allow them to live in Win32 environments:
#!/bin/bash
IFS=$'\n'
function RecurseDirs
{
for f in "$#"
do
newf=echo "${f}" | sed -e 's/[\\/:\*\?#"\|<>]/_/g'
if [ ${newf} != ${f} ]; then
echo "${f}" "${newf}"
mv "${f}" "${newf}"
f="${newf}"
fi
if [[ -d "${f}" ]]; then
cd "${f}"
RecurseDirs $(ls -1 ".")
fi
done
cd ..
}
RecurseDirs .
This is probably a little simplistic, doesn't avoid name collisions, and I'm sure it could be done better -- but this does remove the need to use basename on the find results (in my case) before performing my sed replacement.
I might ask, what are you doing to the found files, exactly?