Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
So, my program is meant to cycle through a directory and its subdirectories, and when it finds a file that is larger than 100K, asks the user if they want to remove it.
I am using find to cycle through the directories. This is my code:
for file in $(find /home/* -print0 | xargs -0)
I have also tried
for file in $(find /home/* -exec process)
Etc, etc. Pretty much everything on the first five pages of Google.
Just to re-iterate, the problem is that find separates filenames with spaces in them. (i.e. "/home/Old data" would become "/home/Old" and "data"
Anyway, are there any better alternatives that I could be using?
Use a while loop to prevent word splitting:
while read -r file; do
# do something here
echo "${file}" # Remember to quote the variable
done < <(find /path -type f -size +200)
This is a problem with the shell, not with find. devnull has suggested a general fix, but for your specific problem, you can do:
find /path -type f -size +100k -exec ls -lh {} \; -exec rm -i {} \;
For files over 100k, this will list its attributes and ask the user whether to delete.
The process substitution $(...) isn't quoted, so all the filenames come out on one line, separated by spaces. There's no way then to tell which spaces are in filenames and which are separators. But if you quote it, "$(...)", then all the output comes out as a single multi-line string. So doing it this way, passing the command output to a for loop, doesn't work. Instead, use the xargs to do the work.
find /home/* -print0 | xargs -0 -i{} ls -l {}
This works exactly like your find|xargs except that the xargs is given a command to execute. In your case it will be a shell script in a file, e.g. mayberemove.sh. Normally xargs appends as many input lines onto the end of the command as it can, but the -i tells it to run the command with the input line in place of the (arbitrary) string {}. In this case it only uses one input line at a time. Because xargs isn't passing the argument through a shell, but instead runs the command using some variety of exec, there's no need for any more quoting. The -0 means that the arguments are delimited by a null byte, as output by find -print0, to avoid problems with whitespace.
Instead of ls -l, you will want something like:
find /home/* -print0 | xargs -0 -i{} ./mayberemove.sh
where mayberemove.sh is careful to quote its arguments, because it is passing them using the shell:
#!/bin/sh
if ....
then
rm "$1"
fi
Related
I am new to bash scripting and need help:
I need to remove specific files from a directory . My goal is to find in each subdirectory a file called "filename.A" and remove all files that starts with "filename" with extension B,
that is: "filename01.B" , "filename02.B" etc..
I tried:
B_folders="$(find /someparentdirectory -type d -name "*.B" | sed 's# (.*\)/.*#\1#'|uniq)"
A_folders="$(find "$B_folders" -type f -name "*.A")"
for FILE in "$A_folders" ; do
A="${file%.A}"
find "$FILE" -name "$A*.B" -exec rm -f {}\;
done
Started to get problems when the directories name contained spaces.
Any suggestions for the right way to do it?
EDIT:
My goal is to find in each subdirectory (may have spaces in its name), files in the form: "filename.A"
if such files exists:
check if "filename*.B" exists And remove it,
That is: remove: "filename01.B" , "filename02.B" etc..
In bash 4, it's simply
shopt -s globstar nullglob
for f in some_parent_directory/**/filename.A; do
rm -f "${f%.A}"*.B
done
If the space is the only issue you can modify the find inside the for as follows:
find "$FILE" -name "$A*.B" -print0 | xargs -0 rm
man find shows:
-print0
True; print the full file name on the standard output, followed by a null character (instead of the newline character that -print uses). This allows
file names that contain newlines or other types of white space to be correctly interpreted by programs that process the find output. This option corre-
sponds to the -0 option of xargs.
and xarg's manual
-0 Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (every character is taken literal-
ly). Disables the end of file string, which is treated like any other argument. Useful when input items might contain white space, quote marks, or
backslashes. The GNU find -print0 option produces input suitable for this mode.
It's a fairly simple question and I'm sure the gurus here can figure it out right away, however I don't seem to be able to make it work (probably some quotes issue.
I want to place all instances of:
`which cat`
With the following:
/bin/cat
I am running the following command:
for file in $(find . -iname 'PATCH*'); do sed 's/\`which cat\`/\'\/bin\/cat/g' $file; done
I believe I have escaped all characters that don't need to be treated as special ones, however it doesn't seem to do the trick.
Please help :)
It is generally not a good idea to iterate over the output of find since file names can contain the $IFS which would break the loop. Use the -exec option of find instead:
find -iname 'PATCH*' -exec sed -i 's#`which cat`#/bin/cat#g' {} \;
Use a different sed delimiter.
sed 's~`which cat`~/bin/cat~g' file
Example:
$ echo '`which cat`' | sed 's~`which cat`~/bin/cat~g'
/bin/cat
I'm trying to run a program on every file on a dir.
But there is spaces in the name of the file. For example, a file can be named «/my/good/path/MY - AWFUL, FILE.DOC»
And when I'm trying to send the path to my the other tool (a python script), I've got an error saying «MY» is not a existing file. :(
Here is my current bash code:
#!/usr/bin/bash
for file in $(find "/my/pash" -name "*.DOC")
do
newvar=`program "$file"`
done
So… where is my problem?
Thanks everyone :)
Some correct answers, but no explanations so far:
a for loop is intended to iterate over words not lines. The given (unquoted) string is subject to word splitting (which is what is troubling you) and filename expansion, and then you iterate over the resulting words. You could set IFS to contain only a newline. The safest way is to use find -print0 and xargs -0 as demonstrated by Vytenis's answer
find -name "*.DOC" -print0 | xargs -r -0 -n1 program
#!/usr/bin/bash
find "/my/pash" -name "*.DOC" | while read file; do
newvar="$(program "$file")"
done
Note that this only fixes the case where a space or tab is in the file name. If you have a newline in the file name, it gets a little more complicated.
That is because the for loop will take every word inside the result of the find as an element to iterate over. for will see it as:
for file in {/my/good/path/MY, -, AWFUL, FILE.DOC}
echo "$file"
done
And will print:
/my/good/path/MY
-
AWFUL,
FILE.DOC
One solution to this problem is to use the xargs program to pass the result of the find as your python program argument:
find "/my/pash" -name "*.DOC" -print0 | xargs -0 -i program "{}"
the loop treats blanks as delimiter, so try this one:
find "/my/pash" -name "*.DOC" | while read file; do
newvar=`program "$file"`
done
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I want to convert all files recursively which include a specific string in the filename. The specific string in the output filename should be replaced with another string.
I already got the first parts - but have no idea how to replace the string in the output filename.
Input files should include '-default-' in the filename which should be relaced by '-head-' in the output file.
Thanks for your help :)
find . -name '*-default-*' -print0 | xargs -0 -L 1 -I {} convert -compress JPEG -resize "238" {} {}
As tripleee notes, xargs doesn't support manipulating substituted filenames. However, a shell loop might do this and be a little easier to comprehend. Like this:
find . -name '*-default-*' |
while IFS='' read -r f; do
convert -compress JPEG -resize 238 "$f" "${f/-default-/-head-}"
done
Assigning empty string to IFS prevents loss of leading and trailing whitespace. The "-r" switch to "read" disables interpreting backslash as an escape character. If you want to avoid the bashism noted by tripleee, use sed.
xargs has no facility for substituting parts of the input filename, but that is easy enough to work around. Run a simple shell script from xargs and you get the full repertoire of the shell at your disposal.
find . -name '*-default-*' -print0 |
xargs -0 -L 1 -I {} bash -c 'f="{}";
convert -compress JPEG -resize 238 "$f" "${f/-default-/-head-}"'
The ${var/str/subst} syntax is a bashism; it will have to be somewhat more involved if you require POSIX compatible shell script.
If you have tricky file names which may contain e.g. double quotes, you might have to use a standalone script; something like this, perhaps:
#!/bin/bash
for f; do
convert -compress JPEG -resize 238 "$f" "${f/-default-/-head-}"'
done
Then you could just ... | xargs ./script
I'm trying to list the files in a directory and do something to them in the Mac OS X prompt.
It should go like this: for f in $(ls -1); do echo $f; done
If I have files without spaces in their names (fileA.txt, fileB.txt), the echo works fine.
If the files include spaces in their names ("file A.txt", "file B.txt"), I get 4 strings (file, A.txt, file, B.txt).
I've tried quoting the listing command, but it only changed the problem.
If I do this: for f in $(ls -1); do echo $f; done
I get: file A.txt\nfile B.txt
(It displays correctly, but it is a single string and I need the 2 lines separated.
Step away from ls if at all possible. Use find from the findutils package.
find /target/path -type f -print0 | xargs -0 your_command_here
-print0 will cause find to output the names separated by NUL characters (ASCII zero). The -0 argument to xargs tells it to expect the arguments separated by NUL characters too, so everything will work just fine.
Replace /target/path with the path under which your files are located.
-type f will only locate files. Use -type d for directories, or omit altogether to get both.
Replace your_command_here with the command you'll use to process the file names. (Note: If you run this from a shell using echo for your_command_here you'll get everything on one line - don't get confused by that shell artifact, xargs will do the expected right thing anyway.)
Edit: Alternatively (or if you don't have xargs), you can use the much less efficient
find /target/path -type f -exec your_command_here \{\} \;
\{\} \; is the escape for {} ; which is the placeholder for the currently processed file. find will then invoke your_command_here with {} ; replaced by the file name, and since your_command_here will be launched by find and not by the shell the spaces won't matter.
The second version will be less efficient since find will launch a new process for each and every file found. xargs is smart enough to pipe the commands to a newly launched process if it can figure it's safe to do so. Prefer the xargs version if you have the choice.
for f in *; do echo "$f"; done
should do what you want. Why are you using ls instead of * ?
In general, dealing with spaces in shell is a PITA. Take a look at the $IFS variable, or better yet at Perl, Ruby, Python, etc.
Here's an answer using $IFS as discussed by derobert
http://www.cyberciti.biz/tips/handling-filenames-with-spaces-in-bash.html
You can pipe the arguments into read. For example, to cat all files in the directory:
ls -1 | while read FILENAME; do cat "$FILENAME"; done
This means you can still use ls, as you have in your question, or any other command that produces $IFS delimited output.
The while loop makes it much easier to do several things to the argument, and makes complex processing more readable in my opinion. A contrived example:
ls -1 | while read FILE
do
echo 1: "$FILE"
echo 2: "$FILE"
done
look --quoting-style option.
for instance, --quoting-style=c would produce :
$ ls --quoting-style=c
"file1" "file2" "dir one"
Check out the manpage for xargs:
it works like this:
ls -1 /tmp/*.jpeg | xargs rm