I am having one of these mornings where nothing goes to plan. I need to move files to a target directory by chunks of 1,000 at time
I wanted to loop thru my files like so
for i in `find . -name '*XML'`
for((b=0; b<1000; b++))
do
mv $i targetdirect/
done
done
But I get a "-bash: syntax error near unexpected token `done:" error.
What I am missing??
The second for loop is a syntax error. Also you should double-quote "$i".
What do you mean by moving 1000 files at a time? Something like this perhaps?
find . -name '*.XML' -print0 | xargs -r0 -n 1000 mv -t targetdirect
The -print0 and corresponding xargs -0 are a GNU extension to handle arbitrary file names. This works because the null character is an invalid character in file names on Unix; hence, it is safe to use as a delimiter between file names. For regularly named files (no quotes, no newlines etc in the file names) this may seem paranoid, but it is well-documented practice and a FAQ.
Your first for loop has no corresponding do (You have two done, but only one do.)
Related
I have a bunch of files with different names in different subdirectories. I created a txt file with those names but I cannot make find to work using the file. I have seen posts on problems creating the list, on not using find (do not understand the reason though). Suggestions? Is difficult for me to come up with an example because I do not know how to reproduce the directory structure.
The following are the names of the files (just in case there is a formatting problem)
AO-169
AO-170
AO-171
The best that I came up with is:
cat ExtendedList.txt | xargs -I {} find . -name {}
It obviously dies in the first directory that it finds.
I also tried
ta="AO-169 AO-170 AO-171"
find . -name $ta
but it complains find: AO-170: unknown primary or operator
If you are trying to ask "how can I find files with any of these names in subdirectories of the current directory", the answer to that would look something like
xargs printf -- '-o\0-name\0%s\0' <ExtendedList.txt |
xargs -r0 find . -false
The -false is just a cute way to let the list of actual predicates start with "... or".
If the list of names in ExtendedList.txt is large, this could fail if the second xargs decides to break it up between -o and -name.
The option -0 is not portable, but should work e.g. on Linux or wherever you have GNU xargs.
If you can guarantee that the list of strings in ExtendedList.txt does not contain any characters which are problematic to the shell (like single quotes), you could simply say
sed "s/.*/-o -name '&'/" ExtendedList.txt |
xargs -r find . -false
I need to create a script that will go through and add underscores to all files in multiple directories, ignoring the files that already have prefixes. For example, _file1, _file2, file3, file4 needs to look like _file1, _file2, _file3, _file4
I've got little to no knowledge of Unix scripting, so a simple explanation would be greatly appreciated!
You could use one liner like this:
find dir_with_files -regextype posix-extended -type f -regex '^.*\/[^_][^\/]*$' -exec rename -v 's/^(.*\/)([^_][^\/]*)$/$1_$2/' '{}' \;
where dir_with_files is upper dir where you search for your files. Then it finds files with names starting not from _, and each of them is renamed.
Before doing any changes you can use rename with params -n -v showing you what operations will take place, without actually executing them.
find dir_with_files -regextype posix-extended -type f -regex '^.*\/[^_][^\/]*$' -exec rename -v -n 's/^(.*\/)([^_][^\/]*)$/$1_$2/' '{}' \;
From the best Bash resource out there:
Create a glob which matches all of the relevant files.
Loop through all of the matching files.
Remove the underscore from the file name and save the result to a variable.
Prepend an underscore to the variable.
echo the original file name followed by the changed file name using proper quotes to check that they look sane (the quotes will not be printed by echo since they are syntax).
Use mv instead of echo to actually rename the files.
In addition:
If your mv supports -n/--no-clobber, use it to avoid the possibility of data loss in case you mess up
I have a large number of files in directory - ~100k. I want to combine them and pipe them to standard output (I need that to upload them as one file elsewhere), but cat $(ls) complains that -bash: /bin/cat: Argument list too long. I know how to merge all those files into a temporary one, but can I just avoid it?
For a start, cat $(ls) is not the right way to go about this - cat * would be more appropriate. If the number of files is too high, you can use find like this:
find -exec cat {} +
This combines results from find and passes them as arguments to cat, executing as many separate instances as needed. This behaves much in the same way as xargs but doesn't require a separate process or the use of any non-standard features like -print0, which is only supported in some versions of find.
find is recursive by default, so you can specify a -maxdepth 1 to prevent this if your version supports it. If there are other things in the directory, you can also filter by -type (but I guess there aren't, based on your original attempt).
find . -type f -print0 |xargs -0 cat
xargs will invoke cat several times, each time with as many arguments as it can fit on the command line (the combined length of the args can be no more than getconf ARG_MAX).
-print0 (seperate files with \0) for find in combination with -0 (process files separated with \0) for xargs is just a good habit to follow as it will prevent the commands from breaking on filenames with special or white characters in them.
I'm trying to run a program on every file on a dir.
But there is spaces in the name of the file. For example, a file can be named «/my/good/path/MY - AWFUL, FILE.DOC»
And when I'm trying to send the path to my the other tool (a python script), I've got an error saying «MY» is not a existing file. :(
Here is my current bash code:
#!/usr/bin/bash
for file in $(find "/my/pash" -name "*.DOC")
do
newvar=`program "$file"`
done
So… where is my problem?
Thanks everyone :)
Some correct answers, but no explanations so far:
a for loop is intended to iterate over words not lines. The given (unquoted) string is subject to word splitting (which is what is troubling you) and filename expansion, and then you iterate over the resulting words. You could set IFS to contain only a newline. The safest way is to use find -print0 and xargs -0 as demonstrated by Vytenis's answer
find -name "*.DOC" -print0 | xargs -r -0 -n1 program
#!/usr/bin/bash
find "/my/pash" -name "*.DOC" | while read file; do
newvar="$(program "$file")"
done
Note that this only fixes the case where a space or tab is in the file name. If you have a newline in the file name, it gets a little more complicated.
That is because the for loop will take every word inside the result of the find as an element to iterate over. for will see it as:
for file in {/my/good/path/MY, -, AWFUL, FILE.DOC}
echo "$file"
done
And will print:
/my/good/path/MY
-
AWFUL,
FILE.DOC
One solution to this problem is to use the xargs program to pass the result of the find as your python program argument:
find "/my/pash" -name "*.DOC" -print0 | xargs -0 -i program "{}"
the loop treats blanks as delimiter, so try this one:
find "/my/pash" -name "*.DOC" | while read file; do
newvar=`program "$file"`
done
I'm trying to list the files in a directory and do something to them in the Mac OS X prompt.
It should go like this: for f in $(ls -1); do echo $f; done
If I have files without spaces in their names (fileA.txt, fileB.txt), the echo works fine.
If the files include spaces in their names ("file A.txt", "file B.txt"), I get 4 strings (file, A.txt, file, B.txt).
I've tried quoting the listing command, but it only changed the problem.
If I do this: for f in $(ls -1); do echo $f; done
I get: file A.txt\nfile B.txt
(It displays correctly, but it is a single string and I need the 2 lines separated.
Step away from ls if at all possible. Use find from the findutils package.
find /target/path -type f -print0 | xargs -0 your_command_here
-print0 will cause find to output the names separated by NUL characters (ASCII zero). The -0 argument to xargs tells it to expect the arguments separated by NUL characters too, so everything will work just fine.
Replace /target/path with the path under which your files are located.
-type f will only locate files. Use -type d for directories, or omit altogether to get both.
Replace your_command_here with the command you'll use to process the file names. (Note: If you run this from a shell using echo for your_command_here you'll get everything on one line - don't get confused by that shell artifact, xargs will do the expected right thing anyway.)
Edit: Alternatively (or if you don't have xargs), you can use the much less efficient
find /target/path -type f -exec your_command_here \{\} \;
\{\} \; is the escape for {} ; which is the placeholder for the currently processed file. find will then invoke your_command_here with {} ; replaced by the file name, and since your_command_here will be launched by find and not by the shell the spaces won't matter.
The second version will be less efficient since find will launch a new process for each and every file found. xargs is smart enough to pipe the commands to a newly launched process if it can figure it's safe to do so. Prefer the xargs version if you have the choice.
for f in *; do echo "$f"; done
should do what you want. Why are you using ls instead of * ?
In general, dealing with spaces in shell is a PITA. Take a look at the $IFS variable, or better yet at Perl, Ruby, Python, etc.
Here's an answer using $IFS as discussed by derobert
http://www.cyberciti.biz/tips/handling-filenames-with-spaces-in-bash.html
You can pipe the arguments into read. For example, to cat all files in the directory:
ls -1 | while read FILENAME; do cat "$FILENAME"; done
This means you can still use ls, as you have in your question, or any other command that produces $IFS delimited output.
The while loop makes it much easier to do several things to the argument, and makes complex processing more readable in my opinion. A contrived example:
ls -1 | while read FILE
do
echo 1: "$FILE"
echo 2: "$FILE"
done
look --quoting-style option.
for instance, --quoting-style=c would produce :
$ ls --quoting-style=c
"file1" "file2" "dir one"
Check out the manpage for xargs:
it works like this:
ls -1 /tmp/*.jpeg | xargs rm