I know this is probably elementary to unix people, but I haven't found a straightforward answer online.
I have a directory with sub-directories. Some of these sub-dirs have .mov files in them. I want to consolidate all the movs to a single directory. I don't need to worry about file naming conflicts because the files are from a digital camera and it names the files incrementally, but divides them into daily folders.
What is the Unix-fu for grabbing all these files and copying (or even better, moving them) to a directory in my home folder?
Thanks.
How about this?
find "$SOURCE_DIRECTORY" -type f -name '*.mov' -exec mv '{}' "$TARGET_DIRECTORY" ';'
If the source and target directories do not overlap this should work fine.
EDIT:
BTW, if you have mixed-case extensions (x.mov, y.Mov, Z.MOV) as is the case with many cameras, this would be better. It uses -iname which is case-insensitive when matching:
find "$SOURCE_DIRECTORY" -type f -iname '*.mov' -exec mv '{}' "$TARGET_DIRECTORY" ';'
Make sure to replace the $SOURCE_DIRECTORY and $TARGET_DIRECTORY variables with the actual directories and that they do not overlap (i.e. the target being somewhere under the source)
EDIT 2:
PS: I just noticed that khachik caught this one with his edit
mv `find . -name "*.mov" | xargs` OUTPUTDIR/
Update after thkala's comment:
find . -iname "*.mov" | while read line; do mv "$line" OUTPUTDIR/; done
If you need to cope with weird filenames (spaces, special characters), try this:
$ cd <source parent directory>
$ find -name '*.mov' -print0 | xargs -0 echo mv -v -t <target directory>
Remove the "echo" above to actually do the move, rather than print what would happen.
"mv -v" gives verbose output, "mv -t ..." specifies the target directory (possibly GNU-specific).
"-print0" and "-0" are extensions to cope with weird filenames. On non-GNU systems you might need to remove those options, which will result in newline-separated data. This will still work on filenames with spaces, but not filenames with newlines (yes, it's possible).
Related
I have been given a list of folders which need to be found and copied to a new location.
I have basic knowledge of bash and have created a script to find and copy.
The basic command I am using is working, to a certain degree:
find ./ -iname "*searchString*" -type d -maxdepth 1 -exec cp -r {} /newPath/ \;
The problem I want to resolve is that each found folder contains the files that I want, but also contains subfolders which I do not want.
Is there any way to limit the recursion so that only the files at the root level of the found folder are copied: all subdirectories and files therein should be ignored.
Thanks in advance.
If you remove -R, cp doesn't copy directories:
cp *searchstring*/* /newpath
The command above copies dir1/file1 to /newpath/file1, but these commands copy it to /newpath/dir1/file1:
cp --parents *searchstring*/*(.) /newpath
for GNU cp and zsh
. is a qualifier for regular files in zsh
cp --parents dir1/file1 dir2 copies file1 to dir2/dir1 in GNU cp
t=/newpath;for d in *searchstring*/;do mkdir -p "$t/$d";cp "$d"* "$t/$d";done
find *searchstring*/ -type f -maxdepth 1 -exec rsync -R {} /newpath \;
-R (--relative) is like --parents in GNU cp
find . -ipath '*searchstring*/*' -type f -maxdepth 2 -exec ditto {} /newpath/{} \;
ditto is only available on OS X
ditto file dir/file creates dir if it doesn't exist
So ... you've been given a list of folders. Perhaps in a text file? You haven't provided an example, but you've said in comments that there will be no name collisions.
One option would be to use rsync, which is available as an add-on package for most versions of Unix and Linux. Rsync is basically an advanced copying tool -- you provide it with one or more sources, and a destination, and it makes sure things are synchronized. It knows how to copy things recursively, but it can't be told to limit its recursion to a particular depth, so the following will copy each item specified to your target, but it will do so recursively.
xargs -L 1 -J % rsync -vi -a % /path/to/target/ < sourcelist.txt
If sourcelist.txt contains a line with /foo/bar/slurm, then the slurm directory will be copied in its entiriety to /path/to/target/slurm/. But this would include directories contained within slurm.
This will work in pretty much any shell, not just bash. But it will fail if one of the lines in sourcelist.txt contains whitespace, or various special characters. So it's important to make sure that your sources (on the command line or in sourcelist.txt) are formatted correctly. Also, rsync has different behaviour if a source directory includes a trailing slash, and you should read the man page and decide which behaviour you want.
You can sanitize your input file fairly easily in sh, or bash. For example:
#!/bin/sh
# Avoid commented lines...
grep -v '^[[:space:]]*#' sourcelist.txt | while read line; do
# Remove any trailing slash, just in case
source=${line%%/}
# make sure source exist before we try to copy it
if [ -d "$source" ]; then
rsync -vi -a "$source" /path/to/target/
fi
done
But this still uses rsync's -a option, which copies things recursively.
I don't see a way to do this using rsync alone. Rsync has no -depth option, as find has. But I can see doing this in two passes -- once to copy all the directories, and once to copy the files from each directory.
So I'll make up an example, and assume further that folder names do not contain special characters like spaces or newlines. (This is important.)
First, let's do a single-pass copy of all the directories themselves, not recursing into them:
xargs -L 1 -J % rsync -vi -d % /path/to/target/ < sourcelist.txt
The -d option creates the directories that were specified in sourcelist.txt, if they exist.
Second, let's walk through the list of sources, copying each one:
# Basic sanity checking on input...
grep -v '^[[:space:]]*#' sourcelist.txt | while read line; do
if [ -d "$line" ]; then
# Strip trailing slashes, as before
source=${line%%/}
# Grab the directory name from the source path
target=${source##*/}
rsync -vi -a "$source/" "/path/to/target/$target/"
fi
done
Note the trailing slash after $source on the rsync line. This causes rsync to copy the contents of the directory, rather than the directory.
Does all this make sense? Does it match your requirements?
You can use find's ipath argument:
find . -maxdepth 2 -ipath './*searchString*/*' -type f -exec cp '{}' '/newPath/' ';'
Notice the path starts with ./ to match find's search directory, ends with /* in order to exclude files in the top level directory, and maxdepth is set to 2 to only recurse one level deep.
Edit:
Re-reading your comments, it seems like you want to preserve the directory you're copying from? E.g. when searching for foo*:
./foo1/* ---> copied to /newPath/foo1/* (not to /newPath/*)
./foo2/* ---> copied to /newPath/foo2/* (not to /newPath/*)
Also, the other requirement is to keep maxdepth at 1 for speed reasons.
(As pointed out in the comments, the following solution has security issues for specially crafted names)
Combining both, you could use this:
find . -maxdepth 1 -type d -iname 'searchString' -exec sh -c "mkdir -p '/newPath/{}'; cp "{}/*" '/newPath/{}/' 2>/dev/null" ';'
Edit 2:
Why not ditch find altogether and use a pure bash solution:
for d in *searchString*/; do mkdir -p "/newPath/$d"; cp "$d"* "/newPath/$d"; done
Note the / at the end of the search string, causing only directories to be considered for matching.
So I have some folder
|-Folder1
||-SubFolder1
||-SubFolder2
|-Folder2
||-SubFolder3
||-SubFolder4
Each subfolder contains several jpg I want to zip to the root folder...
I'm a little bit stuck on "How to enter each folder"
Here is my code:
find ./ -type f -name '*.jpg' | while IFS= read i
do
foldName=${PWD##*/}
zip ../../foldName *
done
The better would be to store FolderName+SubFolderName and give it to the zip command as name...
Zipping JPEGs (for Compression) is Usually Wasted Effort
First of all, attempting to compress already-compressed formats like JPEG files is usually a waste of time, and can sometimes result in archives that are larger than the original files. However, it is sometimes useful to do so for the convenience of having a bunch of files in a single package.
Just something to keep in mind. YMMV.
Use Find's -execdir Flag
What you need is the find utility's -execdir flag. The GNU find man page says:
-execdir command {} +
Like -exec, but the specified command is run from the subdirec‐
tory containing the matched file, which is not normally the
directory in which you started find.
For example, given the following test corpus:
cd /tmp
mkdir -p foo/bar/baz
touch foo/bar/1.jpg
touch foo/bar/baz/2.jpg
you can zip the entire set of files with find while excluding the path information with a single invocation. For example:
find /tmp/foo -name \*jpg -execdir zip /tmp/my.zip {} +
Use Zip's --junk-paths Flag
The zip utility on many systems supports a --junk-paths flag. The man page for zip says:
--junk-paths
Store just the name of a saved file (junk the path), and do not
store directory names.
So, if your find utility doesn't support -execdir, but you do have a zip that supports junking paths, you could do this instead:
find /tmp/foo -name \*jpg -print0 | xargs -0 zip --junk-paths /tmp/my.zip
You can use dirname to get the directory name of a file/directory it is located in.
You can also simplify the find command to search only for directories by using -type d. Then you should use basename to get only the name of the subdirs:
find ./*/* -type d | while read line; do
zip --junk-paths "$(basename $line)" $line/*.jpg
done
Explanation
find ./*/* -type d
will print out all directories located in ./*/* which will result in all subdirs of directories located in the current dir
while read line reads each line from the stream and stores it in the variable "line". Thus $line will be the relative path to the subdir, e.g. "Folder1/Subdir2"
"$(basename $line)" returns the only the name of the subdir, e.g. "Subdir2"
Update: add --junk-paths to the zip command if you do not want the directy paths to be stored in the zip filde
So a little check, I finally got something working:
find ./*/* -type d | while read line; do
#printf '%s\n' "$line"
zip ./"$line" "$line"/*.jpg
done
But this create un archive containing:
Subfolder.zip
Folder
|-Subfolder
||-File1.jpg
||-File2.jpg
||-File3.jpg
Instead I fold like it to be:
Subfolder.zip
|-File1.jpg
|-File2.jpg
|-File3.jpg
So I tried using basename and dirname in differnet combination...Always got some error...
And just to learn how to: what if I would like the new archive to be created in the same root directory as "Folder"?
Ok finally got it!
find ./* -name \*.zip -type f -print0 | xargs -0 rm -rf
find ./*/* -type d | while read line; do
#printf '%s\n' "$line"
zip --junk-paths ./"$line" "$line"/*.jpg
done
find . -name \*.zip -type f -mindepth 2 -exec mv -- '{}' . \;
In first row I simply remove all .zip files,
Then I zip all and in the final row I move all zip to the root directory!
Thanks everbody for your help!
How could I move all .txt files from a folder and all included folders into a target directory .
And preferably rename them to the folder they where included in, although thats not that important. I'm not exactly familiar with bash.
To recursively move files, combine find with mv.
find src/dir/ -name '*.txt' -exec mv -t target/dir/ -- {} +
Or if on a UNIX system without GNU's version of find, such as macOS, use:
find src/dir/ -name '*.txt' -exec mv -- {} target/dir/ ';'
To rename the files when you move them it's trickier. One way is to have a loop that uses "${var//from/to}" to replace all occurrences of from with to in $var.
find src/dir/ -name '*.txt' -print0 | while IFS= read -rd $'\0' file; do
mv -- "$file" target/dir/"${file//\//_}"
done
This is ugly because from is a slash, which needs to be escaped as \/.
See also:
Unix.SE: Understanding IFS= read -r line
BashFAQ: How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
Try this:
find source -name '*.txt' | xargs -I files mv files target
This will work faster than any option with -exec, since it will not invoke a singe mv process for every file which needs to be moved.
If it's just one level:
mv *.txt */*.txt target/directory/somewhere/.
So basically, I have a folder with a bunch of subfolders all with over 100 files in them. I want to take all of the mp3 files (really generic extension since I'll have to do this with jpg, etc.) and move them to a new folder in the original directory. So basically the file structure looks like this:
/.../dir/recup1/file1.mp3
/.../dir/recup2/file2.mp3
... etc.
and I want it to look like this:
/.../dir/music/file1.mp3
/.../dir/music/file2.mp3
... etc.
I figured I would use a bash script that looked along these lines:
#!/bin/bash
STR=`find ./ -type f -name \*.mp3`
FILES=(echo $STR | tr ".mp3 " "\n")
for x in $FILES
do
echo "> [$x]"
done
I just have it echo for now, but eventually I would want to use mv to get it to the correct folder. Obviously this doesn't work though because tr sees each character as a delimiter, so if you guys have a better idea I'd appreciate it.
(FYI, I'm running netbook Ubuntu, so if there's a GUI way akin to Windows' search, I would not be against using it)
If the music folder exists then the following should work -
find /path/to/search -type f -iname "*.mp3" -exec mv {} path/to/music \;
A -exec command must be terminated with a ; (so you usually need to type \; or ';' to avoid interpretion by the shell) or a +. The difference is that with ;, the command is called once per file, with +, it is called just as few times as possible (usually once, but there is a maximum length for a command line, so it might be split up) with all filenames.
You can do it like this:
find /some/dir -type f -iname '*.mp3' -exec mv \{\} /where/to/move/ \;
The \{\} part will be replaced by the found file name/path. The \; part sets the end for the -exec part, it can't be left out.
If you want to print what was found, just add a -print flag like:
find /some/dir -type f -iname '*.mp3' -print -exec mv \{\} /where/to/move/ \;
HTH
I have this code to find files/directories older than 7 days, then execute a mv. However I realise I need a different command for directories and files. -type also does not support fd - a manual says it only supports one character.
find /mnt/third/bt/uploads/ -type f -mtime +7 -exec mv {} /mnt/third/bt/tmp/ \;
How do I move both files and directories >7d into /mnt/third/bt/tmp/ whilst keeping the same structure they had in /mnt/third/bt/uploads/?
Thanks
IMHO, this is a non-trivial problem to do it correctly - at least for me :). I will be happy, if someone more experienced post a better solution.
The script: (must have a GNU find, if your "find" is GNU-version change the gfind to find)
FROMDIR="/mnt/third/bt/uploads"
TODIR="/mnt/third/bt/tmp"
tmp="/tmp/movelist.$$"
cd "$FROMDIR"
gfind . -depth -mtime +7 -printf "%Y %p\n" >$tmp
sed 's/^. //' < $tmp | cpio --quiet -pdm "$TODIR"
while read -r type name
do
case $type in
f) rm "$name";;
d) rmdir "$name";;
esac
done < $tmp
#rm $tmp
Explanation:
find everything what you want move (will copy first and delete after) and store it in a tmpfile (find)
copy a list of things from a tmpfile to the new place (cpio)
and finally remove old files and dirs - based on a list from a tmpfile (while...)
The script does not handling symbolic links, fifo files, etc., and will print zilion errors at the deleting directories what are old, but they're not empty (contain new files or subdirs)
DRY RUN first! :)
If you want to search for both files and directories, find supports boolean operators.