Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 14 years ago.
Improve this question
I would like to know if it is possible to copy/move files to a destination based on the origin name.
Basically, I have a /mail folder, which has several subfolders such as cur and new etc. I then have an extracted backup in /mail/home/username that is a duplicate. mv -f will not work, as I do not have permission to overwrite the directories, but only the files within.
I get errors such as mv: cannot overwrite directory `/home/username/mail/username.com'
What I want to do is for each file in the directory username.com, move it to the folder of the same name in /mail. There could be any number of folders in place of username.com, with seperate sub sirectories of their own.
What is the best way to do this?
I have to do it this way as due to circumstances I only have access to my host with ftp and bash via php.
edit: clarification
I think I need to clarify what happened. I am on a shared host, and apparently do not have write access to the directories themselves. At least the main ones such as mail and public_html. I made a backup of ~/mail with tar, but when trying to extract it extracted to ~/mail/home/mail etc, as I forgot about the full path. Now, I cannot simply untar because the path is wrong, and I cannot mv -f because I only have write access to files, not directories.
For copying, you should consider using cpio in 'pass' mode (-p):
cd /mail; find . -type f | cpio -pvdmB /home/username/mail
The -v is for verbose; -d creates directories as necessary; -m preserves the modification times on the files; -B means use a larger block size, and may be irrelevant here (it used to make a difference when messing with tape devices). Omitted from this list is the -u flag that does unconditional copying, overwriting pre-existing files in target area. The cd command ensures that the path names are correct; if you just did:
find /mail -type f | cpio -pvdmB /home/username
you would achieve the same result, but only by coincidence - because the sub-directory under /home/username was the same as the absolute pathname of the original. If you needed to do:
find /var/spool/mail -type f | cpio -pvdmB /home/username/mail
then the copied files would be found under /home/username/mail/var/spool/mail, which is unlikely to be what you had in mind.
You can achieve a similar effect with (GNU) tar:
(cd /mail; tar -cf - . ) | (cd /home/username/mail; tar -xf - )
This copies directories, not just files. To do that, you need GNU-only facilities:
(cd /mail; find . -type f | tar -cf - -F - ) | (cd /home/username/mail; tar -xf - )
The first solo dash means 'write to stdout'; the second means 'read from stdin'; the '-F' option means 'read the file names to copy from the named file'.
I'm not entirely clear on what it is that you want to do, but you could try the following:
for file in /mail/*; do
mv -f $file /home/username/mail/$(basename $file)
done
This will move every file and subdirectory in /mail from there into /home/username/mail.
Is using tar an option? You could tar up the directory, and extract it under /mail/ (for I am assuming that is what you want roughly) with tar overwriting existing files and directories.
I'm a bit confused about what it is exactly that you want to do. But you should be able to use the approach of Adam's solution and redirect the errors to a file.
for file in /mail/*; do
mv -f $file /home/username/mail/$(basename $file) 2> /tmp/mailbackup.username.errors
done
DIrectories will not be overwritten and you can check the file so that it only contaions errors you anticipate.
Can you untar it again? The -P option to tar will not strip leading "/", so the absolute pathnames will be respected. From your edit, it sounds like this'll fix it.
Even with your clarification I'm still having a problem understanding exactly what you're doing. However, any chance you can use rsync? The src and dest hosts can be the same host for rsync. As I recall, you can tell rsync to only update files that already exist in the destination area (--existing) and also to ignore directory changes (--omit-dir-times).
Again, I'm not quite understanding your needs here, but rsync is very flexible in backing up files and directories.
Good luck.
Related
I'm not allowed to use rsync on the cluster I'm working on so I need to use cp. I want to copy a large directory including all files and subfolders etc. but without any folders that have the name "outdir".
I tried cp -r -v ./!(outdir) ../target-directory/
but it still copies all folders and contents in deeper directories with the name outdir. It only included the outdir folders in the highest directory.
I also tried cp -r ./*/!(outdir) ../target-directory/ but that one copied all files into the folder without keeping any hirarchy or folders etc.
I also tried certain find commands but it didn't work, but maybe I was just doing something stupid. I'm a beginner with bash so if you could explain your answer and what the flags etc. do that would really be helpfull, I've been trying forever now, on what I think shouldn't be that hard to do.
Instead of cp, you can use tar with option --exclude to control what you want copied or not.
The full command is:
tar --exclude="outdir" -cvpf - . | (cd TARGET_DIRECTORY; tar -xpf -)
So any path that contains the "outdir" pattern will be excluded.
Without the --exclude option, it will copy the entire structure of your current directory under TARGET_DIRECTORY.
You can replace the . in the first tar by your desired source directory.
I have following directory structure :
/home/dir1/abc.jpg
/home/dir1/abc.pdf
/home/dir1/dir2/abc.jpg
/home/dir1/dir2/abc1.jpg
/home/dir1/dir2/dir3/abc.jpg
and I want to copy jpg files from them to a new folder which will have same directory structure, for eg.:
/home/newdir1/abc.jpg
/home/newdir1/dir2/abc.jpg
/home/newdir1/dir2/abc1.jpg
/home/newdir1/dir2/dir3/abc.jpg
How to achieve it using rsync or any other software ?
Please help, Many Thanks !!
From the looks of what you've included in your question, there are a couple of things you might try.
You've specified that you want to "move" files. That means you either use the mv command, or use rsync's --remove-source-files option. For example:
mv /source1/* /source2/* /path/to/targetdir/
or
rsync -a /source1/ /source2/ /path/to/targetdir/
You've no doubt already read the part of rsync's man page that explains the difference between source dirs with and without their trailing slash. If not, read up, because it's important.
If your "thousands of source files [with] similar names" need to be matched from within your source directories, leaving some other files behind, you need to determine whether your "similar names" can be differentiated using pathname expansion or if you should use a regular expression. If the former, then adding the pathname expansion to your sources with either mv or rsync should be sufficient. If you need to use a regex, then find may be a better option:
find /source1/ /source2/ -regex ".*/file[A-F][0-9][0-9].txt" -exec mv "{}" /targetdir/ \;
If these don't solve the problem, then you'll need to supply more detail in your question.
I would try a little shell script like this:
#!/bin/sh
cd /home/dir1
JPEGS=`find . -name "*.jpg"`
tar cf - $JPEGS | (cd /home/newdir1 ; tar xf -)
This first gets the list of all your jpg files with their relative paths, then writes a tar file of them to a pipe into a subshell which changes to the new directory, and then extracts the tar from its stdin.
The layout of the two directories in question is:
folder1
-file1
-file2
folder2
-file1
-file2
The files in the directories are named the same. I tried to use rm to programmatically remove files in folder 2:
for f in /Users/michelegiarratana/1/*; do rm /Users/michelegiarratana/2/$f
but this isn't working. Is $f the full path? How do I get the name?
Thanks
As savanto said in his comment -- yes, you are correct, $f here is the full file path. To fix this, you should use the basename builtin, which extracts just the file name. In your example:
for f in /Users/michelegiarratana/1/*; do rm /Users/michelegiarratana/2/$(basename $f); done
Additionally -- your question applies more broadly to bash shells than just the OS X terminal, in the future, you should try looking for answers in the bash tag, as many of them will be useful to you.
I have a lot of files named the same, with a directory structure (simplified) like this:
../foo1/bar1/dir/file_1.ps
../foo1/bar2/dir/file_1.ps
../foo2/bar1/dir/file_1.ps
.... and many more
As it is extremely inefficient to view all of those ps files by going to the
respective directory, I'd like to copy all of them into another directory, but include
the name of the first two directories (which are those relevant to my purpose) in the
file name.
I have previously tried like this, but I cannot get which file is from where, as they
are all named consecutively:
#!/bin/bash -xv
cp -v --backup=numbered {} */*/dir/file* ../plots/;
Where ../plots is the folder where I copy them. However, they are now of the form file.ps.~x~ (x is a number) so I get rid of the ".ps.~*~" and leave only the ps extension with:
rename 's/\.ps.~*~//g' *;
rename 's/\~/.ps/g' *;
Then, as the ps files have hundreds of points sometimes and take a long time to open, I just transform them into jpg.
for file in * ; do convert -density 150 -quality 70 "$file" "${file/.ps/}".jpg; done;
This is not really a working bash script as I have to change the directory manually.
I guess the best way to do it is to copy the files form the beginning with the names
of the first two directories incorporated in the copied filename.
How can I do this last thing?
If you just have two levels of directories, you can use
for file in */*/*.ps
do
ln "$file" "${file//\//_}"
done
This goes over each ps file, and hard links them to the current directory with the /s replaced by _. Use cp instead of ln if you intend to edit the files but don't want to update the originals.
For arbitrary directory levels, you can use the bash specific
shopt -s globstar
for file in **/*.ps
do
ln "$file" "${file//\//_}"
done
But are you sure you need to copy them all to one directory? You might be able to open them all with yourreader */*/*.ps, which depending on your reader may let browse through them one by one while still seeing the full path.
You should run a find command and print the names first like
find . -name "file_1.ps" -print
Then iterate over each of them and do a string replacement of / to '-' or any other character like
${filename/\//-}
The general syntax is ${string/substring/replacement}. Then you can copy it to the required directory. The complete script can be written as follows. Haven't tested it (not on linux at the moment), so you might need to tweak the code if you get any syntax error ;)
for filename in `find . -name "file_1.ps" -print`
do
newFileName=${filename/\//-}
cp $filename YourNewDirectory/$newFileName
done
You will need to place the script in the same root directory or change the find command to look for the particular directory if you are placing the above script in some other directory.
References
string manipulation in bash
find man page
Is anybody able to point me in the right direction for writing a batch script for a UNIX shell to move files into a zip one at at time and then delete the original.
I cant use the standard zip function because i don't have enough space to fit the zip being created.
So any suggestions please
Try this:
zip -r -m source.zip *
Not a great solution but simple, i ended up finding a python script that recursively zips a folder and just added a line to delete the file after it is added to the zip
You can achieve this using find as
find . -type f -print0 | xargs -0 -n1 zip -m archive
This will move every file into the zip preserving the directory structure. You are then left with empty directories that you can easily remove. Moreover using find gives you a lot of freedom on what files you want to compress.
I use :
zip --move destination.zip src_file1 src_file2
Here the detail of "--move" option from the man pages
--move
Move the specified files into the zip archive; actually, this
deletes the target directories/files after making the specified zip
archive. If a directory becomes empty after removal of the files, the
directory is also removed. No deletions are done until zip has
created the archive without error. This is useful for conserving disk
space, but is potentially dangerous so it is recommended to use it in
combination with -T to test the archive before removing all input
files.