The GNU/Linux version of cp has a nice --update flag:
-u, --update
copy only when the SOURCE file is newer than the destination file or when the destination file is missing
The Mac OS X version of cp lacks this flag.
What is the best way to get the behavior of cp --update by using built-in system command line programs? I want to avoid installing any extra tools (including the GNU version of cp).
rsync has an -u/--update option that works just like GNU cp:
$ rsync -u src dest
Also look at rsync's other options, which are probably what you actually want:
-l, --links copy symlinks as symlinks
-H, --hard-links preserve hard links
-p, --perms preserve permissions
--executability preserve executability
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve times
-a, --archive
This is equivalent to -rlptgoD. It is a quick way of saying you want recursion
and want to preserve almost everything (with -H being a notable omission). The
only exception to the above equivalence is when --files-from is specified, in which
case -r is not implied.
Note that -a does not preserve hardlinks, because finding multiply-linked files is
expensive. You must separately specify -H.
Related
I install files using standard POSIX utilities cp, install, sed, sh.
It is possible to fix any permission with chmod / chown / chgrp but it is dangerous to temporarily expose sensitive data and fix it later.
"Standard" way to deal with the problem is to use install -m MODE -u USER -g GRP.
What if I need to process file with a "dumb" utility (like grep / sed / awk / sh )? How can I prevent data leak from such tools? By using umask 777?
I consider following dangerous:
base64 -d secret.txt >/etc/app.key
sed -e '/^#.*/d' </etc/default/app.cfg >/etc/app.cfg
because file content might be accessible to other users if umask is too open. Also I have to "fix" user/group after redirections...
PS Seems install is not in POSIX... https://pubs.opengroup.org/onlinepubs/9699919799/utilities/contents.html
Also GNU install doesn't read from pipe, so following trick is impossible:
sed ... < $SRC | install -m MODE -u USER -g GRP - $DEST
Some shell allow process substitution (<(cmd) syntax) or one might create named pipe as workaround...
After reading POSIX I see that there is no guaranty for mkdir, cp and other tools to respect umask. Actually umask is a process property and is handled by kernel/syscals.
I'd better use non-standard GNU install with -m MODE (-u, -g).
For dumb tools GNU Bash with process substitution would be handy:
install -m 0700 <(sed ... $SRC) $DST
But I'm not sure of FIFO permissions...
I have an rsync script that backs up user's home folder using this line of code:
/usr/bin/caffeinate -i /usr/bin/rsync -rlptDn --human-readable --progress --ignore-existing --update $PATH/$NAME/ --exclude=".*" --exclude="Public" --exclude="Library" /Volumes/Backup/Users/$NAME\ -\ $DATE
How do I ignore everything in ~/Library/ but their ~/Library/Mail/? I wanted to include this rsync flag, --include="/Library/Mail", but I'm not sure if I should depend too much on rsync exclusions & inclusions as it can become unreliable and varies between different versions of OS X rsync.
Maybe a command-line regex tool would be more useful? Example:
ls -d1 ~/Library/*| grep -v '^mail' > $ALIST
exec <${ALIST}
read SRC
do
.
.
.
$RSYNC..etc...
rsync's --include and --exclude options obey what is called "precedence" so there's a firm rule you can rely which means what you explicitly include before you exclude will be what is sent.
In your case, add --include ~/Library/Mail before the first --exclude.
Assume there are some folders with these structures
/bench1/1cpu/p_0/image/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_1/image/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/2cpu/p_0/image/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_1/image/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
....
What I want to do is to scp the following folders
/bench1/1cpu/p_0/image/
/bench1/1cpu/p_1/image/
/bench1/2cpu/p_0/image/
/bench1/2cpu/p_1/image/
As you can see I want to recursively use scp but excluding all folders that name "fl_X". It seems that scp has not such option.
UPDATE
scp has not such feature. Instead I use the following command
rsync -av --exclude 'fl_*' user#server:/my/dir
But it doesn't work. It only transfers the list of folders!! something like ls -R
Although scp supports recursive directory copying with the -r option, it does not support filtering of the files. There are several ways to accomplish your task, but I would probably rely on find, xargs, tar, and ssh instead of scp.
find . -type d -wholename '*bench*/image' \
| xargs tar cf - \
| ssh user#remote tar xf - -C /my/dir
The rsync solution can be made to work, but you are missing some arguments. rsync also needs the r switch to recurse into subdirectories. Also, if you want the same security of scp, you need to do the transfer under ssh. Something like:
rsync -avr -e "ssh -l user" --exclude 'fl_*' ./bench* remote:/my/dir
You can specify GLOBIGNORE and use the pattern *
GLOBIGNORE='ignore1:ignore2' scp -r source/* remoteurl:remoteDir
You may wish to have general rules which you combine or override by using export GLOBIGNORE, but for ad-hoc usage simply the above will do. The : character is used as delimiter for multiple values.
Assuming the simplest option (installing rsync on the remote host) isn't feasible, you can use sshfs to mount the remote locally, and rsync from the mount directory. That way you can use all the options rsync offers, for example --exclude.
Something like this should do:
sshfs user#server: sshfsdir
rsync --recursive --exclude=whatever sshfsdir/path/on/server /where/to/store
Note that the effectiveness of rsync (only transferring changes, not everything) doesn't apply here. This is because for that to work, rsync must read every file's contents to see what has changed. However, as rsync runs only on one host, the whole file must be transferred there (by sshfs). Excluded files should not be transferred, however.
If you use a pem file to authenticate u can use the following command (which will exclude files with something extension):
rsync -Lavz -e "ssh -i <full-path-to-pem> -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --exclude "*.something" --progress <path inside local host> <user>#<host>:<path inside remote host>
The -L means follow links (copy files not links).
Use full path to your pem file and not relative.
Using sshfs is not recommended since it works slowly. Also, the combination of find and scp that was presented above is also a bad idea since it will open a ssh session per file which is too expensive.
You can use extended globbing as in the example below:
#Enable extglob
shopt -s extglob
cp -rv !(./excludeme/*.jpg) /var/destination
This one works fine for me as the directories structure is not important for me.
scp -r USER#HOSTNAME:~/bench1/?cpu/p_?/image/ .
Assuming /bench1 is in the home directory of the current user. Also, change USER and HOSTNAME to the real values.
On Linux, I have a --parents option available for the cp command so I can do
cp --parents test/withintest/go.rb test2
http://www.gnu.org/software/coreutils/manual/html_node/cp-invocation.html
On Mac, I do not have this option available. Is there a way to do this on Mac? Why is this option not available?
PS. The purpose of --parents is the following:
‘--parents’ Form the name of each destination file by appending to the
target directory a slash and the specified name of the source file.
The last argument given to cp must be the name of an existing
directory.
For example, the command:
cp --parents a/b/c existing_dir
copies the file a/b/c to existing_dir/a/b/c, creating any missing intermediate directories.
This bothered me quite a lot as well.
A workaround for this could be to use rsync.
rsync -R test/withintest/go.rb test2
has the same effect as cp --parents and OS X comes standard with rsync.
You can use the ditto command on Mac OS X:
The basic form
ditto <src-path> <dst-path>
does what you want. There's a lot more options too - check out the man page.
You can install the GNU version of cp using MacPorts.
After MacPorts is installed you can install the coreutils packages:
sudo port install coreutils
Then you will be able to use the GNU version cp and other core utilitites (ls, date, cat, etc.) by prefixing the command with a g:
gcp --parents test/withintest/go.rb test2
If you want these GNU versions to be used by default you can add the GNU bin update your path. Add the following to your ~/.bash_profile:
export PATH="/opt/local/libexec/gnubin:$PATH"
The Homebrew way:
Install coreutils
brew install coreutils
Use the GNU g- prefixed command
gcp --parents test/withintest/go.rb test2
I used rsync and what I did was:
rsync -R dir/**/file.json destination
Try
mkdir -p `dirname "$file_path"` && cp "$old_dir/$file_path" "$file_path"
This first creates the directory with all itermediates in the relative file path. Then it copies the file to the newly created directory.
I would not replace mac cp with GNU cp. I would also not used ditto because it is not cross-platform. Instead use cross-platform tools, such as rsync:
rsync <srcDir/srcFile> <dst>
Result: dst/srcDir/srcFile
I'm writing some small bash scripts for copiyng certain files/directories in GNU/Linux and Solaris. Everything is OK in Linux, but cp command hasn't the same options in Linux and Solaris.
Copy command is something like this:
cp -ruv $source $dest
Unfortunately I don't know how to achieve copy verbose and copy update in Solaris. Any idea?
Thanks
Unfortunately, cp under Solaris doesn't have that option. man solaris should reveal that.
Are you comfortable making your script depend on rsync?
Or, if possible, you can install the coreutils package and use GNU's cp.
I ran into a similar issue myself and found that gcp takes care of it too. I've made installing coreutils part of my standard system setup.
I run these on a new Solaris install:
pkgadd -d http://get.opencsw.org/now
pkgutil -U
pkgutil -i -y coreutils
pkgutil -a vim
pkgutil -i -y vim
pkgutil -i -y findutils
Remember to add the path - and the documentation path - to your profile, and possibly to the system profile at /etc/profile:
# Set the program path
PATH=$PATH:/usr/sfw/bin:/usr/sfw/sbin:/usr/openwin/bin:/opt/csw/bin:/usr/ccs/bin:/usr/local/bin:/usr/local
export PATH
# Set the documentation path
MANPATH="$MANPATH:/usr/share/man:/opt/sfw/man:/opt/csw/man"
export MANPATH
It sounds like you might be new to Solaris - as I am relatively new. I also do these, which shouldn't affect anything.
I set VIM as the default editor instead of VI - it's compatible, but has more features, including ANSI color, and some terminal emulators will pass your mouse clicks and scrolling through for even more flexibility:
# Set the default editor
EDITOR=vim
export EDITOR
Then if you are still using the default prompt that doesn't say anything, you might want to add some information - this version requires a Bash shell:
# Set the command prompt, which includes the username, host name, and the current path.
PS1='\u#\h:\w>'
export PS1
To recreate verbose mode, you can tee the output to the controlling terminal (/dev/tty) while the stdoout output of tee itself is passed to cp via xargs.
find /some/source/directory -type f | \
tee /dev/tty | xargs -I {} cp {} /copy/to/this-directory/
Replace the find with whatever you like, so long as it passes the paths to the files to be copied through the pipe to tee.
Tested on a standard Solaris 10 system without extra GNU utils.