Here is a test:
$ ln -nfs ~/.ssh ssh # Create a symlink in current dir.
$ ack -g ssh # => nothing found
$ ack -a -g ssh # => same here
$ find . -name ssh # => found: ./ssh
What I am doing wrong? )
Two things are going against your expectations.
First, ack will ignore symlinks if you don't use the --follow switch.
Second, and more importantly, ack -g does not find directories. It only finds files. ack is a file-based utility. It is not a generic utility like find is.
Related
My script is running perfectly on co-workers devices (MacOSX with Docker Desktop same as me), but gives me every time the same error and it does not move or only half, the libraries in the deps directory:
OSError: [Errno 18] Invalid cross-device link: '/tmp/pip-target-dzwe_2kc/lib/python/numpy' ->
'/foo/python/numpy'
My script :
#!/bin/bash
export PKG_DIR='python'
export SIDE_DEPS_DIR='deps'
rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}
rm -rf ${SIDE_DEPS_DIR} && mkdir -p ${SIDE_DEPS_DIR}
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.8 \
pip3 install -r requirements.txt -t ${PKG_DIR}
# move stuff to deps
find /${PKG_DIR} -maxdepth 1 -type d \
\( -name "pandas*" -o -name "numpy*" -o -name "numpy.libs*" -o -name "scipy*" -o -name "scipy.libs*" \) -exec mv '{}' ${SIDE_DEPS_DIR} \;
# zip side dependencies
zip -r ge_deps.zip deps
# zip layer
zip -r layers-python38-great-expectations.zip python
It's a script which uses a public lambda docker image to create a lambda layer (basically a zip that contains libraries) and which removes unwanted libraries to put them in another folder deps.
The above code will use the public Docker image lambci / lambda and will install in the empty python directory, libraries which come from a python package which is called 'great-expectations' and which helps to test pipelines of data (which is specified in requirements.txt and is great-expectations==0.12.7)
I have been stuck with this problem for a while and have not found a solution.
Had this exact problem just now.
/tmp and /foo are different devices - /tmp is within the docker OS and /foo is mapped to your local OS.
pip seems to be using shutil.rename() to move the built package from tmp to the final output location (/foo). This fails because they are different devices. Ideally pip would use shutil.move() instead, which will deal with a cross-device move.
As a workaround, you can change the temp folder used by PIP by setting TMPDIR before invoking the pip command. i.e. export TMPDIR=/foo/tmp before calling pip in the docker image. So, the whole command might be something like
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.8 \
/bin/bash -c "export TMPDIR=/foo/tmp && pip3 install -r requirements.txt -t ${PKG_DIR}"
(multiple commands soln taken from https://www.edureka.co/community/10736/how-to-run-multiple-commands-in-docker-at-once - open to better suggestions!)
This will likely be slower because it's using the local OS for temp files, but it avoids the attempted 'rename' across devices from the temp folder to the final output folder.
I'm working on an automated installation of a openSUSE system using AutoYAST, and I'm stumped on a small detail. In order to setup relevant applications in the user's environment, I try to symlink to all applications located in /usr/local/bin in ~/bin (so say /usr/local/bin has the addr2line utility, then I want to have a symlink to that in ~/bin).
I've tried to execute the following snipped to accomplish this:
su -c "for program in `ls /usr/local/bin`; do ln -s /usr/local/bin/$program ~/bin/$program; done" <user>
This snippet executes in the post-script phase of the automatic installation, which is executed as root (and seeing as I want the owner of the symlinks to be the user, this command is executed using su).
However, this does not work, and gives the following output:
++ ls /usr/local/bin
+ su -c 'for program in addr2line
ar
as
c++
c++filt
cpp
elfedit
g++
gcc
gcc-ar
gcc-nm
gcc-ranlib
gcov
gprof
i686-pc-linux-gnu-c++
i686-pc-linux-gnu-g++
i686-pc-linux-gnu-gcc
i686-pc-linux-gnu-gcc-4.9.3
i686-pc-linux-gnu-gcc-ar
i686-pc-linux-gnu-gcc-nm
i686-pc-linux-gnu-gcc-ranlib
ld
ld.bfd
nm
objcopy
objdump
ranlib
readelf
size
strings
strip; do ln -s /usr/local/bin/ ~/bin/; done' <user>
bash: -c: line 1: syntax error near unexpected token `ar'
bash: -c: line 1: `ar'
I've tried several variations of the command, but all seem to not exactly do what I want.
For example, I've also tried:
su -c "for program in /usr/local/bin/*; do ln -s $program ~/bin/; done" <user>
But this only created a symlink to /usr/local/bin in ~/bin.
So I'm a bit stuck on this one... Does anybody have an idea?
You're using double quotes to define your su command, so $program is being evaluated immediately. You want it evaluated when su executes the command. Use single quotes instead:
su -c 'for program in `ls /usr/local/bin`; do ln -s /usr/local/bin/$program ~/bin/$program; done' <user>
You can also use cp -s to create symlinks on a system with GNU cp (like your suse system), which gives you the ability to use recursion and the other fun options of cp.
In the end, I decided to go with the command posted by pacholik to fix this, as my original attempt was over-engineered and thus not necessary.
ln -s /usr/local/bin/* ~/bin
I have an rsync script that backs up user's home folder using this line of code:
/usr/bin/caffeinate -i /usr/bin/rsync -rlptDn --human-readable --progress --ignore-existing --update $PATH/$NAME/ --exclude=".*" --exclude="Public" --exclude="Library" /Volumes/Backup/Users/$NAME\ -\ $DATE
How do I ignore everything in ~/Library/ but their ~/Library/Mail/? I wanted to include this rsync flag, --include="/Library/Mail", but I'm not sure if I should depend too much on rsync exclusions & inclusions as it can become unreliable and varies between different versions of OS X rsync.
Maybe a command-line regex tool would be more useful? Example:
ls -d1 ~/Library/*| grep -v '^mail' > $ALIST
exec <${ALIST}
read SRC
do
.
.
.
$RSYNC..etc...
rsync's --include and --exclude options obey what is called "precedence" so there's a firm rule you can rely which means what you explicitly include before you exclude will be what is sent.
In your case, add --include ~/Library/Mail before the first --exclude.
This problem is killing me and I feel like I've tried everything.
First off, the problem started happening when upgrading to Capistrano 3. Capistrano now utilizes /usr/bin/env before every command when deploying, to make sure the environment setup is correct.
When Capistrano goes to create symlinks to the necessary shared directory and respective files, it attempts commands like:
/usr/bin/env ln -s /full/path /different/full/path
...and then it errors out:
/usr/bin/env: ln: Too many levels of symbolic links
I realize it's not Capistrano's fault, so I began troubleshooting by ssh'ing to my server and trying the same command, and I receive the same error (which at least is good for consistency). I then try the same command without /usr/bin/env:
ln -s /full/path /different/full/path
And it works!!!! Maybe you can see the real solution that I can't?
here is the output of just the /usr/bin/env command:
rvm_bin_path=/home/deployer/.rvm/bin
GEM_HOME=/home/deployer/.rvm/gems/ruby-1.9.3-p392
TERM=xterm-256color
SHELL=/bin/bash
IRBRC=/home/deployer/.rvm/rubies/ruby-1.9.3-p392/.irbrc
SSH_CLIENT=...
OLDPWD=/home/deployer/Sites/example.com
MY_RUBY_HOME=/home/deployer/.rvm/rubies/ruby-1.9.3-p392
SSH_TTY=/dev/pts/0
USER=deployer
LS_COLORS= .....
_system_type=Linux
rvm_path=/home/deployer/.rvm
SSH_AUTH_SOCK=....
rvm_prefix=/home/deployer
MAIL=/var/mail/deployer
PATH=/home/deployer/.rvm/gems/ruby-1.9.3-p392/bin:/home/deployer/.rvm/gems/ruby-1.9.3-p392#global/bin:/home/deployer/.rvm/rubies/ruby-1.9.3-p392/bin:/home/deployer/.rvm/bin:/opt/rubyee/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/deployer/.rvm/bin
PWD=/home/deployer/Sites
LANG=en_US.UTF-8
_system_arch=i386
_system_version=12.04
rvm_version=1.26.4 (latest)
SHLVL=1
HOME=/home/deployer
LOGNAME=deployer
GEM_PATH=/home/deployer/.rvm/gems/ruby-1.9.3-p392:/home/deployer/.rvm/gems/ruby-1.9.3-p392#global
SSH_CONNECTION=....
LESSOPEN=| /usr/bin/lesspipe %s
LESSCLOSE=/usr/bin/lesspipe %s %s
RUBY_VERSION=ruby-1.9.3-p392
_system_name=Ubuntu
_=/usr/bin/env
I have also tried commands like the following, to find potential symlink loops:
find . -maxdepth 20 -type l -exec ls -ld {} +
But is not producing correct results:
lrwxrwxrwx 1 deployer deployer ...
You might not being using the same ln utility.
When invoking it directly from the interactive shell, ln might be overridden e.g. by an alias or by some shell function ln() {...;}.
This does not happen when /usr/bin/env tries to do that (AFAIK it looks for ln in PATH). I suspect that the ln it finds has issues, so you are getting this error.
This is an example scenario that might be similar to your case:
# start from an empty directory
$ ls -l
total 0
# create a problematic `ln` in the current directory
$ ln -s ln ln
$ ls -l
total 0
lrwxrwxrwx 1 me me 2 Jan 7 20:28 ln -> ln
# have an alias for the "real" ln
$ alias ln=/bin/ln
# mess up PATH
$ PATH="$PWD"
Now let's try the two alternatives, /usr/bin/env goes first:
$ /usr/bin/env ln -s /some/path /tmp/path
/usr/bin/env: ln: Too many levels of symbolic links
Then plain ln (remember that we aliased it):
$ ln -s /some/path /tmp/path
$ echo $?
0
$ /bin/ls -l /tmp/path
lrwxrwxrwx 1 me me 10 Jan 7 20:31 /tmp/path -> /some/path
So my suggestion is: look at issues with ln, e.g. by finding all different alternatives that might be visible. In bash you might run this:
$ type -a ln
Try this to find symlink loops:
find . -follow -printf ""
if you are using docker so you should install ruby in your case
docker run ruby
source
https://github.com/docker/for-win/issues/5763#issuecomment-585749243
Assume there are some folders with these structures
/bench1/1cpu/p_0/image/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_1/image/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/2cpu/p_0/image/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_1/image/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
....
What I want to do is to scp the following folders
/bench1/1cpu/p_0/image/
/bench1/1cpu/p_1/image/
/bench1/2cpu/p_0/image/
/bench1/2cpu/p_1/image/
As you can see I want to recursively use scp but excluding all folders that name "fl_X". It seems that scp has not such option.
UPDATE
scp has not such feature. Instead I use the following command
rsync -av --exclude 'fl_*' user#server:/my/dir
But it doesn't work. It only transfers the list of folders!! something like ls -R
Although scp supports recursive directory copying with the -r option, it does not support filtering of the files. There are several ways to accomplish your task, but I would probably rely on find, xargs, tar, and ssh instead of scp.
find . -type d -wholename '*bench*/image' \
| xargs tar cf - \
| ssh user#remote tar xf - -C /my/dir
The rsync solution can be made to work, but you are missing some arguments. rsync also needs the r switch to recurse into subdirectories. Also, if you want the same security of scp, you need to do the transfer under ssh. Something like:
rsync -avr -e "ssh -l user" --exclude 'fl_*' ./bench* remote:/my/dir
You can specify GLOBIGNORE and use the pattern *
GLOBIGNORE='ignore1:ignore2' scp -r source/* remoteurl:remoteDir
You may wish to have general rules which you combine or override by using export GLOBIGNORE, but for ad-hoc usage simply the above will do. The : character is used as delimiter for multiple values.
Assuming the simplest option (installing rsync on the remote host) isn't feasible, you can use sshfs to mount the remote locally, and rsync from the mount directory. That way you can use all the options rsync offers, for example --exclude.
Something like this should do:
sshfs user#server: sshfsdir
rsync --recursive --exclude=whatever sshfsdir/path/on/server /where/to/store
Note that the effectiveness of rsync (only transferring changes, not everything) doesn't apply here. This is because for that to work, rsync must read every file's contents to see what has changed. However, as rsync runs only on one host, the whole file must be transferred there (by sshfs). Excluded files should not be transferred, however.
If you use a pem file to authenticate u can use the following command (which will exclude files with something extension):
rsync -Lavz -e "ssh -i <full-path-to-pem> -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --exclude "*.something" --progress <path inside local host> <user>#<host>:<path inside remote host>
The -L means follow links (copy files not links).
Use full path to your pem file and not relative.
Using sshfs is not recommended since it works slowly. Also, the combination of find and scp that was presented above is also a bad idea since it will open a ssh session per file which is too expensive.
You can use extended globbing as in the example below:
#Enable extglob
shopt -s extglob
cp -rv !(./excludeme/*.jpg) /var/destination
This one works fine for me as the directories structure is not important for me.
scp -r USER#HOSTNAME:~/bench1/?cpu/p_?/image/ .
Assuming /bench1 is in the home directory of the current user. Also, change USER and HOSTNAME to the real values.