There are a number of files that I have to check if they exist in a directory. They follow a standard naming convention aside from the file extension so I want to use a wild card e.g:
YYYYMM=201403
FILE_LIST=`cat config.txt`
for file in $FILE_LIST
do
FILE=`echo $file | cut -f1 -d"~"`
SEARCH_NAME=$FILE$YYYYMM
ANSWER=`ssh -q userID#servername 'ls /home/to/some/directory/$SEARCH_NAME* | wc -l'`
returnStatus=$?
if [ $ANSWER=1 ]; then
echo "FILE FOUND"
else
echo "FILE NOT FOUND"
fi
done
The wildcard is not working, any ideas for how to make it visible to the shell?
I had much the same question just now. In despair, I just gave up and used pipes with grep and xargs to get wildcard-like functionality.
Was (none of these worked - and tried others):
ssh -t r#host "rm /path/to/folder/alpha*"
ssh -t r#host "rm \"/path/to/folder/alpha*\" "
ssh -t r#host "rm \"/path/to/folder/alpha\*\" "
Is:
ssh -t r#host "cd /path/to/folder/ && ls | grep alpha | xargs rm"
Note: I did much of my troubleshooting with ls instead of rm, just in case I surprised myself.
It's way better to use STDIN:
echo "rm /path/to/foldef/alpha*" | ssh r#host sh
With this way you can still use shell variables to construct the command. e.g.:
echo "rm -r $oldbackup/*" | ssh r#host sh
Related
What am I doing wrong? Trying to execute chmod o-w for multiple paths after ssh to one server.
File 1.txt contains two columns; one with same server called SERVER_hostname and second with different paths. I want script to ssh to that specific server hostname, and then make myself root(either toor, sudo eksh) and then run the command 'chmod o-w ' to those different paths from second column.
#!/bin/bash
read -r -a server < 1.txt
echo "${server[0]}"
echo "ssh -oBatchMode=yes -q "$(echo "${server[0]}")" '"$(cat 1.txt | awk '{print " sudo eksh ; \ chmod o-w " $NF";"}')"'" | sh
./254.sh
SERVER_hostname
awk: warning: escape sequence `\ ' treated as plain ` '
chmod: changing permissions of ‘/etc/nginx-controller/agent.configurator.conf.default’: Operation not permitted
chmod: changing permissions of ‘/etc/nginx-controller/agent.controller.conf.default’: Operation not permitted
chmod: changing permissions of ‘/etc/nginx-controller/copyright’: Operation not permitted
chmod: missing operand after ‘o-w’
Try 'chmod --help' for more information.
1.txt
SERVER_hostname /etc/nginx-controller/agent.configurator.conf.default
SERVER_hostname /etc/nginx-controller/agent.controller.conf.default
SERVER_hostname /etc/nginx-controller/copyright
There are various ways of achieving this. But all have some tricks up their sleeve due to the usage of ssh. When using ssh in loops or complex constructs, you always have to be aware that ssh will slurp your /dev/stdin
The quickest way to implement would be to do multiple calls to ssh using a while-loop to read the file (See BashFAQ#001). However, we force ssh to use /dev/null as input stream. This way we avoid that the while loop is broken:
while read -r host file; do
[ "$host" ] || continue
[ "$file" ] || continue
</dev/null ssh -oBatchMode=yes -q "${host}" -- sudo eksh -c "chmod o-w -- ${file}"
done < file.txt
The above method will perform multiple calls to ssh and might not be the most efficient way of doing things. You could build up the command using an array to contain the command arguments (See BashFAQ#050). In the case of the OP, this would be the different filenames:
file_list=()
while read -r h f; do [ "$f" ] && file_list+=( "${f}" ); [ "$h" ] && host="$h"; done < file.txt
ssh -oBatchMode=yes -q "${host}" -- sudo eksh -c "chmod o-w -- ${file_list[#]}"
But again, there is an issue here if your argument list is too long. So the trick is now to use xargs directly over ssh. You could do something like this:
file_list=()
while read -r h f; do [ "$f" ] && file_list+=( "${f}" ); [ "$h" ] && host="$h"; done < file.txt
printf "%s\n" "${file_list[#]}" | ssh "$host" "cat - | sudo eksh -c 'xargs chmod o-w --'"
Note: for some reason the command ssh "$host" sudo eksh -c 'xargs chmod o-w --' does not work. That is why we introduce the cat -
I have the below block of shell script code in Jenkinsfile
stage("Compose Source Structure")
{
sh '''
set -x
rm -vf config
wget -nv --no-check-certificate https://test-company/k8sconfigs/test-config
export KUBECONFIG=$(pwd)/test-config
kubectl config view
ns_exists=$(kubectl get namespaces | grep ${consider_namespace})
echo "Validating k8s namespace"
if [ -z "$ns_exists" ]
then
echo "No namespace ${consider_namespace} exists in the cluster ${source_cluster}"
exit 1
else
echo "scanning namespace \'${namespace}\'"
mkdir -p "${HOME}/cluster-backup/${namespace}"
while read -r resource
do
echo "scanning resource \'${resource}\'"
mkdir -p "${HOME}/sync-cluster/${namespace}/${resource}"
while read -r item
do
echo "exporting item \'${item}\'"
kubectl get "$resource" -n "$namespace" "$item" -o yaml > "${HOME}/sync-cluster/${namespace}/${resource}/${BUILD_NUMBER}-${source_cluster}-${consider_namespace}-$item.yaml"
done < <(kubectl get "$resource" -n "$namespace" 2>&1 | tail -n +2 | awk \'{print $1}\')
done < <(kubectl api-resources --namespaced=true 2>/dev/null | tail -n +2 | awk \'{print $1}\')
fi
'''
Unfortunately, I am getting error like below:
++ kubectl get namespaces
++ grep test
+ ns_exists='test Active 2d20h'
+ echo 'Validating k8s namespace'
Validating k8s namespace
/home/jenkins/workspace/k8s-sync-from-cluster#tmp/durable-852103cd/script.sh: line 24: syntax error near unexpected token `<'
I did try to escape "<" with "", so I did like the below
\<
But still having no success, any idea what I am doing wrong here?
From the docs for the sh step (emphasis mine):
Runs a Bourne shell script, typically on a Unix node. Multiple lines are accepted.
An interpreter selector may be used, for example: #!/usr/bin/perl
Otherwise the system default shell will be run, using the -xe flags (you can specify set +e and/or set +x to disable those).
The system default shell on your Jenkins server may be sh, not bash. POSIX sh will not recognize <(command) process substitution.
To specifically use the bash shell, you must include a #!/usr/bin/env bash shebang immediately after your triple quote. Putting a shebang on the next line will have no effect.
I also took the liberty of fixing shellcheck warnings for your shell code, and removing \' escapes that are not necessary.
Try this:
stage("Compose Source Structure")
{
sh '''#!/usr/bin/env bash
set -x
rm -vf config
wget -nv --no-check-certificate https://test-company/k8sconfigs/test-config
KUBECONFIG="$(pwd)/test-config"
export KUBECONFIG
kubectl config view
ns_exists="$(kubectl get namespaces | grep "${consider_namespace}")"
echo "Validating k8s namespace"
if [ -z "$ns_exists" ]
then
echo "No namespace ${consider_namespace} exists in the cluster ${source_cluster}"
exit 1
else
echo "scanning namespace '${namespace}'"
mkdir -p "${HOME}/cluster-backup/${namespace}"
while read -r resource
do
echo "scanning resource '${resource}'"
mkdir -p "${HOME}/sync-cluster/${namespace}/${resource}"
while read -r item
do
echo "exporting item '${item}'"
kubectl get "$resource" -n "$namespace" "$item" -o yaml > "${HOME}/sync-cluster/${namespace}/${resource}/${BUILD_NUMBER}-${source_cluster}-${consider_namespace}-$item.yaml"
done < <(kubectl get "$resource" -n "$namespace" 2>&1 | tail -n +2 | awk '{print $1}')
done < <(kubectl api-resources --namespaced=true 2>/dev/null | tail -n +2 | awk '{print $1}')
fi
'''
}
I'm trying to clean up this script I have and this piece of code is annoying me because I know it can be more DRY:
if grep --version | grep "GNU" > /dev/null ;
then
grep -P -r -l "\x0d" $dir | grep "${fileRegex}"
else
grep -r -l "\x0d" $dir | grep "{$fileRegex}"
fi
My thoughts are to somehow conditionally set a string variable to either "grep -P" or "egrep" and then in a single line do something like:
$(cmdString) -r -l "\x0d" $dir | grep "${fileRegex}"
Or something like that but it doesn't work.
Are you worried about a host which has GNU grep but not egrep? Do such hosts exist?
If not why not just always use egrep? (Though -P and egrep are not the same thing.)
That being said you don't use strings for this (see BashFAQ#50).
You use arrays: grepcmd=(egrep) or grepcmd=(grep -P) and then "${grepcmd[#]}" ....
You can also avoid needing perl mode entirely if you use $'\r' or similar (assuming your shell understands that quoting method).
You can do this:
if grep --version | grep "GNU" > /dev/null
then
cmdString=(grep -P)
else
cmdString=(egrep)
fi
"${cmdString[#]}" -r -l "\x0d" "$dir" | grep "{$fileRegex}"
#Etan Reisner's suggestion worked well. For those that are interested in the final code (this case is for tabs, not windows line endings but it is similar):
fileRegex=${1:-".*\.java"}
if grep --version | grep "GNU" > /dev/null ;
then
cmdString=(grep -P)
else
cmdString=(grep)
fi
arr=$("${cmdString[#]}" -r -l "\x09" . | grep "${fileRegex}")
if [ -n "$dryRun" ]; then
for i in $arr; do echo "$i"; done
else
for i in $arr; do expand -t 7 "$i" > /tmp/e && mv /tmp/e "$i"; done
fi
I have following code in my build script:
if [ -z "$1" ]; then
make -j10 $1 2>&1 | tee log.txt && notify-send -u critical -t 7 "BUILD DONE"
else
make -j10 $1 2>&1 | tee log.txt | grep -i --color "Error" && notify-send -u critical -t 7 "BUILD DONE"
fi
I tried to optimize it to:
local GREP=""
[[ ! -z "$1" ]] && GREP="| grep -i --color Error" && echo "Grepping for ERRORS"
make -j10 $1 2>&1 | tee log.txt "$GREP" && notify-send -u critical -t 7 "BUILD DONE"
But error thrown in make line if $1 isn't empty. I just can't figure out how to pass command with grep pipe through the variable.
Like others have already pointed out, you cannot, in general, expect a command in a variable to work. This is a FAQ.
What you can do is execute commands conditionally. Like this, for example:
( make -j10 $1 2>&1 && notify-send -u critical -t 7 "BUILD DONE" ) |
tee log.txt |
if [ -z "$1" ]; then
grep -i --color "Error"
else
cat
fi
This has the additional unexpected benefit that the notify-send is actually conditioned on the exit code of make (which is probably what you intended) rather than tee (which I would expect to succeed unless you run out of disk or something).
(Or if you want the notification regardless of the success status, change && to just ; -- I think this probably makes more sense.)
This is one of those rare Useful Uses of cat (although I still feel the urge to try to get rid of it!)
You can't put pipes in command variables:
$ foo='| cat'
$ echo bar $foo
bar | cat
The linked article explains how to do such things very well.
As mentioned in #l0b0's answer, the | will not be interpreted as you are hoping.
If you wanted to cut down on repetition, you could do something like this:
if [ $(make -j10 "$1" 2>&1 > log.txt) ]; then
[ "$1" ] && grep -i --color "error" log.txt
notify-send -u critical -t 7 "BUILD DONE"
fi
The inside of the test is common to both branches. Instead of using tee so that the output can be piped, you can just indirect the output to log.txt. If "$1" isn't empty, grep for any errors in log.txt. Either way, do the notify-send.
There are a number of files that I have to check if they exist in a directory. They follow a standard naming convention aside from the file extension so I want to use a wild card e.g:
YYYYMM=201403
FILE_LIST=`cat config.txt`
for file in $FILE_LIST
do
FILE=`echo $file | cut -f1 -d"~"`
SEARCH_NAME=$FILE$YYYYMM
ANSWER=`ssh -q userID#servername 'ls /home/to/some/directory/$SEARCH_NAME* | wc -l'`
returnStatus=$?
if [ $ANSWER=1 ]; then
echo "FILE FOUND"
else
echo "FILE NOT FOUND"
fi
done
The wildcard is not working, any ideas for how to make it visible to the shell?
I had much the same question just now. In despair, I just gave up and used pipes with grep and xargs to get wildcard-like functionality.
Was (none of these worked - and tried others):
ssh -t r#host "rm /path/to/folder/alpha*"
ssh -t r#host "rm \"/path/to/folder/alpha*\" "
ssh -t r#host "rm \"/path/to/folder/alpha\*\" "
Is:
ssh -t r#host "cd /path/to/folder/ && ls | grep alpha | xargs rm"
Note: I did much of my troubleshooting with ls instead of rm, just in case I surprised myself.
It's way better to use STDIN:
echo "rm /path/to/foldef/alpha*" | ssh r#host sh
With this way you can still use shell variables to construct the command. e.g.:
echo "rm -r $oldbackup/*" | ssh r#host sh