jenkins stuck after restarting remote service agent - bash

This is part of bigger code that checking the OS version and choosing the correct condition by it.
( after checking OS version, it will go to this if condition: )
if [ "$AK" = "$OS6" ]
then
if [ "$(ls -la /etc/init.d/discagent 2>/dev/null | wc -l)" == 1 ]
then
/etc/init.d/discagent restart 2>&1 > /dev/null
/etc/init.d/discagent status |& grep -qe 'running'
if [ $? -eq 0 ] ; then
echo " Done "
else
echo " Error "
fi
fi
fi
If im hashing service discagent restart the pipeline is passing.
But if im not hashing it then it hang and not given any errors, And on the output file it is showing only the second server (out of few) that its hang on, And not moving to the next server.
what could be the issue?
p. S
while running it direct on the server it is working.

Run this script manually on the servers, that this script will be running on.
You can use xtrace -x which would show each statement before being executed, when you use with -v with -x so this would be -xv and the statement would be outputted, and then the line after the output of the statement with the variables substituted before the code is substituted.
using -u would show you the exact error when this occurs.
Another command is the trap command to debug your bash script.
More information on the following webpage,
https://linuxconfig.org/how-to-debug-bash-scripts

Related

How can I test a command before a pipe, capture it's output and understand if the command ran successfully or not

I'm writting a script that will execute commands on different servers automatically and I am trying to log all it's content and output. I am having difficulties with redirections and command status output. Meaning was the command successfull and what is its output?
I have tried many directions such redirecting the command output to a function or file. I have tried a simple if statement. Both worked for there respected function. But when I am trying to combine both of them the script always return the command to be successfull. And to some level it is.
#!/bin/bash
function Executing(){
if [ "$1" != "" ]; then
if [ $DEBUG = "true" ]; then
echo "$1" | Debug i s
if $1 2>&1 | Debug o o;then
echo "$1" | Debug s r
else
echo "$1" | Debug e r
fi
else
eval $1
fi
fi
}
Executing "apt-get update"
Also note that the system requires sudo to execute apt-get update. Thus the script should report & log an error. Right now tho, whenever I call the Executing function, the function returns a successful execution. I am pretty sure it does because of the added pipe | debug o o that captures the output and formats it. Which I'll later redirect to a log file.
So I tested if apt-get update;then echo yes;else echo no; fi which worked in my terminal. But as soon as I pipe the output to the function debug, the if statement returns true.
Try with set -o pipefail.
Check the manual: The Set Builtin
Usually, bash returns the exit code of the last command of a pipe group, so your Debug command was the one that was checked. With that setting, bash returns the exit status of the last (rightmost) command to exit with a non-zero status. If your command fails, that is the status that get propagated.

egrep is returning 0 in bash script but executing manually the command returning correct value | unix bash

I'm executing a bash script that returns me if ftp connection failed or was success using egrep, the issue is that when I'm trying to get a word with egrep is returning 0 but If I execute the command manually is returning 2.
this is my code:
#Create the FTP Connection.
for ip_address in ${IP_ADDRESS[#]}; do
ftp ${ip_address} <<ftp_commands > ${FTP_RESULTS}
user "${USER_ID}" "${USER_PASSWORD}"
pwd
bye
ftp_commands
ftp_result_id=`egrep -c "Login failed|Connection refused|Not connected|Connection timed out" ${FTP_RESULTS}`
if [ ${ftp_result_id} -gt 0 ]; then
echo "$(date +%m/%d/%y_%H:%M:%S) - ${ip_address} - Not connected" >> ${CONNECTION_RESULTS_FILE}
else
echo "$(date +%m/%d/%y_%H:%M:%S) - ${ip_address} - Connected" >> ${CONNECTION_RESULTS_FILE}
fi
done
the ftp_results_id is returning 0 in the egrep -c command, but I'm executing manually after it ran and create the file "FTP_RESULTS" and is working, it suppose that found 2 matches with "Not connected"
any suggestion?
The egrep -c command counts the matches.
Then you use a condition to do something if there are more than 0 matches.
A simpler and better solution is to use the exit code of egrep.
egrep exits with 0 (= success) if it found a match,
and non-zero otherwise.
You can write the if statement like this:
if egrep -q "Login failed|Connection refused|Not connected|Connection timed out" "${FTP_RESULTS}"; then
This is equivalent to the logic in your posted code.
There's no need for the ftp_result_id variable.
And there's no need to save the output of egrep.
I added the -q flag so that egrep doesn't produce any output.
None needed.

Condition to check if a backup ran successfully and start another one

I'm trying to set a condition to a crontab script for backup to don't start another backup if the last one is not yet completed or if the script is still running, in case a backup will run slower or something like this. For this I created something similar but linux first is creating the process and than will execute the scripts commands so it will allways exist with "process is running":
ps auxw | grep backup.sh | grep -v grep > /dev/null
if [ $? = 0 ]; then
echo "process is running"
exit 1
else
./backup.sh
fi
If the code snippet comes from backup.sh file, then you can put the above verification into a separate file. Then grep will not match "itself".
Another way is using additional in-use files. Create the in-use file and - in case when the file exists - exit 1. Just make sure the in-use file is removed after the script finishes.
#!/usr/bin/env bash
set -o errexit
trap cleanup ERR INT QUIT
cleanup()
{
rm -f "$INUSE"
}
INUSE=/home/abc/inuse/backup.inuse
if [ if -f "$INUSE" ]; then
echo "process is running"
exit 1
else
touch "$INUSE"
fi
# backup starts in here
# end of backup
cleanup

Not able to force exit on Jenkins Build

I've been having a lot of trouble with this so here goes.
I have a Jenkins build that executes the following shell script:
#!/bin/sh -x
if [ 'grep -c "It misses" log' -gt 0 ];
then exit 1;
fi
I know that the grep returns 1 when it finds something and technically Jenkins should mark the build as failed on a non-zero exit, but the jenkins still marks it as a success.
The console output for the jenkins build when running the script is:
Started by user bla
[project_name] $ /bin/sh -x /var/tmp/hudson41276.sh
+ [ grep -c "It misses" log -gt 0 ]
Finished: SUCCESS
Could anybody give me a hand and point out what I'm missing here?
Thanks,
CJ
If I understand right, you want the job to fail if "It misses" is not found in file "log". You can do this by not using the -c option of grep, just redirect the output like this:
grep "It misses" log > /dev/null
Grep will return 0 if it finds the phrase, and the job will succeed. If it does not find the phrase, grep will return 1, and the job will fail. If you want it the other way around (fail if it does find the phrase) just use grep -v. $? is your friend when you want to be sure of the exit status of a shell command.
Try this:
#!/bin/sh
set -e
grep -c "It misses" log
set -e: Exit at the first error.
grep -c 'arg': Exit 1 if nothing was grepped.
The problem is with your script, not Jenkins. The part of your script where you attempt to compare the exit code of grep looks like this:
if [ 'grep -c "It misses" log' -gt 0 ] ...
This will not even execute grep. In fact, it is simply comparing a string to a number.
You were probably attempting to do:
if [ `grep -c "It misses" log` -gt 0 ] ...
Note the use of backticks (`). Shell will execute grep and replace the backticks with the output of grep.
Bonus item: the condition in the if statement is actually a command that gets executed and its exit code determines where the execution will continue. So... why not use the grep command and it's useful exit code as the condition? (grep will exit with code 0 when it finds matches.)
if grep "It misses" log; then
exit 1
fi
It's shorter, much more readable and ever performs better because it does not need to execute so many commands.
Such a short if statement could even be replaced with a one-liner:
grep "It misses" log && exit 1
By default jenkins start shell with -e, so it exists at first error.
You could turn it off by
set +e
do failing task..

Catching errors in Bash with glassfish commands [return code in pipes]

I am writing a bash script to manage deployments to a GF server for several environments. What I would like to know is how can I get the result of a GF command and then determine whether to continue or exit.
For example
Say I want to redeploy, I have this script
$GF_ASADMIN --port $GF_PORT redeploy --name $EAR_FILE_NAME --keepstate=true $EAR_FILE | tee -a $LOG
The variables are already defined. So GF will start to redeploy and either suceed or fail. I want to check if it does and act accordingly. I have this right after it.
RC=$?
if [[ $RC -eq 0 ]];
then echoInfo "Application Successfully redeployed!" | tee -a $LOG;
else
echoError "Failed to redeploy application!"
exit 1
fi;
However, it doesnt really seem to work .
The problem is the pipe
$GF_ASADMIN ... | tee -a $LOG
$? reflects the return code of tee.
Your are looking for PIPESTATUS. See man bash:
PIPESTATUS
An array variable (see Arrays below) containing a list of exit
status values from the processes in the most-recently-executed
foreground pipeline (which may contain only a single command).
See also this example to clarify the PIPESTATUS
false | true
echo ${PIPESTATUS[#]}
Output is: 1 0
The corrected code is:
RC=${PIPESTATUS[0]}
Or try using a code block redirect, for example:
{
if "$GF_ASADMIN" --port $GF_PORT redeploy --name "$EAR_FILE_NAME" --keepstate=true "$EAR_FILE"
then
echo Info "Application Successfully redeployed!"
else
echo Error "Failed to redeploy application!" >&2
exit 1
fi
} | tee -a "$LOG"

Resources