I am using gcloud commands to deploy simple VMs. I am using startup script to configure all the required packages on the machine. Our packages are fetched from nexus. Sometimes our changes or nexus network issues result in failure of startup script. I configured a block of code to validate startup script return code. I run gcloud create command and provide startup script, the code block to validate startup script output looks like this..
echo "Step 2: sleep till startup script finishes"
## block to check return code of the VM startup script
MAX_WAIT_TIME=1200
WAIT_TIME=0
RC_CODE=""
echo "[INFO] waiting for startup-script return code"
while [ -z $RC_CODE ] && [ $WAIT_TIME -le $MAX_WAIT_TIME ]
do
RC_CODE=$(gcloud compute instances get-serial-port-output gce-$GCP_ZONE-d-$APP-dev \
--project=$GCP_PROJECT_ID \
--zone=$GCP_ZONE | awk '/^[0-9]{4}/ && /google_metadata_script_runner/ && /startup-script-url exit status/ {sub("\r", "", $NF); print $NF}')
if [[ -z $RC_CODE ]] ; then
echo -n "."
WAIT_TIME=$((WAIT_TIME+10))
sleep 10
else
if [[ $RC_CODE -eq 0 ]] ; then
echo -e "\n[INFO] startup script completed with return code $RC_CODE."
break
else
echo -e "\n[INFO] Startup script completed with return code $RC_CODE."
exit $RC_CODE
fi
fi
done
# to check timeout scenario
if [[ -z $RC_CODE ]]
then
echo "[INFO] Startup script timed out after $((MAX_WAIT_TIME/60))."
echo "[INFO] Startup script completed with return code 1."
exit 1
fi
My output looks like this,
+ ./sh/create-dev-vm-app.sh
Deleted [https://www.googleapis.com/compute/v1/projects/project-name/zones/europe-west1-c/instances/gce-europe-west1-c-d-myapp-dev].
Created [https://www.googleapis.com/compute/v1/projects/project-name/zones/europe-west1-c/instances/gce-europe-west1-c-d-myapp-dev].
waiting for startup-script return code
Specify --start=2473 in the next get-serial-port-output invocation to get only the new output starting from here.
.
Specify --start=32157 in the next get-serial-port-output invocation to get only the new output starting from here.
.
Specify --start=37602 in the next get-serial-port-output invocation to get only the new output starting from here.```
.
Startup script completed with return code 0.
How can I suppress the these 'Specify' lines from appearing on o/p screen? AND/OR print all startup-script messages received from get-serial-port-output output after I receive the return code.
Swallow standard error to avoid the extraneous output:
gcloud ... 2>/dev/null | awk ...
Related
Inside shell script it is picking some files from UNIX directory. Sometimes when any of these file missed then throw error & shell script got failed. I need to add Exceptional Handling in shell script. Here is the below shell script where it is picking files from directory.
#!/bin/ksh
..........
while read file
do
upd_date=$((`date +%s`-`stat -c %Y ${file}`))
file_nm=`basename "${file}"`
if [[ `lsof | grep "${file_nm}"` != '' || $upd_date -lt 60 ]];
then
log_info "Waiting for the file ${file} to complete transfer/skip"
else
log_info "Data file found ${file}"
fi
done<${TEMP}/filepath
...............
exit 0
In the line upd_date=.. throwing error whenever file missed & shell script got failed. So I need to add exception handling if file missed then it will show in log & script will execute successfully.
Use continue in between while loop. continue will skip the current iteration in for, while and until loop.
#!/bin/ksh
..........
while read file
do
[ -f "$file" ] || continue;
upd_date=$((`date +%s`-`stat -c %Y ${file}`))
file_nm=`basename "${file}"`
if [[ `lsof | grep "${file_nm}"` != '' || $upd_date -lt 60 ]];
then
log_info "Waiting for the file ${file} to complete transfer/skip"
else
log_info "Data file found ${file}"
fi
done<${TEMP}/filepath
...............
exit 0
I'm trying to implement a bash script who supposed to search for a word in a Python script terminal output.
The Python script doesn't stop so "&" in the end of the command is needed but the "if [ $? == 0 ] ; then" condition doesn't work.
How it can be solved?
Thanks, Gal.
#!/bin/bash
#Check if Pixhawk is connected
PORT=/dev/ttyPixhawk
end=$((SECONDS+3))
not_exists=f
/usr/local/bin/mavproxy.py --daemon --non-interactive --master=$PORT | grep 'Failed' &> /dev/null &
while [ $SECONDS -lt $end ] ; do
if [ $? == 0 ] ; then
not_exists=t
fi
sleep 1
done
if [ $not_exists=t ] ; then
echo "Not Exists"
else
echo "Exists"
fi
kill $(pgrep -f '/usr/local/bin/mavproxy.py')
Bash doesn't know anything about the output of background commands. Check for yourself with [ 5444 -lt 3 ] & echo $?.
your if statement wouldn't work in any case because $? checks for the return value of the most recent previous command, which in this case is your while loop.
You have a few different options. If you're waiting for some output, and you know how long it is in the output until whatever target you're looking for occurs, you can have the python write to a file and keep checking on the file size with a timeout for failure.
You can also continue with a simple timed approach as you have where you just check the output after a few seconds and decide success or failure based on that.
You can make your python script actually end, or provide more error messages, or write only the relevant parts to file that way.
Furthermore, you really should run your script through shellcheck.net to notice more problems.
You'll need to define your goal and use case more clearly to get real help; all we can really say is "your approach will not work, but there are definitely approaches which will work"
You are checking the status of grep command output inside while loop using $?. This can be done if $? is the next command to be fired after grep and if grep is not a back-group process . But in your script, $? will return the status of while [$SECONDS -lt $end ]. You can try to re-direct the output to a temp file and check it's status
/usr/local/bin/mavproxy.py --daemon --non-interactive --master=$PORT | grep 'Failed' &> tmp.txt &
sleep 3
# If file exists and it's size is greater than 0, [ -s File] will return true
if [ -s tmp.txt ]; then
echo 'pattern exists'
else
echo 'pattern not exists'
fi
I need to create a shell script to place some indicator/flag files in a directory say /dir1/dir2/flag_file_directory based on the request flags received from a shell script in a directory /dir1/dir2/req_flag_file_directory and the source files present in a directory say dir1/dir2/source_file_directory. For this I need to run a script using a while condition in an infinite loop as I do not know when the source files will be made available.
So, my implementation plan is somewhat like this - Lets say JOB1 which is scheduled to run at some time in the morning will first place(touch) a request flag (eg. touch /dir1/dir2/req_flag_file_directory/req1.req), saying that this job is running, so look for the Source files of pattern file_pattern_YYYYMMDD.CSV (the file patterns are different for different jobs) present in the source file directory, if they are present, then count the number. If the count of the files is correct, then first delete the request flag for this job and then touch a indicator/flag file in the /dir1/dir2/flag_file_directory. This indicator/flag file will then be used as an indicator that the source files are all present and the job can be continued to load these files into our system.
I will have all the details related to the jobs and their flag files in a file whose structure is as shown below. Based on the request flag, the script should know what other criterias it should look for before placing the indicator file:
request_flags|source_name|job_name|file_pattern|file_count|indicator_flag_file
req1.req|Sourcename1|jobname1|file_pattern_1|3|ind1.ind
req2.req|Sourcename2|jobname2|file_pattern_2|6|ind2.ind
req3.req|Sourcename3|jobname3|file_pattern_3|1|ind3.ind
req**n**.req|Sourcename**n**|jobname**n**|file_pattern_**n**|2|ind**n**.ind
Please let me know how this can be achieved and also if you have other suggestions or solutions too
Rather have the service daemon script polling in an infinite loop (i.e. waking up periodically to check if it needs to do work), you could use file locking and a named pipe to create an event queue.
Outline of the service daemon, daemon.sh. This script will loop infinitely, blocking by reading from the named pipe at read line until a message arrives (i.e., some other process writes to $RequestPipe).
#!/bin/bash
# daemon.sh
LockDir="/dir1/dir2/req_flag_file_directory"
LockFile="${LockDir}/.MultipleWriterLock"
RequestPipe="${LockDir}/.RequestQueue"
while true ; do
if read line < "$RequestPipe" ; then
# ... commands to be executed after message received ...
echo "$line" # for example
fi
done
An outline of requestor.sh, the script that wakes up the service daemon when everything is ready. This script does all the preparation necessary, e.g. creating files in req_flag_file_directory and source_file_directory, then wakes the service daemon script by writing to the named pipe. It could even send a message that that contains more information for the service daemon, say "Job 1 ready".
#!/bin/bash
# requestor.sh
LockDir="/dir1/dir2/req_flag_file_directory"
LockFile="${LockDir}/.MultipleWriterLock"
RequestPipe="${LockDir}/.RequestQueue"
# ... create all the necessary files ...
(
flock --exclusive 200
# Unblock the service daemon/listener by sending a line of text.
echo Wake up sleepyhead. > "$RequestPipe"
) 200>"$LockFile" # subshell exit releases lock automatically
daemon.sh fleshed out with some error handling:
#!/bin/bash
# daemon.sh
LockDir="/dir1/dir2/req_flag_file_directory"
LockFile="${LockDir}/.MultipleWriterLock"
RequestPipe="${LockDir}/.RequestQueue"
SharedGroup=$(echo need to put a group here 1>&2; exit 1)
#
if [[ ! -w "$RequestPipe" ]] ; then
# Handle 1st time. Or fix a problem.
mkfifo --mode=775 "$RequestPipe"
chgrp "$SharedGroup" "$RequestPipe"
if [[ ! -w "$RequestPipe" ]] ; then
echo "ERROR: request queue, can't write to $RequestPipe" 1>&2
exit 1
fi
fi
while true ; do
if read line < "$RequestPipe" ; then
# ... commands to be executed after message received ...
echo "$line" # for example
fi
done
requestor.sh fleshed out with some error handling:
#!/bin/bash
# requestor.sh
LockDir="/dir1/dir2/req_flag_file_directory"
LockFile="${LockDir}/.MultipleWriterLock"
RequestPipe="${LockDir}/.RequestQueue"
SharedGroup=$(echo need to put a group here 1>&2; exit 1)
# ... create all the necessary files ...
#
if [[ ! -w "$LockFile" ]] ; then
# Handle 1st time. Or fix a problem.
touch "$LockFile"
chgrp "$SharedGroup" "$LockFile"
chmod 775 "$LockFile"
if [[ ! -w "$LockFile" ]] ; then
echo "ERROR: write lock, can't write to $LockFile" 1>&2
exit 1
fi
fi
if [[ ! -w "$RequestPipe" ]] ; then
# Handle 1st time. Or fix a problem.
mkfifo --mode=775 "$RequestPipe"
chgrp "$SharedGroup" "$RequestPipe"
if [[ ! -w "$RequestPipe" ]] ; then
echo "ERROR: request queue, can't write to $RequestPipe" 1>&2
exit 1
fi
fi
(
flock --exclusive 200 || {
echo "ERROR: write lock, $LockFile flock failed." 1>&2
exit 1
}
# Unblock the service daemon/listener by sending a line of text.
echo Wake up sleepyhead. > "$RequestPipe"
) 200> "$LockFile" # subshell exit releases lock automatically
Still having some doubts about the contents of requests file, but I think I've come up with a rather simple solution:
#!/bin/bash
DETAILS_FILE="details.txt"
DETAILS_LINES=$((`wc -l $DETAILS_FILE|awk '{print $1}'`-1)) # to remove banner line (first line)
DETAILS=`tail -$DETAILS_LINES $DETAILS_FILE|tr '\n\r' ' '`
PIDS=()
IFS=' '
waitall () { # PIDS...
## Wait for children to exit and indicate whether all exited with 0 status.
local errors=0
while :; do
debug "Processes remaining: $*"
for pid in $#; do
echo "PID: $pid"
shift
if kill -0 "$pid" 2>/dev/null; then
debug "$pid is still alive."
set -- "$#" "$pid"
elif wait "$pid"; then
debug "$pid exited with zero exit status."
else
debug "$pid exited with non-zero exit status."
((++errors))
fi
done
(("$#" > 0)) || break
# TODO: how to interrupt this sleep when a child terminates?
sleep ${WAITALL_DELAY:-1}
done
((errors == 0))
}
debug () { echo "DEBUG: $*" >&2; }
#function to check for # of sourcefiles matching pattern in dir
#params: req3.req Sourcename3 jobname3 file_pattern_3 1 ind3.ind
check () {
NOFILES=`find $2 -type f | egrep -c $4`
if [ $NOFILES -eq "$5" ];then
echo "Touching file $6. done."
touch $6
else
echo "$NOFILES matching $4 pattern. exiting"
fi
}
echo "parsing $DETAILS_FILE file..."
read -a lines <<< "$DETAILS"
for line in "${lines[#]}"
do
IFS='|'
read -a ARRAY <<< "$line"
echo "Line processed. Dispatching job ${ARRAY[2]}..."
check ${ARRAY[#]} &
IFS=' '
PIDS="$PIDS $!"
#echo $PIDS
done
waitall ${PIDS}
wait
Although not exactly in a infinite loop. This script is intended to run in a crontab.
First it reads details.txt file, as per your example.
After parsing all details, this script dispatches the check function, with sole purpose of counting the number of files matching file_pattern of each source_name folder, and if the number of files is equal to file_count, then touches the indicator_flag_file.
Hope that helps!
I need to launch a Java application on Centos (5.9) startup.
I am trying to start a simple script (named "lanzar.sh") on Centos at boot time:
#!/bin/sh
cd /home/someuser/Desktop/Dist
java -jar SomeApp.jar
I append the line "/bin/sh /home/someuser/Desktop/Dist/lanzar.sh" to /etc/rc.d/rc.local. But the java application does not start. I have:
Granted 755 rights to the /etc/rc.d/rc.local file
Write the content of the "lanzar.sh" into /etc/rc.d/rc.local. Separated with semicolon, and in different lines.
Changing "lanzar.sh" of location.
Other things, taken from other threads that did not work for me.
My rc.loca looks like:
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
#
#Some comment
#Some comment
#Some comment
touch /var/lock/subsys/local
/bin/sh /home/fernando/Desktop/Dist/lanzar.sh
Note: I know similar questions have been asked before, but after testing many of the answers that I have found by googling with no success, I had to ask this myself.
I highly recommend that you explore the /etc/init.d directory of your server and the /etc/rc3.d directory. See how the names of the files in /etc/rc3.d are symbolically linked to the names in the /etc/init.d directory. Notice how the files in /etc/rc3.d all start with Sxx or Kxxwherexx is a number between 00 to 99.
What I am about to tell you is officially all wrong. These startup scripts are way more complicated today that what I describe, but it's a basic outline of what's going on.
In standard Unix and Linux, startup scripts were normally stored in /etc/init.d and then linked to the /etc/rcX.d directory where X stood for what was called the Init States of the server. (Yes, I'm linking to an SCO Unix page, but they were all pretty similar).
Note that Init State 3 is running in multi-user mode and that all the daemons are started. This is why I am telling you to look in /etc/rc3.d.
When the server enters that init state, it runs all of the script starting with S in alphabetical order. It runs each script with the parameter start after it. So, S01xxxx starts before S03xxx which starts before S99xxxxx.
When the server exits that init state, it runs all of the scripts that start with K in alphabetical order, and passes the stop parameter to them.
Now, Centos, Redhat, and Fedora setup handles a lot of this for you. You specify which service you depend upon, and it figures out startup and shutdown order. However, nothing is preventing you from munging a startup script and creating your own links.
By the way, speaking about Java programs that startup and shutdown... Jenkins is a Java program that's started in a very similar way as your program. Here's the /etc/init.d script I got off of Jenkins website:
#!/bin/bash
#
# Startup script for Jenkins
#
# chkconfig: - 84 16
# description: Jenkins CI server
# Source function library.
. /etc/rc.d/init.d/functions
[ -z "$JAVA_HOME" -a -x /etc/profile.d/java.sh ] && . /etc/profile.d/java.sh
JENKINS_HOME=/var/jenkins
WAR="$JENKINS_HOME/jenkins.war"
LOG="/var/log/jenkins.log"
LOCK="/var/lock/subsys/jenkins"
export JENKINS_HOME
RETVAL=0
pid_of_jenkins() {
pgrep -f "java.*jenkins"
}
start() {
[ -e "$LOG" ] && cnt=`wc -l "$LOG" | awk '{ print $1 }'` || cnt=1
echo -n $"Starting jenkins: "
cd "$JENKINS_HOME"
nohup java -jar "$WAR" --httpPort=-1 --ajp13Port=8010 --prefix=/jenkins >> "$LOG" 2>&1 &
while { pid_of_jenkins > /dev/null ; } &&
! { tail +$cnt "$LOG" | grep -q 'Winstone Servlet Engine .* running' ; } ; do
sleep 1
done
pid_of_jenkins > /dev/null
RETVAL=$?
[ $RETVAL = 0 ] && success $"$STRING" || failure $"$STRING"
echo
[ $RETVAL = 0 ] && touch "$LOCK"
}
stop() {
echo -n "Stopping jenkins: "
pid=`pid_of_jenkins`
[ -n "$pid" ] && kill $pid
RETVAL=$?
cnt=10
while [ $RETVAL = 0 -a $cnt -gt 0 ] &&
{ pid_of_jenkins > /dev/null ; } ; do
sleep 1
((cnt--))
done
[ $RETVAL = 0 ] && rm -f "$LOCK"
[ $RETVAL = 0 ] && success $"$STRING" || failure $"$STRING"
echo
}
status() {
pid=`pid_of_jenkins`
if [ -n "$pid" ]; then
echo "jenkins (pid $pid) is running..."
return 0
fi
if [ -f "$LOCK" ]; then
echo $"${base} dead but subsys locked"
return 2
fi
echo "jenkins is stopped"
return 3
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
restart)
stop
start
;;
*)
echo $"Usage: $0 {start|stop|restart|status}"
exit 1
esac
exit $RETVAL
It'll give you something to work with.
Bash: I want to run a command and pipe the results through some filter, but if the command fails, I want to return the command's error value, not the boring return value of the filter:
E.g.:
if !(cool_command | output_filter); then handle_the_error; fi
Or:
set -e
cool_command | output_filter
In either case it's the return value of cool_command that I care about -- for the 'if' condition in the first case, or to exit the script in the second case.
Is there some clean idiom for doing this?
Use the PIPESTATUS builtin variable.
From man bash:
PIPESTATUS
An array variable (see Arrays
below) containing a list of exit
status values from the processes in
the most-recently-executed foreground
pipeline (which may contain only a
single command).
If you didn't need to display the error output of the command, you could do something like
if ! echo | mysql $dbcreds mysql; then
error "Could not connect to MySQL. Did you forget to add '--db-user=' or '--db-password='?"
die "Check your credentials or ensure server is running with /etc/init.d/mysqld status"
fi
In the example, error and die are defined functions. elsewhere in the script. $dbcreds is also defined, though this is built from command line options. If there is no error generated by the command, nothing is returned. If an error occurs, text will be returned by this particular command.
Correct me if I'm wrong, but I get the impression you're really looking to do something a little more convoluted than
[ `id -u` -eq '0' ] || die "Must be run as root!"
where you actually grab the user ID prior to the if statement, and then perform the test. Doing it this way, you could then display the result if you choose. This would be
UID=`id -u`
if [ $UID -eq '0' ]; then
echo "User is root"
else
echo "User is not root"
exit 1 ##set an exit code higher than 0 if you're exiting because of an error
fi
The following script uses a fifo to filter the output in a separate process. This has the following advantages over the other answers. First, it is not bash specific. In particular it does not rely on PIPESTATUS. Second, output is not stalled until the command has completed.
$ cat >test_filter.sh <<EOF
#!/bin/sh
cmd()
{
echo $1
echo $2 >&2
return $3
}
filter()
{
while read line
do
echo "... $line"
done
}
tmpdir=$(mktemp -d)
fifo="$tmpdir"/out
mkfifo "$fifo"
filter <"$fifo" &
pid=$!
cmd a b 10 >"$fifo" 2>&1
ret=$?
wait $pid
echo exit code: $ret
rm -f "$fifo"
rmdir "$tmpdir"
EOF
$ sh ./test_filter.sh
... a
... b
exit code: 10