I have a bash script that iterates over a folder of sql scripts. I want it to run intelligently when the script encounters an error. I don't want to run a step twice.
for a in $files_list; do
command 1
command 2
.
.
done
If 1,2,3,4,5 are the files to be executed and script exits at step 3 with some error. In next execution, I want to run from step 3 only. I have a variable in place for detecting at which step the script exited. I am stuck at starting the iteration from that variable.
Is it something that can be done? Any suggestion is accepted. I'm open to change my logic as well.
Didn't really spend much time but Let's say all the commands you want to execute are stored in a file test.txt
test.txt
ls
cal
du
d
date
ls
The following bash script takes two arguments filename where the commands are stored and the checkpoint from where it needs to start.
test.sh
## Function to read a file line by line
checkpoint=$2
filename=$1
checkpointReached=false
read_file_line_by_line() {
local file="$1"
while IFS= read -r line
do
if [ $checkpointReached == "false" ] && [ "$line" != "$checkpoint" ] ; then
echo "...... Finding checkpoint....... current line is $line"
fi
if [ "$line" == "$checkpoint" ]; then
echo " *************************************************
Checkpoint reached. Commands after checkpoint
*****************************************************"
checkpointReached=true
continue
fi
if $checkpointReached; then
echo " $line"
execute_command $line
fi
done < "$file"
}
function execute_command() {
local command="$1"
echo " Executing .......... $command"
if eval $command; then
echo " Command executed successfully"
else
echo " Command failed. Exiting........"
echo " Please note the new checkpoint.... $command"
exit 1
fi
}
read_file_line_by_line "$filename"
Let's run it the first time, the checkpoint is the first command i.e. ls and its expected to fail at d which is invalid command
╰─ bash test.sh text.txt "ls"
*************************************************
Checkpoint reached. Commands after checkpoint
*****************************************************
Executing .......... ls
test.sh test.yaml text.txt
Command executed successfully
du
Executing .......... du
24 .
Command executed successfully
d
Executing .......... d
test.sh: line 33: d: command not found
Command failed. Exiting........
Please note the new checkpoint.... d
Now we know the checkpoint. So we pass it along. Assuming that d is fixed now, we change it with echo for the sake of the script. In your case the script d would be fixed.
╰─ bash test.sh text.txt "echo"
...... Finding checkpoint....... current line is ls
...... Finding checkpoint....... current line is du
*************************************************
Checkpoint reached. Commands after checkpoint
*****************************************************
Executing .......... echo
Command executed successfully
date
Executing .......... date
Mon Sep 12 14:50:44 +04 2022
Command executed successfully
Let me know if this helps.
Related
Inside shell script it is picking some files from UNIX directory. Sometimes when any of these file missed then throw error & shell script got failed. I need to add Exceptional Handling in shell script. Here is the below shell script where it is picking files from directory.
#!/bin/ksh
..........
while read file
do
upd_date=$((`date +%s`-`stat -c %Y ${file}`))
file_nm=`basename "${file}"`
if [[ `lsof | grep "${file_nm}"` != '' || $upd_date -lt 60 ]];
then
log_info "Waiting for the file ${file} to complete transfer/skip"
else
log_info "Data file found ${file}"
fi
done<${TEMP}/filepath
...............
exit 0
In the line upd_date=.. throwing error whenever file missed & shell script got failed. So I need to add exception handling if file missed then it will show in log & script will execute successfully.
Use continue in between while loop. continue will skip the current iteration in for, while and until loop.
#!/bin/ksh
..........
while read file
do
[ -f "$file" ] || continue;
upd_date=$((`date +%s`-`stat -c %Y ${file}`))
file_nm=`basename "${file}"`
if [[ `lsof | grep "${file_nm}"` != '' || $upd_date -lt 60 ]];
then
log_info "Waiting for the file ${file} to complete transfer/skip"
else
log_info "Data file found ${file}"
fi
done<${TEMP}/filepath
...............
exit 0
I am using gcloud commands to deploy simple VMs. I am using startup script to configure all the required packages on the machine. Our packages are fetched from nexus. Sometimes our changes or nexus network issues result in failure of startup script. I configured a block of code to validate startup script return code. I run gcloud create command and provide startup script, the code block to validate startup script output looks like this..
echo "Step 2: sleep till startup script finishes"
## block to check return code of the VM startup script
MAX_WAIT_TIME=1200
WAIT_TIME=0
RC_CODE=""
echo "[INFO] waiting for startup-script return code"
while [ -z $RC_CODE ] && [ $WAIT_TIME -le $MAX_WAIT_TIME ]
do
RC_CODE=$(gcloud compute instances get-serial-port-output gce-$GCP_ZONE-d-$APP-dev \
--project=$GCP_PROJECT_ID \
--zone=$GCP_ZONE | awk '/^[0-9]{4}/ && /google_metadata_script_runner/ && /startup-script-url exit status/ {sub("\r", "", $NF); print $NF}')
if [[ -z $RC_CODE ]] ; then
echo -n "."
WAIT_TIME=$((WAIT_TIME+10))
sleep 10
else
if [[ $RC_CODE -eq 0 ]] ; then
echo -e "\n[INFO] startup script completed with return code $RC_CODE."
break
else
echo -e "\n[INFO] Startup script completed with return code $RC_CODE."
exit $RC_CODE
fi
fi
done
# to check timeout scenario
if [[ -z $RC_CODE ]]
then
echo "[INFO] Startup script timed out after $((MAX_WAIT_TIME/60))."
echo "[INFO] Startup script completed with return code 1."
exit 1
fi
My output looks like this,
+ ./sh/create-dev-vm-app.sh
Deleted [https://www.googleapis.com/compute/v1/projects/project-name/zones/europe-west1-c/instances/gce-europe-west1-c-d-myapp-dev].
Created [https://www.googleapis.com/compute/v1/projects/project-name/zones/europe-west1-c/instances/gce-europe-west1-c-d-myapp-dev].
waiting for startup-script return code
Specify --start=2473 in the next get-serial-port-output invocation to get only the new output starting from here.
.
Specify --start=32157 in the next get-serial-port-output invocation to get only the new output starting from here.
.
Specify --start=37602 in the next get-serial-port-output invocation to get only the new output starting from here.```
.
Startup script completed with return code 0.
How can I suppress the these 'Specify' lines from appearing on o/p screen? AND/OR print all startup-script messages received from get-serial-port-output output after I receive the return code.
Swallow standard error to avoid the extraneous output:
gcloud ... 2>/dev/null | awk ...
I am trying to find whether a particular process ID is running. If I type on command line:
ps -x|cut -c1-5|grep 7032
I get 7032 as the answer. But if I put into shell script:
read pid_found < "ps -x|cut -c1-5|grep $old_pid"
I get the error message:
./test.sh: line 11: ps -x|cut -c1-5|grep 7032: No such file or directory
Note that the expansion of $old_pid is a not a problem -- the error message correctly says grep 7032. Every similar question on StackOverflow has answers suggesting to use
variable_name=$(command)
but this does not work for me either, with any command. At least
read variable_name < "command"
works elsewhere in the script
Edit: I managed to get it to work by removing all pipes within quotation marks and putting all intermediate values into files. It's ugly, but it works:
#!/bin/bash
cd /home/automan/vms_feed
read -d $'\x04' old_pid < "OldPID"
ps -x|cut -c1-5 > "Processes"
grep $old_pid "Processes"|wc -c > "PID_found"
read -d $'\x04' pid_found < "PID_found"
echo "Running: test, old_pid =" $old_pid
if [ $pid_found -gt 0 ]
then
echo "Running: test, old pid found" $pid_found
else
echo "Running: test, old pid not found"
fi
echo $$ > OldPID
echo "Finished: test"
I have below lines of code in my bash script which reads a file line by line and compares its float value with the given threshold.
This code works just fine on local (mac system) and returns the correct output.
........
........
cat perf/perftestresult.txt | sed -n "s/^.*throughput]\\s*\\(.*,\\s\\)\\(.*\\)/\\2/p" >
$filename
buildFailed=false
while IFS= read -r line
do
if [ 1 -eq "$( echo "${line} < ${threshold}" | bc )" ]
then
buildFailed=true
break
else
continue
fi
done < "$filename"
echo $buildFailed
.......
.......
However, the same script returns below error when run in Jenkins pipeline.
line 141: 1: not found
12:53:51 sh: out of range
Line 141 in script is the If condition
Can someone please help me fix this issue?
My professor gave me this confusing assignment :
write a program/script which asks the user for a filename to get input
from, a filename to send errors and a filename to write output if the
filename for output and errors are the same, use the proper redirects
input file will have instructions for your program such as: CHANGE:
this command will tell your program to change the redirects like this:
CHANGE STDIN newfilename CHANGE STOUT newoutfile CHANGE STDERR
newerorfile STOP: stop the program and exit STOP
any other input should be written to the output file Write the number
of lines copied to standard error
your error file might look like this:
1
2
ERROR: Input file not found
filename
CHANGE: redirecting output to filename
3
4
CHANGE: redirecting errors to
**at this point the new error file would start with
5
5
6
STOP requested
**
here are some sample files:
> fileio.test.1
line 1 of first input file
line 2 of first inputt file
CHANGE STDIN stdin.1
since input has changed this line should never
get read EXIT
> stdin.1
line 1 of stdin.1
line 2 of stdin.1
CHANGE STDERR stderr.1
line 3 of stdin.1
CHANGE STDOUT stdout.1
line 4 of stdin.1
CHANGE STDOUT stdout.2
line 5 of stdin.1
CHANGE STDIN stdin.2
since input has changed this line should never get read EXIT
> stdin.2
this should go wherever it's supposed to
EXIT
this line shouldn't be read since our program should have exited by
now
However, I am not quite sure how to interpret these directions. As a stab in the dark, I've done:
#!/bin/bash
read -p "Provide filename to receive input from: " input
read -p "Provide filename to send errors to: " errorFile
read -p "Provide filename to write output to: " writeFile
#set -x
if [ -z $input ]; then
echo "Input file needed."
exit=1
elif [ -z $errorFile ] || [ -z $writeFile ]; then
echo "Supply the correct number of filenames."
exit=1
elif [ $errorFile == $writeFile ]; then
exec 2> $errorFile
exec >&2
else
exec 2> $errorFile
exec 1> $writeFile
fi
while read -r line
do
eval $(echo "$line")
done< $input
Edit: For a function to interpret the input, I was thinking I could match for the specific commands in the testfiles and then trigger the corresponding bash command. Anything that doesn't match would be echo'd and directed to its appropriate file.
ChangeFD () {
if [[ "CHANGE STDE" == ${line:0:11} ]]; then
exec $(echo "2>${line:14}")
elif [[ "CHANGE STDO" == ${line:0:11} ]]; then
exec $(echo "1>${line:14}")
elif [[ "CHANGE STDI" == ${line:0:11} ]]; then
exec $(echo "0<${line:13}")
else
echo $1
fi
}
to count the number of lines (and output that count at regular intervals) you will need to create another variable.
Before your while loop
count=0
and within the while loop (probably just before the 'done < $input' line)
# increment the counter
count=$(( count + 1 ))
# output the counter var to stderr (wherever it is pointing)
echo $count >>&2
exec $(echo "0<${line:13}")
This doesn't work, because redirection operators are not recognized in the replacement of $(…); just use
exec <${line:13}
instead.