Could someone help me understand the following piece of code which is deciding on the start and end dates to pick data out of a db.
# Get the current time as the stop time.
#
stoptime=`date +"%Y-%m-%d %H:00"`
if test $? -ne 0
then
echo "Failed to get the date"
rm -f $1/.optpamo.pid
exit 4
fi
#
# Read the lasttime file to get the start time
#
if test -f $1/optlasttime
then
starttime=`cat $1/optlasttime`
# if the length of the chain is zero
# (lasttime is empty) It is updated properly
# (and I wait for the following hour)
if test -z "$starttime"
then
echo "Empty file lasttime"
echo $stoptime > $1/optlasttime
rm -f $1/.optpamo.pid
exit 5
fi
else
# If lasttime does not exist I create, it with the present date
# and I wait for the following hour
echo "File lasttime does not exist"
echo $stoptime > $1/optlasttime
rm -f $1/.optpamo.pid
exit 6
fi
Thanks
The script checks to see if there's a non-empty file named optlasttime in the directory specified as an argument ($1). If so, the script exits successfully (status 0). If the file doesn't exist or is empty, the current hour formatted as 2010-01-07 14:00 is written to the file, another file named .optpamo.pid is deleted from the argument directory and the script exits unsuccessfully (status 5 or 6).
This script is obviously a utility being called by some outer process, to which you need to refer for full understanding.
1.) Sets stop time to current time
2.) Checks if file $1/optlasttime exists (where $1 is passed into the script)
a.) if $1/optlasttime exists it checks the contents of the file (which it is assumed that if it does have contents it is a timestamp)
b.) if $1/optlasttime does not exist it populates the $1/optlasttime file with the stoptime.
I copied and pasted a small snippet of this into a file I called test.ksh
stoptime=`date +"%Y-%m-%d %H:00"`
if test $? -ne 0
then
echo "Failed to get the date"
rm -f $1/.optpamo.pid
exit 4
fi
Then I ran it at the commandline, like so:
zhasper#berens:~$ ksh -x ./temp.ksh
+ date '+%Y-%m-%d %H:00'
+ stoptime='2010-01-08 18:00'
+ test 0 -ne 0
The -x flag to ksh makes it print out each commandline, in full, as it executes. Comparing what you see here with the snippet of shell script above should tell you something about how ksh is interpreting the file.
If you run this over the whole file you should get a good feel for what it's doing.
To learn more, you can read man ksh, or search for ksh scripting tutorial online.
Together, these three things should help you learn a lot more than us simply telling you what the script does.
Related
I want to run a Bash script every minute (through a CRON entry) to launch a series of Python scripts in a granular (time-wise) fashion.
So far, this is the script I've made:
# set the current date
DATE=`date +%Y-%m-%d`
# set the current system time (HH:MM)
SYSTIME=`date +%H-%M`
# parse all .py script files in the 'daily' folder
for f in daily/*.py; do
if [[ -f $f ]]; then
# set the script name
SCRIPT=$(basename $f)
# get the script time
SCRTIME=`echo $SCRIPT | cut -c1-5`
# execute the script only if its intended execution time and the system time match
if [[ $SCRTIME == $SYSTIME ]]; then
# ensure the directory exists
install -v -m755 -d done/$DATE/failed
# execute the script
python3 evaluator.py $f > done/$DATE/daily-$SCRIPT.log &
# evaluate the result
if [ $? -eq 0 ]; then
# move the script to the 'done' folder
cp $f done/$DATE/daily-$SCRIPT
else
# log the failure
mv done/$DATE/daily-$SCRIPT.log done/$DATE/failed/
# move the script to the 'retry' folder for retrial
cp $f retry/daily-$SCRIPT
fi
fi
fi
done
Let's say we have the following files in a folder called daily/ (for daily execution):
daily/08-00-script-1.py
daily/08-00-script-2.py
daily/08-05-script-3.py
daily/09-20-script-4.py
The idea for granular execution is that a CRON task runs this script every minute. I fetch the system time and extract the execution time for each script and when the time matches between the system and the script file, it gets handed over to Python. So far, so good.
I know this script is not right in the sense that it gets a subshell for each script that's going to be executed but the following code is wrong as I know Bash automatically returns 0 on subshell invocation (if I did read correctly while searching on Google) and what I need is for each subshell to execute the code below so, if it fails, it gets sent to another folder (retry/) which is controlled by another Bash script running checks every 30 minutes for retrial (it's the same script as this one minus the checking part).
So, basically, I need to run this:
# evaluate the result
if [ $? -eq 0 ]; then
# move the script to the 'done' folder
cp $f done/$DATE/daily-$SCRIPT
else
# log the failure
mv done/$DATE/daily-$SCRIPT.log done/$DATE/failed/
# move the script to the 'retry' folder for retrial
cp $f retry/daily-$SCRIPT
fi
For every subshell-ed execution. How can I do this the right way?
Bash may return 0 for every sub-shell invocation, but if you wait for the result, then you will get the result (and I see no ampersand). If the python3 commands is relaying the exit code of the script, then your code will work. If your code does not catch an error, then it is the fault of python3 and you need to create error communication. Redirecting the output of stderr might be helpful, but first verify that your code does not work.
i have UNI work where i have to create a shell script that monitors a directory with a few text files that can inform you (can be manual check to be informed) if they have been modified. honestly im not sure where to start and im bad at linux and cant find anything to help. i can only use standard tools in ubuntu. any help would be great. thankyou
update - this is what i have so far and i need a way to verify that the values printed are the same after altering a file (if they are not the same then print whats files have been changed)
also sorry first time using the site trying to learn..
#!/bin/sh
echo "press 1 to check - press 2 to exit"
while :
do
read INPUT_STR
case $INPUT_STR in
1)
echo "checking sums"
md5sum Target/bob
md5sum Target/bec
md5sum Target/john
md5sum Target/mary
md5sum Target/mike
;;
2)
break
;;
*)
echo "incorrect input"
esac
done
echo "thankyou for using IDS"
You mean something like that?
#!/bin/bash
WORKDIR=/home
TARGETS=(
"$WORKDIR/bob"
"$WORKDIR/bec"
"$WORKDIR/john"
"$WORKDIR/mary"
"$WORKDIR/mike"
)
for target in "${TARGETS[#]}"; do
md5file="${target##*/}.md5"
if [[ -e "$md5file" ]]; then
md5sum --quiet -c "$md5file"
else
echo "create md5sum referenz file $md5file"
md5sum "$target"/* > "$md5file"
fi
done
At first run it creates for every directory a reference file. On the second run the directory will be compare with reference file. Modifications will be shown. When you delete one reference file, it will be created again on the next run.
explanation
# loop over the array with the targets
for target in "${TARGETS[#]}"; do
# remove all directories except the last, append .md5
# and assign it to the variable md5file
md5file="${target##*/}.md5"
# if md5file exists, use it as reference and check if
# modifications where happend
if [[ -e "$md5file" ]]; then
# use the md5file as reference and only show
# modified files, ignore ok messages
md5sum --quiet -c "$md5file"
else
# when no md5file exists
echo "create md5sum referenz file $md5file"
# use target, append /* to it and run md5sum
# redirect output to md5file (reference file for next run)
md5sum "$target"/* > "$md5file"
fi
done
I am running quantum chemical calculations by providing the command molcas -f file.input. I now have need for putting the molcas -f into a script that also tails the last 100 lines of the generated file.log, for me to quickly confirm that everything finished the way it's supposed to. So I want to run the script run.sh:
#!/bin/bash
molcas -f [here read the file.input from command line]
tail -100 [here read the file.log]
The question is how I can make the script read the argument I give, and then find on its own the output file (which has the same filename, but with a different extension).
Follow-up
Say I have a bunch of numbered files file-1, file-2, ..., file-n. I would save time if I instead of running
./run.sh file-1.input file-1.log
I run
./run.sh n n
or
./run.sh n.input n.log
assuming that the actual filename and placement of the number n is given in the script. Can that be done?
With this code:
#!/bin/bash
molcas -f "$1"
tail -100 "$2"
You will need to execute the script run.sh as follows:
./run.sh file.input file.log
to be hornest I have/had no clue over molcas, so I jumed to this side to get basic understandings.
The syntax shoould look like this ...
#!/bin/bash
# waiting for input
read -p "Enter a filename (sample.txt): " FILE
# checking for existing file
if [ -e "$FILE" ]; then
read -p "Enter a command for moculas: " CMD
else
echo "Sorry, \"${FILE}\" was not found. Exit prgramm."
exit 1
fi
# I am not sure how this command works.
# maybe you have to edit this line by your self.
molcas $FILE -f "$CMD"
# checking for programm-errors
ERRNO=$?
if [ "$ERRNO" != "" ] && [ "$ERRNO" -gt 0 ]; then
echo "Molcas returned an error. Number: ${ERRNO}"
exit 1
fi
# cuts off the fileending (For example: sample.txt gets sample)
FILENAME="${FILE%.*}"
# checking other files
ERRFILE="${FILENAME}.err"
tail -n 100 $ERRFILE
LOGFILE="${FILENAME}.log"
tail -n 100 $LOGFILE
exit 0
I would have posted more, but its not clear what to do with this data.
Hope this helps a bit.
I have the following bash script, that I launch using the terminal.
dataset_dir='/home/super/datasets/Carpets_identification/data'
dest_dir='/home/super/datasets/Carpets_identification/augmented-data'
# if dest_dir does not exist -> create it
if [ ! -d ${dest_dir} ]; then
mkdir ${dest_dir}
fi
# for all folder of the dataset
for folder in ${dataset_dir}/*; do
curr_folder="${folder##*/}"
echo "Processing $curr_folder category"
# get all files
for item in ${folder}/*; do
# if the class dir in dest_dir does not exist -> create it
if [ ! -d ${dest_dir}/${curr_folder} ]; then
mkdir ${dest_dir}/${curr_folder}
fi
# for each file
if [ -f ${item} ]; then
# echo ${item}
filename=$(basename "$item")
extension="${filename##*.}"
filename=`readlink -e ${item}`
# get a certain number of patches
for i in {1..100}
do
python cropper.py ${filename} ${i} ${dest_dir}
done
fi
done
done
Given that it needs at least an hour to process all the files.
What happens if I change the '100' with '1000' in the last for loop and launch another instance of the same script?
Will the first process count to 1000 or will continue to count to 100?
I think the file will be readonly when a bash process executes it. But you can force the change. The already running process will count to its original value, 100.
You have to take care about the results. You are writing in the same output directory and have to expect side effects.
"When you make changes to your script, you make the changes on the disk(hard disk- the permanent storage); when you execute the script, the script is loaded to your memory(RAM).
(see https://askubuntu.com/questions/484111/can-i-modify-a-bash-script-sh-file-while-it-is-running )
BUT "You'll notice that the file is being read in at 8KB increments, so Bash and other shells will likely not load a file in its entirety, rather they read them in in blocks."
(see https://unix.stackexchange.com/questions/121013/how-does-linux-deal-with-shell-scripts )
So, in your case, all your script is loaded in the RAM memory by the script interpretor, and then executed. Meaning that if you change the value, then execute it again, the first instance will still have the "old" value.
I'm working on a script that has gotten so complex I want to include an easy option to update it to the most recent version. This is my approach:
set -o errexit
SELF=$(basename $0)
UPDATE_BASE=http://something
runSelfUpdate() {
echo "Performing self-update..."
# Download new version
wget --quiet --output-document=$0.tmp $UPDATE_BASE/$SELF
# Copy over modes from old version
OCTAL_MODE=$(stat -c '%a' $0)
chmod $OCTAL_MODE $0.tmp
# Overwrite old file with new
mv $0.tmp $0
exit 0
}
The script seems to work as intended, but I'm wondering if there might be caveats with this kind of approach. I just have a hard time believing that a script can overwrite itself without any repercussions.
To be more clear, I'm wondering, if, maybe, bash would read and execute the script line-by-line and after the mv, the exit 0 could be something else from the new script. I think I remember Windows behaving like that with .bat files.
Update: My original snippet did not include set -o errexit. To my understanding, that should keep me safe from issues caused by wget.
Also, in this case, UPDATE_BASE points to a location under version control (to ease concerns).
Result: Based on the input from these answers, I constructed this revised approach:
runSelfUpdate() {
echo "Performing self-update..."
# Download new version
echo -n "Downloading latest version..."
if ! wget --quiet --output-document="$0.tmp" $UPDATE_BASE/$SELF ; then
echo "Failed: Error while trying to wget new version!"
echo "File requested: $UPDATE_BASE/$SELF"
exit 1
fi
echo "Done."
# Copy over modes from old version
OCTAL_MODE=$(stat -c '%a' $SELF)
if ! chmod $OCTAL_MODE "$0.tmp" ; then
echo "Failed: Error while trying to set mode on $0.tmp."
exit 1
fi
# Spawn update script
cat > updateScript.sh << EOF
#!/bin/bash
# Overwrite old file with new
if mv "$0.tmp" "$0"; then
echo "Done. Update complete."
rm \$0
else
echo "Failed!"
fi
EOF
echo -n "Inserting update process..."
exec /bin/bash updateScript.sh
}
(At least it doesn't try to continue running after updating itself!)
The thing that makes me nervous about your approach is that you're overwriting the current script (mv $0.tmp $0) as it's running. There are a number of reasons why this will probably work, but I wouldn't bet large amounts that it's guaranteed to work in all circumstances. I don't know of anything in POSIX or any other standard that specifies how the shell processes a file that it's executing as a script.
Here's what's probably going to happen:
You execute the script. The kernel sees the #!/bin/sh line (you didn't show it, but I presume it's there) and invokes /bin/sh with the name of your script as an argument. The shell then uses fopen(), or perhaps open() to open your script, reads from it, and starts interpreting its contents as shell commands.
For a sufficiently small script, the shell probably just reads the whole thing into memory, either explicitly or as part of the buffering done by normal file I/O. For a larger script, it might read it in chunks as it's executing. But either way, it probably only opens the file once, and keeps it open as long as it's executing.
If you remove or rename a file, the actual file is not necessarily immediately erased from disk. If there's another hard link to it, or if some process has it open, the file continues to exist, even though it may no longer be possible for another process to open it under the same name, or at all. The file is not physically deleted until the last link (directory entry) that refers to it has been removed, and no processes have it open. (Even then, its contents won't immediately be erased, but that's going beyond what's relevant here.)
And furthermore, the mv command that clobbers the script file is immediately followed by exit 0.
BUT it's at least conceivable that the shell could close the file and then re-open it by name. I can't think of any good reason for it to do so, but I know of no absolute guarantee that it won't.
And some systems tend to do stricter file locking that most Unix systems do. On Windows, for example, I suspect that the mv command would fail because a process (the shell) has the file open. Your script might fail on Cygwin. (I haven't tried it.)
So what makes me nervous is not so much the small possibility that it could fail, but the long and tenuous line of reasoning that seems to demonstrate that it will probably succeed, and the very real possibility that there's something else I haven't thought of.
My suggestion: write a second script whose one and only job is to update the first. Put the runSelfUpdate() function, or equivalent code, into that script. In your original script, use exec to invoke the update script, so that the original script is no longer running when you update it. If you want to avoid the hassle of maintaining, distributing, and installing two separate scripts. you could have the original script create the update script with a unique in /tmp; that would also solve the problem of updating the update script. (I wouldn't worry about cleaning up the autogenerated update script in /tmp; that would just reopen the same can of worms.)
Yes, but ... I would recommend you keep a more layered version of your script's history, unless the remote host can also perform version-control with histories. That being said, to respond directly to the code you have posted, see the following comments ;-)
What happens to your system when wget has a hiccup, quietly overwrites part of your working script with only a partial or otherwise corrupt copy? Your next step does a mv $0.tmp $0 so you've lost your working version. (I hope you have it in version control on the remote!)
You can check to see if wget returns any error messages
if ! wget --quiet --output-document=$0.tmp $UPDATE_BASE/$SELF ; then
echo "error on wget on $UPDATE_BASE/$SELF"
exit 1
fi
Also, Rule-of-thumb tests will help, i.e.
if (( $(wc -c < $0.tmp) >= $(wc -c < $0) )); then
mv $0.tmp $0
fi
but are hardly foolproof.
If your $0 could windup with spaces in it, better to surround all references like "$0".
To be super-bullet proof, consider checking all command returns AND that Octal_Mode has a reasonable value
OCTAL_MODE=$(stat -c '%a' $0)
case ${OCTAL_MODE:--1} in
-[1] )
printf "Error : OCTAL_MODE was empty\n"
exit 1
;;
777|775|755 ) : nothing ;;
* )
printf "Error in OCTAL_MODEs, found value=${OCTAL_MODE}\n"
exit 1
;;
esac
if ! chmod $OCTAL_MODE $0.tmp ; then
echo "error on chmod $OCTAL_MODE %0.tmp from $UPDATE_BASE/$SELF, can't continue"
exit 1
fi
I hope this helps.
Very late answer here, but as I just solved this too, I thought it might help someone to post the approach:
#!/usr/bin/env bash
#
set -fb
readonly THISDIR=$(cd "$(dirname "$0")" ; pwd)
readonly MY_NAME=$(basename "$0")
readonly FILE_TO_FETCH_URL="https://your_url_to_downloadable_file_here"
readonly EXISTING_SHELL_SCRIPT="${THISDIR}/somescript.sh"
readonly EXECUTABLE_SHELL_SCRIPT="${THISDIR}/.somescript.sh"
function get_remote_file() {
readonly REQUEST_URL=$1
readonly OUTPUT_FILENAME=$2
readonly TEMP_FILE="${THISDIR}/tmp.file"
if [ -n "$(which wget)" ]; then
$(wget -O "${TEMP_FILE}" "$REQUEST_URL" 2>&1)
if [[ $? -eq 0 ]]; then
mv "${TEMP_FILE}" "${OUTPUT_FILENAME}"
chmod 755 "${OUTPUT_FILENAME}"
else
return 1
fi
fi
}
function clean_up() {
# clean up code (if required) that has to execute every time here
}
function self_clean_up() {
rm -f "${EXECUTABLE_SHELL_SCRIPT}"
}
function update_self_and_invoke() {
get_remote_file "${FILE_TO_FETCH_URL}" "${EXECUTABLE_SHELL_SCRIPT}"
if [ $? -ne 0 ]; then
cp "${EXISTING_SHELL_SCRIPT}" "${EXECUTABLE_SHELL_SCRIPT}"
fi
exec "${EXECUTABLE_SHELL_SCRIPT}" "$#"
}
function main() {
cp "${EXECUTABLE_SHELL_SCRIPT}" "${EXISTING_SHELL_SCRIPT}"
# your code here
}
if [[ $MY_NAME = \.* ]]; then
# invoke real main program
trap "clean_up; self_clean_up" EXIT
main "$#"
else
# update myself and invoke updated version
trap clean_up EXIT
update_self_and_invoke "$#"
fi