I have been trying to create a trap in a script to basically create some logs of a script that has been running in the background.
Whenever I introduce a for loop in the trap, the script stops doing what it is supposed to do:
trap 'terminate' 10
...
write_log(){
local target=$1
local file="/tmp/"$target"_log.txt"
local lines=$(cat /tmp/"$target"_log.txt | wc -l)
printf "Log for $target\n" >> "log.txt" # This line is printed
for ((i=1;i<=$lines;i++)); # Nothing in this loop happens
do
local start_date=$(date -d "$(sed -n ""$i"p") $file | cut -f1")
local end_date=$(date -d "$sed -n ""$i"p") $file | cut -f2")
printf "Logged in $start_date, logged out $end_date" > "log.txt"
done
}
terminate(){
for target
do
echo "In the for loop!"
end_session "$target"
write_log "$target"
done
exit 0
}
When I run my script in the background and kill it with
kill -10 (process_id)
the script stops, and starts doing the cleanup, until the point where it finds a for loop. When I remove the for loop in terminate() and instead do individual calls to end_session() and write_log(), end_session() works just fine, and write_log() works fine--until it reaches the for loop.
I am probably missing something basic, but I have looked at this for a while now and can't seem to figure out what is happening. Is there any limitation to for loops in traps?
No arguments are passed to terminate when it is invoked by the trap, so its loop executes zero times (because for target; do …; done is a shorthand for for target in "$#"; do …; done, and in a function, "$#" is the list of arguments to the function, not to the shell script as a whole).
If that's not what you want to have happen, you have to arrange to pass the relevant arguments to terminate in the trap. For example, you could pass all the arguments to the script via a global array:
args=( "$#" )
and inside terminate:
for target in "${args[#]}"
However, what's best depends on what you want to achieve.
The function is hanging because the parentheses are messed up in the date commands. Try this:
local start_date=$(date -d "$(sed -n ${i}p "$file" | cut -f1)")
local end_date=$(date -d "$(sed -n ${i}p "$file" | cut -f2)")
Related
So I have this shell script that I think should run a given number of times, sleep then resume, and output the results to a log file
#!/bin/bash
log=/path/to/file/info.log
a=$(COMMAND1 | cut -d : -f 2)
b=$(COMMAND2 | grep VALUE| cut -c 7,8)
for i in {1..4}
do
echo "Test" $i >> $log
date >> $log
echo $a >> $log
echo "$((-113 + (($b * 2)))) VALUE" >> $log
sleep 60
done
When I run ps -ef | grep scriptname.sh it seems the script does run. Executes once then the PID is gone as if the run has completed.
I have tested the script and know that it is running and capturing the data I want. But I do not understand why its not incrementing and not sure why its ending earlier than expected.
info.log output sample
Test {1..4}
DATE IN UTC
EXPECTED VALUE OF a
EXPECTED VALUE OF b
Note that the output is literally "Test {1..4}" not "Test 1" "Test 2" Test 3" and so on, as I would expect.
I have run the script as ./scriptname.sh & and as /path/to/file/scriptname.sh &
I have read that there is a difference in running the script with sh and bash though I dont fully understand what effect that would have on the script. I am not a software person at all.
I have tried to run the script with nohup to keep it running in the background if I close the terminal. I also thought the & in the command was supposed to keep the script running in the background. Still it seems the script does not continue to run
I previously asked this question and it was closed, citing that it was similar to a post about the difference between sh and bash...but thats not my main question.
also echo "$BASH_VERSION" returns nothing, a blank line. echo "$-" returns smi, and I have no idea what that means. but bash --version returns:
BusyBox v1.17.1 (2019-11-26 10:41:00 PST) built-in shell (ash)
Enter 'help' for a list of built-in commands.
So my questions are:
If running the script with sh - is that done with ./scriptname.sh & and running the script with bash is /path/to/file/scriptname.sh &...and if so what effect does that have on how the script code is processed? that is - is using sh or bash. I do not fully understand the difference between the two
why does the script not continue to run when I close the terminal? This is my big concern. I would like to run this script hourly for a set period of time. Every time I try something and come back I get one instance in the log.
Neither brace expansion nor seq are part of the POSIX specification. Use a while loop.
log=/path/to/file/info.log
a=$(COMMAND1 | cut -d : -f 2)
b=$(COMMAND2 | grep VALUE| cut -c 7,8)
i=1
while [ "$i" -le 4 ]; do
printf 'Test %s\n' "$i"
date
printf '%s\n' "$a"
printf '%s\n' "$((-113 + (($b * 2)))) VALUE"
sleep 60
i=$((i+1))
done >> "$log"
(I suspect that you want to move the assignments to a and b inside the loop as well; right now, you are simply writing identical files to the log at each iteration.)
Is there a way to speed up the below shell script? It's taking me a good 40 mins to update about 150000 files everyday. Sure, given the volume of files to create & update, this may be acceptable. I don't deny that. However, if there is a much more efficient way to write this or re-write the logic entirely, I'm open to it. Please I'm looking for some help
#!/bin/bash
DATA_FILE_SOURCE="<path_to_source_data/${1}"
DATA_FILE_DEST="<path_to_dest>"
for fname in $(ls -1 "${DATA_FILE_SOURCE}")
do
for line in $(cat "${DATA_FILE_SOURCE}"/"${fname}")
do
FILE_TO_WRITE_TO=$(echo "${line}" | awk -F',' '{print $1"."$2".daily.csv"}')
CONTENT_TO_WRITE=$(echo "${line}" | cut -d, -f3-)
if [[ ! -f "${DATA_FILE_DEST}"/"${FILE_TO_WRITE_TO}" ]]
then
echo "${CONTENT_TO_WRITE}" >> "${DATA_FILE_DEST}"/"${FILE_TO_WRITE_TO}"
else
if ! grep -Fxq "${CONTENT_TO_WRITE}" "${DATA_FILE_DEST}"/"${FILE_TO_WRITE_TO}"
then
sed -i "/${1}/d" "${DATA_FILE_DEST}"/"${FILE_TO_WRITE_TO}"
"${DATA_FILE_DEST}"/"${FILE_TO_WRITE_TO}"
echo "${CONTENT_TO_WRITE}" >> "${DATA_FILE_DEST}"/"${FILE_TO_WRITE_TO}"
fi
fi
done
done
There are still parts of your published script that are unclear like the sed command. Although I rewrote it with saner practices and much less external calls witch should really speed it up.
#!/usr/bin/env sh
DATA_FILE_SOURCE="<path_to_source_data/$1"
DATA_FILE_DEST="<path_to_dest>"
for fname in "$DATA_FILE_SOURCE/"*; do
while IFS=, read -r a b content || [ "$a" ]; do
destfile="$DATA_FILE_DEST/$a.$b.daily.csv"
if grep -Fxq "$content" "$destfile"; then
sed -i "/$1/d" "$destfile"
fi
printf '%s\n' "$content" >>"$destfile"
done < "$fname"
done
Make it parallel (as much as you can).
#!/bin/bash
set -e -o pipefail
declare -ir MAX_PARALLELISM=20 # pick a limit
declare -i pid
declare -a pids
# ...
for fname in "${DATA_FILE_SOURCE}/"*; do
if ((${#pids[#]} >= MAX_PARALLELISM)); then
wait -p pid -n || echo "${pids[pid]} failed with ${?}" 1>&2
unset 'pids[pid]'
fi
while IFS= read -r line; do
FILE_TO_WRITE_TO="..."
# ...
done < "${fname}" & # forking here
pids[$!]="${fname}"
done
for pid in "${!pids[#]}"; do
wait -n "$((pid))" || echo "${pids[pid]} failed with ${?}" 1>&2
done
Here’s a directly runnable skeleton showing how the harness above works (with 36 items to process and 20 parallel processes at most):
#!/bin/bash
set -e -o pipefail
declare -ir MAX_PARALLELISM=20 # pick a limit
declare -i pid
declare -a pids
do_something_and_maybe_fail() {
sleep $((RANDOM % 10))
return $((RANDOM % 2 * 5))
}
for fname in some_name_{a..f}{0..5}.txt; do # 36 items
if ((${#pids[#]} >= MAX_PARALLELISM)); then
wait -p pid -n || echo "${pids[pid]} failed with ${?}" 1>&2
unset 'pids[pid]'
fi
do_something_and_maybe_fail & # forking here
pids[$!]="${fname}"
echo "${#pids[#]} running" 1>&2
done
for pid in "${!pids[#]}"; do
wait -n "$((pid))" || echo "${pids[pid]} failed with ${?}" 1>&2
done
Strictly avoid external processes (such as awk, grep and cut) when processing one-liners for each line. fork()ing is extremely inefficient in comparison to:
Running one single awk / grep / cut process on an entire input file (to preprocess all lines at once for easier processing in bash) and feeding the whole output into (e.g.) a bash loop.
Using Bash expansions instead, where feasible, e.g. "${line/,/.}" and other tricks from the EXPANSION section of the man bash page, without fork()ing any further processes.
Off-topic side notes:
ls -1 is unnecessary. First, ls won’t write multiple columns unless the output is a terminal, so a plain ls would do. Second, bash expansions are usually a cleaner and more efficient choice. (You can use nullglob to correctly handle empty directories / “no match” cases.)
Looping over the output from cat is a (less common) useless use of cat case. Feed the file into a loop in bash instead and read it line by line. (This also gives you more line format flexibility.)
So I am trying to create a script that will wait for a certain string in the output from the command that's starting another script.
I am running into a problem where my script will not move past this line of code
$(source path/to/script/LOOPER >> /tmp/looplogger.txt)
I have tried almost every variation I can think of for this line
ie. (./LOOPER& >> /tmp/looplogger.txt)
bash /path/to/script/LOOPER 2>1& /tmp/looplogger.txt etc.
For Some Reason I cannot get it to run in a subshell and have the rest of the script go about its day.
I am trying to run a script from another script and access it's output then parse line by line until a certain string is found
Then once that string is found my script would kill said script (which I am aware if it is sourced then then the parent script would terminate as well).
The script that is starting looper then trying to kill it-
#!/bin/bash
# deleting contents of .txt
echo "" > /tmp/looplogger.txt
#Code cannot get past this command
$(source "/usr/bin/gcti/LOOPER" >> /tmp/ifstester.txt)
while [[ $(tail -1 /tmp/looplogger.txt) != "Kill me" ]]; do
sleep 1
echo ' in loop ' >> /tmp/looplogger.txt
done >> /tmp/looplogger.txt
echo 'Out of loop' >> looplogger.txt
#This kill command works as intended
kill -9 $(ps -ef | grep LOOPER | grep -v grep | awk '{print $2}')
echo "Looper was killed" > /tmp/looplogger.txt
I have tried using while IFS= read -r as well. for the above script. But I find it's syntax alittle confusing.
Looper Script -
./LOOPER
#!/bin/bash
# Script to test with scripts that kill & start processes
let i=0
# Infinite While Loop
while :
do
i=$((i+1))
until [ $i -gt 10 ]
do
echo "I am looping :)"
sleep 1
((i=i+1))
done
echo "Kill me"
sleep 1
done
Sorry for my very wordy question.
Log files is written line by line by underwater drones on a server. TWhen at surface, the drones speak slowly to the server (say ~200o/s on a phone line which is not stable) and only from time to time (say every ~6h). Depending on the messages, I have to execute commands on the server while the drones are online and when they hang up other commands. Other processes may be looking at the same files with similar tasks.
A lot can be found on this website on somewhat similar problems but the solution I have built on is still unsatisfactory. Presently I'm doing this with bash
while logfile_drone=`inotifywait -e create --format '%f' log_directory`; do
logfile=log_directory/${logfile_drone}
while action=`inotifywait -q -t 120 -e modify -e close --format '%e' ${logfile} ` ; do
exidCode=$?
lastLine=$( tail -n2 ${logFile} | head -n1 ) # because with tail -n1 I can got only part of the line. this happens quite often
match =$( # match set to true if lastLine matches some pattern )
if [[ $action == 'MODIFY' ]] && $match ; then # do something ; fi
if [[ $( echo $action | cut -c1-5 ) == 'CLOSE' ]] ; then
# do something
break
fi
if [[ $exitCode -eq 2 ]] ; then break ; fi
done
# do something after the drone has hang up
done # wait for a new call from the same or another drone
The main problems are :
the second inotify misses lines, may be because of the other processes looking at the same file.
the way I catch the time out doesn't seem to work.
I can't monitor 2 drones simultaneously.
Basically the code works more or less but isn't very robust. I wonder if problem 3 can be managed by putting the second while loop in a function which is put in background when called. Finally, I wonder if a higher level language (I'm familiar with php which has a PECL extension for inotify) would not do this much better. However, I imagine that php will not solve problem 3 better than than bash.
Here is the code where I'm facing the problem of abrupt exit from the while loop, implemented according to Philippe's answer, which works fine otherwise:
while read -r action ; do
...
resume=$( grep -e 'RESUMING MISSION' <<< $lastLine )
if [ -n "$resume" ] ; then
ssh user#another_server "/usr/bin/php /path_to_matlab_command/matlabCmd.php --drone=${vehicle}" &
fi
if [ $( echo $action | cut -c1-5 ) == 'CLOSE' ] ; then ... ; sigKill=true ; fi
...
if $sigKill ; then break; fi
done < <(inotifywait -q -m -e modify -e close_write --format '%e' ${logFile})
When I comment the line with ssh the script can exit properly with a break triggered by CLOSE, otherwise the while loop finishes abruptly after the ssh command. The ssh is put in background because the matlab code runs for long time.
monitor mode (-m) of inotifywait may serve better here :
inotifywait -m -q -e create -e modify -e close log_directory |\
while read -r dir action file; do
...
done
monitor mode (-m) does not buffer, it just print all events to standard output.
To preserve the variables :
while read -r dir action file; do
echo $dir $action $file
done < <(inotifywait -m -q -e create -e modify -e close log_directory)
echo "End of script"
So I've looked up other questions and answers for this and as you can imagine, there are lots of ways to find this. However, my situation is kind of different.
I'm able to check whether a bash script is already running or not and I want to kill the script if it's already running.
The problem is that with the below code, -since I'm running this within the same script- the script kills itself too because it sees a script already running.
result=`ps aux | grep -i "myscript.sh" | grep -v "grep" | wc -l`
if [ $result -ge 1 ]
then
echo "script is running"
else
echo "script is not running"
fi
So how can I check if a script is already running besides it's own self and kill itself if there's another instance of the same script is running, else, continue without killing itself.
I thought I could combine the above code with $$ command to find the script's own PID and differentiate them this way but I'm not sure how to do that.
Also a side note, my script can be run multiple times at the same time within the same machine but with different arguments and that's fine. I only need to identify if script is already running with the same arguments.
pid=$(pgrep myscript.sh | grep -x -v $$)
# filter non-existent pids
pid=$(<<<"$pid" xargs -n1 sh -c 'kill -0 "$1" 2>/dev/null && echo "$1"' --)
if [ -n "$pid" ]; then
echo "Other script is running with pid $pid"
echo "Killing him!"
kill $pid
fi
pgrep lists the pids that match the name myscript.sh. From the list we filter current $$ shell with grep -v. It the result is non-empty, then you could kill the other pid.
Without the xargs, it would work, but the pgrep myscript.sh will pick up the temporary pid created for command substitution or the pipe. So the pid will never be empty and the kill will always execute complaining about the non-existent process. To do that, for each pid in pids, I check if the pid exists with kill -0. If it does, then it is outputted, effectively filtering all nonexistent pids.
You could also use a normal for loop to filter the pids:
# filter non-existent pids
pid=$(
for i in $pid; do
if kill -0 "$i" 2>/dev/null; then
echo "$i"
fi
done
)
Alternatively, you could use flock to lock the file and use lsof to list current open files with filtering the current one. As it is now, I think it will kill also editors that are editing the file and such. I believe the lsof output could be better filtered to accommodate this.
if [ "${FLOCKER}" != "$0" ]; then
pids=$(lsof -p "^$$" -- ./myscript.sh | awk 'NR>1{print $2}')
if [ -n "$pids" ]; then
echo "Other processes with $(echo $pids) found. Killing them"
kill $pids
fi
exec env FLOCKER="$0" flock -en "$0" "$0" "$#"
fi
I would go with either of 2 ways to solve this problem.
1st solution: Create a watchdog file lets say a .lck file kind of on a location before starting the script's execution(Make sure we use trap etc commands in case script is aborted so that .lck file should be removed) AND remove it once execution of script is completed successfully.
Example script for 1st solution: This is just an example a test one. We need to take care of interruptions in the script, lets say script got interrupted by a command or etc then we could use trap in it too, since at that time it would have not been completed but you may need to kick it off again(since last time it was not completed).
cat file.ksh
#!/bin/bash
PWD=`pwd`
watchdog_file="$PWD/script.lck"
if [[ -f "$watchdog_file" ]]
then
echo "Please wait script is still running, exiting from script now.."
exit 1;
else
touch $watchdog_file
fi
while true
do
echo "singh" > test1
done
if [[ -f "$watchdog_file" ]]
then
rm "$watchdog_file"
fi
2nd solution: Take pid of current running shell using $$ save it in a file. Then check if that process is still running come out of script if NOT running then move on to run statements in script.