I have a list of IP addresses and I have to run a command for every single IP address.
I did it with this code:
array=($(</tmp/ip-addresses.txt))
for i in "${array[#]}"; do
./command start $i &
done
Now, the list of IP addresses is constantly refreshed every 2 minutes and I need to kill every command that is no longer with the new IP addresses.
Practically, the command needs to be executed again every 2 minutes with the refreshed IP addresses and all the another old IP needs to be killed.
How can I do that?
A simple workaround: (Not tested)
sleep_delay=120 # 2 mins
while true; do
(
array=($(</tmp/ip-addresses.txt))
for i in "${array[#]}"; do
./command start $i &
done
sleep $(( sleep_delay + 2 )) # 2 can be any number >0
) & PPID=$!
sleep $sleep_delay
pkill -9 -p $PPID
done
Note: I have not optimized your code, just added a wrapper around your code.
EDIT:
Edited code to satisfy requirement that the old processes should not be killed, if the IP is still same.
NOTE: I haven't tested the code myself, so be careful while using the kill command. You can test by putting echo before the kill statement. If it works well, you can use the script...
declare -A pid_array
while true; do
array=($(</tmp/ip-addresses.txt))
for i in `printf "%s\n" ${!pid_array[#]} | grep -v -f <(printf "%s\n" ${array[#]})`; do
kill -9 ${pid_array[$i]} # please try to use better signal to kill, than SIGKILL
unset pid_array[$i]
done
for i in "${array[#]}"; do
if [ -z "${pid_array[$i]}" ]; then
./command start $i & pid_array[$i]=$!
fi
done
sleep 120
done
Your script would be changed like:
#!/bin/bash
GAP=120
while :
do
#Do your stuff here
sleep $GAP
done
exit 0
After two minutes it would read from refreshed file
Related
right now I'm using this script for a program:
export FREESURFER_HOME=$HOME/freesurfer
source $FREESURFER_HOME/SetUpFreeSurfer.sh
cd /home/ubuntu/fastsurfer
datadir=/home/ubuntu/moya/data
fastsurferdir=/home/ubuntu/moya/output
mkdir -p $fastsurferdir/logs # create log dir for storing nohup output log (optional)
while read p ; do
echo $p
nohup ./run_fastsurfer.sh --t1 $datadir/$p/orig.nii \
--parallel --threads 16 --sid $p --sd $fastsurferdir > $fastsurferdir/logs/out-${p}.log &
sleep 3600s
done < /home/ubuntu/moya/data/subjects-list.txt
Instead of using sleep 3600s, as the program needs around an hour, I'd like to use wait until all processes (several PIDS) are finished.
If this is the right way, can you tell me how to do that?
BR Alex
wait will wait for all background processes to finish (see help wait). So all you need is to run wait after creating all of the background processes.
This may be more than what you are asking for but I figured I would provide some methods for controlling the number of threads you want to have running at once. I find that I always want to limit the number for various reasons.
Explaination
The following will limit concurrent threads to max_threads running at one time. I am also using the main design pattern so we have a main that runs the script with a function run_jobs that handles the calling and waiting. I read all of $p into an array, then traverse that array as we launch threads. It will either launch a thread up to 4 or wait 5 seconds, once there are at least one less than four it will start another thread. When finished it waits for any remaining to be done. If you want something more simplistic I can do that as well.
#!/usr/bin/env bash
export FREESURFER_HOME=$HOME/freesurfer
source $FREESURFER_HOME/SetUpFreeSurfer.sh
typeset max_threads=4
typeset subjects_list="/home/ubuntu/moya/data/subjects-list.txt"
typeset subjectsArray
run_jobs() {
local child="$$"
local num_children=0
local i=0
while [[ 1 ]] ; do
num_children=$(ps --no-headers -o pid --ppid=$child | wc -w) ; ((num_children-=1))
echo "Children: $num_children"
if [[ ${num_children} -lt ${max_threads} ]] ;then
if [ $i -lt ${#subjectsArray[#]} ] ;then
((i+=1))
# RUN COMMAND HERE &
./run_fastsurfer.sh --t1 $datadir/${subjectsArray[$i]}/orig.nii \
--parallel --threads 16 --sid ${subjectsArray[$i]} --sd $fastsurferdir
fi
fi
sleep 10
done
wait
}
main() {
cd /home/ubuntu/fastsurfer
datadir=/home/ubuntu/moya/data
fastsurferdir=/home/ubuntu/moya/output
mkdir -p $fastsurferdir/logs # create log dir for storing nohup output log (optional)
mapfile -t subjectsArray < ${subjects_list}
run_jobs
}
main
Note: I did not run this code since you have not provided enough information to actually do so.
I have two bash scripts:
a.sh:
echo "running"
doit=true
if [ $doit = true ];then
./b.sh &
fi
some-long-operation-binary
echo "done"
b.sh:
for i in {0..50}; do
echo "counting";
sleep 1;
done
I get this output:
> ./a.sh
running
counting
Why do I only see the first "counting" from b.sh and then nothing anymore? (Currently some-long-operation-binary just sleep 5 for this example). I first thought that due to setting b.sh in the background, its STDOUT is lost, but why do I see the first output? More importantly: is b.sh still running and doing its thing (its iteration)?
For context:
b.sh is going to poll a service provided by some-long-operation-binary, which is only available after some time the latter has run, and when ready, would write its content to a file.
Apologies if this is just rubbish, it's a bit late...
You should add #!/bin/bash or the like to b.sh that uses a Bash-like expansion, to make sure Bash is actually running the script. Otherwise there may be (indeed) only one loop iteration happening.
When you start a background process, it is usually a good practice to kill it and wait for it, no matter which way the script exits.
#!/bin/bash
set -e -o pipefail
declare -i show_counter=1
counter() {
local -i i
for ((i = 0;; ++i)); do
echo "counting $((i))"
sleep 1
done
}
echo starting
if ((show_counter)); then
counter &
declare -i counter_pid="${!}"
trap 'kill "${counter_pid}"
wait -n "${counter_pid}" || :
echo terminating' EXIT
fi
sleep 10 # long-running process
So I've looked up other questions and answers for this and as you can imagine, there are lots of ways to find this. However, my situation is kind of different.
I'm able to check whether a bash script is already running or not and I want to kill the script if it's already running.
The problem is that with the below code, -since I'm running this within the same script- the script kills itself too because it sees a script already running.
result=`ps aux | grep -i "myscript.sh" | grep -v "grep" | wc -l`
if [ $result -ge 1 ]
then
echo "script is running"
else
echo "script is not running"
fi
So how can I check if a script is already running besides it's own self and kill itself if there's another instance of the same script is running, else, continue without killing itself.
I thought I could combine the above code with $$ command to find the script's own PID and differentiate them this way but I'm not sure how to do that.
Also a side note, my script can be run multiple times at the same time within the same machine but with different arguments and that's fine. I only need to identify if script is already running with the same arguments.
pid=$(pgrep myscript.sh | grep -x -v $$)
# filter non-existent pids
pid=$(<<<"$pid" xargs -n1 sh -c 'kill -0 "$1" 2>/dev/null && echo "$1"' --)
if [ -n "$pid" ]; then
echo "Other script is running with pid $pid"
echo "Killing him!"
kill $pid
fi
pgrep lists the pids that match the name myscript.sh. From the list we filter current $$ shell with grep -v. It the result is non-empty, then you could kill the other pid.
Without the xargs, it would work, but the pgrep myscript.sh will pick up the temporary pid created for command substitution or the pipe. So the pid will never be empty and the kill will always execute complaining about the non-existent process. To do that, for each pid in pids, I check if the pid exists with kill -0. If it does, then it is outputted, effectively filtering all nonexistent pids.
You could also use a normal for loop to filter the pids:
# filter non-existent pids
pid=$(
for i in $pid; do
if kill -0 "$i" 2>/dev/null; then
echo "$i"
fi
done
)
Alternatively, you could use flock to lock the file and use lsof to list current open files with filtering the current one. As it is now, I think it will kill also editors that are editing the file and such. I believe the lsof output could be better filtered to accommodate this.
if [ "${FLOCKER}" != "$0" ]; then
pids=$(lsof -p "^$$" -- ./myscript.sh | awk 'NR>1{print $2}')
if [ -n "$pids" ]; then
echo "Other processes with $(echo $pids) found. Killing them"
kill $pids
fi
exec env FLOCKER="$0" flock -en "$0" "$0" "$#"
fi
I would go with either of 2 ways to solve this problem.
1st solution: Create a watchdog file lets say a .lck file kind of on a location before starting the script's execution(Make sure we use trap etc commands in case script is aborted so that .lck file should be removed) AND remove it once execution of script is completed successfully.
Example script for 1st solution: This is just an example a test one. We need to take care of interruptions in the script, lets say script got interrupted by a command or etc then we could use trap in it too, since at that time it would have not been completed but you may need to kick it off again(since last time it was not completed).
cat file.ksh
#!/bin/bash
PWD=`pwd`
watchdog_file="$PWD/script.lck"
if [[ -f "$watchdog_file" ]]
then
echo "Please wait script is still running, exiting from script now.."
exit 1;
else
touch $watchdog_file
fi
while true
do
echo "singh" > test1
done
if [[ -f "$watchdog_file" ]]
then
rm "$watchdog_file"
fi
2nd solution: Take pid of current running shell using $$ save it in a file. Then check if that process is still running come out of script if NOT running then move on to run statements in script.
I need to know an another process pid which is executed as a command in the bash script like a nmap scan etc as an when the process is created I can display a bar or a spinner
#!/bin/sh
G='\033[0;32m'
B='\033[0m'
${G}Enter IP TAIL LIKE 0.1 OR 1.1${B}"
read v
${G}ENTER Device NAME ${B}"
read k
nmap -A -Pn -sV 192.168.$v -oN /tmp/op
//here i want a while loop for displaying a spinner
edit code is updated Thts the code in case tht was not visible properly ^^
As the pid dies when its completed I can use it as a count or a flag or something similar to end the spinner.
I'm working on Kali nethunter.
P.S This is my first time on stack overflow pls forgive me if its too stupid or tell me if I missed something.
A starting point:
#!/bin/bash
echo "Enter IP:"
read -r ip
# `&` runs in parallel
# always quote variables
nmap "$ip" &
# `$!` get's the background process PID
pid=$!
...
# you can see if a pid is running by checking exit status of `kill -0`
while kill -0 "$pid" 2>&1 >/dev/null; do
printf ...
...
done
nmap -A -Pn -sV 192.168.0.1 -p -oN /tmp/op >/dev/null & pid=$!
i=1
sp="/-\|"
echo -n ' '
while kill -0 "$pid" 2>&1 >/dev/null;
do
printf "\b${sp:i++%${#sp}:1}"
done
*Problem solved thanks to * kamil cuk
I have a python script called hdsr_writer.py. I can launch this script in shell by calling
"python hdsr_writer.py 1234"
where 1234 is a parameter.
I made a shell script to increase the number and execute the python script with the number every 1 second
for param from 1 to 100000
python hdsr_writer.py $param &
sleep (1)
Usually, the python script executes its task within 0.5 second. However, there are times at which the python script gets stuck and resides in the system for longer than 30 seconds. I don't want that. So I would like to monitor life time of each python process executed. If it has stayed for longer than 2 second it would be killed and re-executed 2 times at most.
Note: I would like do this in the shell script not python script because I could not change the python script.
Update: More explainations about my question
Please note that: launching a new python process and monitoring python processes are independent jobs. Launching job doesn't care how many python processes are running and how "old" they are, just calls "python hdsr_writer.py $param &" every 1 second after increasing param. On the other hand, monitoring job periodically checks life time of all hdsr_writer python processes. If one has resided more than 2 second in memory, kills it, and re-runs it at most of 2 times.
Not so short answer
#/bin/bash
param=1
while [[ $param -lt 100000 ]]; do
echo "param=$param"
chances=3
while [[ $chances -gt 0 ]]; do
python tst.py $param &
sleep 2
if [[ "$(jobs | grep 'Running')" == "" ]]; then
chances=0
else
kill -9 $(jobs -l | awk '{print $2}')
chances=$(($chances-1))
if [[ $chances -gt 0 ]]; then
echo "one more chance for parameter $param"
fi
fi
done
param=$(($param+1))
done
UPD
This is another answer as requested by OP.
Here is still 2 scripts in one. But they can be spitted in two files.
Please pay attention that $() & is used to run sub-shells in background
#!/bin/bash
# Script launcher
pscript='rand.py'
for param in {1..10}
do
# start background sub-shell, where python with $param is started
echo $(
left=3
error_on_exit=1
# go if any chances left and previous run exits not with code 0
while [[ ( ( $left -gt 0 ) && ( $error_on_exit -ne 0 ) ) ]]; do
left=$(($left-1))
echo "param=$param; chances left $left "
# run python and grab python exit code (=0 if ok)
python $pscript $param
error_on_exit=$?
done
) &
done
# Script controller
# just kills python processes older than 2 seconds
# exits after no python left
# $(...) & can be removed if this code goes to separate script
$(while [[ $(ps | grep -v 'grep' | grep -c python ) != "0" ]]
do
sleep 0.5
killall -9 -q --older-than 2s python
done) &
Use a combination of sleep and nohup commands. After sleep time use kill to finish the execution of python script. You can check if the process is running with ps command.
#!/usr/bin/ksh
for param from {1..100000}
nohup python hdsr_writer.py $param &
pid=$!
sleep(2)
if [ ps -p $pid ]
then
kill -9 $pid
fi
done
Re-answer:
I'd use two scripts, the first one (script1.ksh):
#!/usr/bin/ksh
for param from {1..1000000}
nohup script2.sh $param &
done
And the second (script2.ksh):
#!/usr/bin/ksh
for i from {1..3}
python hsdr_write.py $1 &
pid=$!
sleep(2)
if [ ps -p $pid ]
then
kill -9 $pid
else
echo 'Finalizado'$1 >> log.txt
return
fi
done
The first script will launch all yours processes one after the other. The second one will check his own python process.