I'm trying to record system metrics using top while other processes are running. I'm hoping to chain things together, like so:
#!/bin/bash
# Redirect `top` output
top -U $USER > file.txt &&
# Then run a process that just sleeps for 4 seconds
python3 -c 'import time;time.sleep(4)' &&
# Then run another process that does the same
python3 -c 'import time;time.sleep(4)'
When I run this, however, the latter two (Python) processes never complete. My goal is to start recording from top before any of the other processes start, then once those processes complete, stop recording from top.
#run the first command in background
top -U $USER > file.txt &
# Then run a process that just sleeps for 4 seconds
python3 -c 'import time;time.sleep(4)' &&
# Then run another process that does the same
python3 -c 'import time;time.sleep(4)' &&
# kill the command in background
kill %1
Related
I'm trying to run a script that is required to have an exit code of 0. Unfortunalty I cannot use an init.d or other startup script to control this this, so I must make this work.
Basically if I understand AWS's docs correctly (elastic beanstalk), I need be able to run the following two commands and exit with a 0 and provide no other output to stdout.
As the root user I need to cd to a particular dir and run these two commands:
pkill -f que
bundle exec que
In my actually script I have:
#!/usr/bin/env bash
su -s /bin/bash -c "cd /some/dir && nohup pkill -f que &>/dev/null &"
sleep 10
su -s /bin/bash -c "cd /some/dir && nohup bundle exec que &"
Which still causes this error to be raised:
returned non-zero exit status 1 (Executor::NonZeroExitStatus)
Any tips for how to silently run those commands correctly?
I'm also looking at these for ideas:
https://blog.eq8.eu/article/aws-elasticbeanstalk-hooks.html
http://www.dannemanne.com/posts/post-deployment_script_on_elastic_beanstalk_restart_delayed_job
But its still not clear to me how this is supposed to exit successfully
Perhaps I'm missing something, but wouldn't this be easily solved by using two shell scripts? One with cd, pkill, and bundle. Call this script (foo.sh) something like:
#!/usr/bin/env bash
su -c ./foo.sh > /dev/null 2>&1 < /dev/null
exit 0
I am writing a script that is runs a program using nohup that contains parameters, thus I wrap the command into a shell command.
I am trying to get the pid of the program but am running into difficulties.
This is what I am trying to run:
nohup bash -c '../command --dir /mnt/raid/test/test.txt "https://www.google.com/" > temp.txt 2>&1 &'; APP_PID=$! | {
#APP_PID=$!
echo "the pid is ${APP_PID}"
}
The result of this script shows an empty pid as below:
the pid is
I have tried numerous variations and have searched Stack Overflow but could not find anything related.
Question is what am I doing wrong?
The background process is being created as a grand child of your shell (by bash -c) and you won't be able to get the PID of the grand child from the outer shell. You could do this instead:
nohup bash -c '../command --dir /mnt/raid/test/test.txt "https://www.google.com/" > temp.txt 2>&1'& APP_PID=$! && {
echo "the pid is ${APP_PID}"
}
Here, nohup runs in background and you get its PID.
I have written a script that has several functions. There is several functions which are bash script - which make a call to an expect script. When I test the function in terminal it runs fine and completes - however when I test my entire script it hangs. If I remove the expect script's it will finish but it runs cumulative time and not what I would expect of a complete time of 6m30.400s which is the longest time'd function.
Is this the correct output to background the process and send stdout to the bit bucket - so that all processes will run concurrently backup_ironport &> /dev/null eg.
I ran the script with the first 6 functions and I expected it to take 2m48 seconds but it took 4m47 seconds - which seems to be running them separately and not concurrently.
time ./network-bak.sh
real 4m47.033s
If I let it run all functions it hangs for over 15 minutes and I have to stop it. If I run functions separately in the shell to test it will complete. I ran bash -x and script and it just sits on the first expect script backup_cisco_firewall and goes no-where.
If I check the process in the system: it sits there forever
ps aux |grep fw-bak-expect
user 30925 0.0 0.0 0 0 pts/7 Z 10:18 0:00 [fw-bak- expect.s] <defunct>
Here is a snip of the functions of the script and the run process: I tested each function in terminal with time function - and put the real time next to the function with the # tag to show how long it took to run.
backup_fortigate()
{ for fortigate in `cat "$h5"`; do scp $fortigate:sys_config "$b3"/$fortigate-$date; done; }
backup_cisco_firewall()
{ cd "$sc" ; for fw in `cat "$h2"`; do ./fw-bak-expect.sh $fw ; done; }
########################
# Start of MAIN #
# First Run the Backups#
########################
rotate &
rpid=$!
backup_ironport &> /dev/null # real 0m27.490s
backup_fortigate &> /dev/null # real 0m40.816s
backup_nexus &> /dev/null # real 0m35.346s
backup_switch-router &> /dev/null # real 2m48.649s
backup_rsa &> /dev/null # real 0m1.017s
backup_tlite &> /dev/null # real 0m29.589s
backup_cisco_firewall &> /dev/null # real 6m30.400s # no sys-context
backup_sw-no-pk &> /dev/null # real 4m6.729s
backup_esx &> /dev/null # real 1m24.330s
wait
##############################
# Now we confirm the backups #
##############################
confirm_backup > /dev/null
search_for_backups > /dev/null
vh1=$(wc -l < "$f1")
vh2=$(wc -l < "$f2")
backup_verify
# zero the verification files for the next run
cat /dev/null > "$f1"
cat /dev/null > "$f2"
cat /dev/null > "$mh"
kill -9 $rpid
echo "\b\b "
The &> syntax is a short-hand to redirect both stdout and stderr at the same time, here to /dev/null. If you want to background the function call and drop the output, you need to include the actual backgrounding token:
...
backup_ironport &> /dev/null &
...
I want to distribute the work from a master server to multiple worker servers using batches.
Ideally I would have a tasks.txt file with the list of tasks to execute
cmd args 1
cmd args 2
cmd args 3
cmd args 4
cmd args 5
cmd args 6
cmd args 7
...
cmd args n
and each worker server will connect using ssh, read the file and mark each line as in progress or done
#cmd args 1 #worker1 - done
#cmd args 2 #worker2 - in progress
#cmd args 3 #worker3 - in progress
#cmd args 4 #worker1 - in progress
cmd args 5
cmd args 6
cmd args 7
...
cmd args n
I know how to make the ssh connection, read the file, and execute remotely but don't know how to make the read and write an atomic operation, in order to not have cases where 2 servers start the same task, and how to update the line.
I would like for each worker to go to the list of tasks and lock the next available task in the list rather than the server actively commanding the workers, as I will have a flexible number of workers clones that I will start or close according to how fast I will need the tasks to complete.
UPDATE:
and my ideea for the worker script would be :
#!/bin/bash
taskCmd=""
taskLine=0
masterSSH="ssh usr#masterhost"
tasksFile="/path/to/tasks.txt"
function getTask(){
while [[ $taskCmd == "" ]]
do
sleep 1;
taskCmd_and_taskLine=$($masterSSH "#read_and_lock_next_available_line $tasksFile;")
taskCmd=${taskCmd_and_taskLine[0]}
taskLine=${taskCmd_and_taskLine[1]}
done
}
function updateTask(){
message=$1
$masterSSH "#update_currentTask $tasksFile $taskLine $message;"
}
function doTask(){
return $taskCmd;
}
while [[ 1 -eq 1 ]]
do
getTask
updateTask "in progress"
doTask
taskErrCode=$?
if [[ $taskErrCode -eq 0 ]]
then
updateTask "done, finished successfully"
else
updateTask "done, error $taskErrCode"
fi
taskCmd="";
taskLine=0;
done
You can use flock to concurrently access the file:
exec 200>>/some/any/file ## create a file descriptor
flock -w 30 200 ## concurrently access /some/any/file, timeout of 30 sec.
You can point the file descriptor to your tasks list or any other file, but of course the same file in order to flock work. The lock will me removed as soon as the process that created it is done or fail. You can also remove the lock by yourself when you don't need it anymore:
flock -u 200
An usage sample:
ssh user#x.x.x.x '
set -e
exec 200>>f
echo locking...
flock -w 10 200
echo working...
sleep 5
'
set -e fails the script if any step fails. Play with the sleep time and execute this script in parallel. Just one sleep will execute at a time.
Check if you are reinventing GNU Parallel:
parallel -S worker1 -S worker2 command ::: arg1 arg2 arg3
GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.
If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:
GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:
Installation
If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
Learn more
See more examples: http://www.gnu.org/software/parallel/man.html
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel
try to implement something like
while read line; do
echo $line
#check if the line contains the # char, if not execute the ssh, else nothing to do
checkAlreadyDone=$(grep "^#" $line)
if [ -z "${checkAlreadyDone}" ];then
<insert here the command to execute ssh call>
<here, if everything has been executed without issue, you should
add a commad to update the file taskList.txt
one option could be to insert a sed command but it should be tested>
else
echo "nothing to do for $line"
fi
done < taskList.txt
Regards
Claudio
I think I have successfully implemented one: https://github.com/guo-yong-zhi/DistributedTaskQueue
It is mainly based on bash, ssh and flock, and python3 is required for string processing.
I have 3 commands that I am trying to run when the system start as a cronjob.
# Sleep at startup
sleep 2m
#command num 1:
./trace.out
sleep 5
#Command num 2:
java -jar file.jar
sleep 5
#Command num 3:
sh ./script.sh
is there any way to make this script more efficient using a loop, some way to make sure every script is running before executing the next one.
I would use && between each command as it executes each command, only if the previous one succeeded! For example:
# Sleep at startup
sleep 2m
./trace.out && java -jar file.jar && sh ./script.sh