In bash, nvidia-smi command gives you information about the GPU.
We also have option to get this periodically such as nvidia-smi -lms 50
I want to get this info only as long as a particular process is running.
Pseudocode
nvidia-smi -lms 50 & > logfile.txt
(time ./process1) > timelog.txt
while process1 is running:
keep nvidia-smi running
kill nvidia-smi
How can I do this in bash, cleanly, such that once my bash script exits no process that starts here is left behind for me to clean?
A direct nvidia-smi based solution would be preferred to a bash based one, but the latter is also perfectly fine.
Run both in the background, then wait for the one your job depends on.
nvidia-smi -lms 50 > logfile.txt &
nvpid=$!
time ./process1 > timelog.txt &
prpid=$!
wait "$prpid"
kill "$nvpid"
Related
I am looking for an alternative to something like ssh user#node1 uptime && ssh user#node2 uptime, where both of the SSH-commands are run simultaneosly. As they are both blocking until the command returns, && and ; between them don't work.
My goal is to run infinite while loops on both nodes via SSH. So the first one would never return, and the second one would never be run. I would then like to save the output after terminating the loops with Ctrl+C to a log-file and read that one via Python.
Is there an easy solution to this?
Thanks in advance!
Capturing SSH output
On the one hand, you need to capture the ssh output/error and store it into a file so that you can process it afterwards with Python. To this purpose you can:
1- Store output and error directly into a file
ssh user#node cmd 2>&1 > session.log
2- Show output/error in the console while storing it into a file (I would recommend this one)
ssh user#node cmd 2>&1 | tee session.log
Check this for further information about the tee command.
Running commands in parallel
On the other hand, you want to run both commands in parallel and block the current bash process. You can achieve this by:
1- Blocking the current bash process until their childs are done.
cmd1 & ; cmd2 & ; wait
Check this for further information about the wait command.
2- Spawning the child processes and freeing the current bash process. Notice that the processes will be kept alive although the main process ends.
nohup cmd & ; nohup cmd &
The whole thing
I would recommend combining both approaches using tee (so you can still see the ssh outputs on your terminal) and blocking the current process until everything is done (so that when you kill the main process all the processes are killed too).
ssh user#node1 uptime 2>&1 | tee session1.log & ; ssh user#node2 uptime 2>&1 | tee session2.log & ; wait
The perf-stat command in linux runs until crtl-c is pressed. I am trying to use this command in script to profile a loop. The recommended solution to simulate sending crtl-c is to issue a kill command with -2 or -SIGINT flag.
However this does not work for me. I am on RHEL.
The script more or less looks as follows:
for i in {1..12}
do
pid=$1
perf stat -e dTLB-loads -p $pid > perf.out&
perf_pid=$!
sleep 10
kill -SIGINT $perf_pid
done
Even after the kill the perf process is still active. All the ctrl-c's are executed at the end when the script gets over.
Reading the man page for perf, I came across the --control option which seems is the proper approach to profile a portion of running command.
However, this option is not available on RHEL .
I was able find workaround by using the -INT option for kill mentioned here. For some reason -2 or -SIGINT doesn't work on RHEL.
I sometimes launch long running tasks on my server and want the server to do something after those tasks finish (usually shut down). If there was only one task, I could simply type the next command into the window running the task, then bash will run it after the current one finishes. But what if there was multiple processes that I want to wait on?
In my workflow, the different tasks are running in different panes on tmux, so I cannot directly use wait since the processes I want to wait for are not child processes in one particular pane.
I have included a possible approach as an answer below.
This question's answer offers a related solution:
tail --pid=$pid -f /dev/null
However, that particular answer can only handle waiting for one process, but we can extend it using wait for multiple processes, then run our own command on completion:
tail --pid=$pid1 -f /dev/null &
tail --pid=$pid2 -f /dev/null &
tail --pid=$pid3 -f /dev/null &
tail --pid=$pid4 -f /dev/null &
wait; <your-command-here>
I have many processes and each of them take a lot of time to come up (5-10 min).
I am running my script in abc#abc1 and ssh to xyz#xyz1 to bring up the daemons.
There in the other machine(xyz#xyz1) I want to bring up 10 processes in parallel (call there startup scripts).
Then after 10 min I will check there status are they up or down.
I am doing this because I want the execution time of (my) script to be minimum.
How to do this using shell script with minimum amount of time ?
Thanks
Something like this should get your processes started:
for cmd in bin/proc1 bin/proc2 bin/procn; do
logfile=var/${cmd#bin/}.out
ssh xyz#xyz1 "bash -c '$cmd > $logfile 2>&1 &' && echo 'started $cmd in the background. See $logfile for its output.'"
done
My current script looks like this:
cd ~/.wine/drive_c/
echo "test123" > foo$$.txt
wine start "C:\foo$$.txt"
wineserver -w
echo "Wine is done!"
which works fine when only one program is running in wine at a time. However if I run this a second time, before the first program is done, both scripts will wait for each others programs to exit.
This does not work:
cd ~/.wine/drive_c/
echo "test123" > foo$$.txt
$(wine start "C:\foo$$.txt") &
wait ${!}
echo "Wine is done!"
as it will exit before you close the text editor.
I need to use the start command, because I want a file to be run with its default editor/viewer.
To wait for the process started by wine to exit, you can pipe its output to another program.
On my system, the following achieves the desired effect:
wine "program.exe" | cat
echo "program.exe has finished"
If you want to wait asynchronously:
wine "program.exe" | cat & pid=$!
# ...
wait $pid
echo "program.exe has finished"
wineserver has a --wait flag which can be used to do exactly that.
However if you run multiple programs at once, it will wait for all of them to finish.
wine <program> waits until the program exits. wine start program does not.
A summary:
wine <program> starts the program and waits until it is finished. I recommend using this method.
wine start <program> starts the program and immediately exits without waiting. The program will keep running in the background.
wine start \wait <program> starts the program and waits until it is finished. This is the same behavior as wine <program>.
wineserver --wait waits until all programs and all services in Wine are finished. This command does not launch any program itself but waits for existing programs and services.
Services like services.exe, plugplay.exe, and winedevice.exe keep on running a few seconds after the last program finishes, and wineserver --wait also waits until these services exit.
Some of these services hold state and write their state (and the registry) to disk when they exit. So if you want to backup or remove your wine prefix, make sure to wait until these services have exited.
What happens is that wine just asks wineserver to start the program and exits, and I have found no good mechanism to get notifications from wineserver about the processes that it spawns.
My suggestion would be to wait for the completion of the process started by wineserver using one of the methods in How to wait for exit of non-children processes, but you need to know its PID. Possible ideas: run wineserver -f -d | grep init_thread( and get PIDs from there, but I can see no easy way to find out which is which, to avoid race conditions, and to ignore noise; or try to find your process in the output of ps, but it's ugly, and definitely not robust.
If nothing better surfaces, you might want to suggest the addition of such a feature to the Wine devs (probably as a flag to wine).