I have a shell script which runs very large simulation binaries. This becomes problematic when I want to request some output of variables in the script. For instance, when I run 10 large simulations, I want to be able to print which iteration I am on without having to wait a minute or two for the current simulation to terminate.
Currently, I am using the trap command. However, the script does not react immediately to signals but will only execute the binded function when the current iteration terminates. I will post the code if anyone needs it.
You should start threads for each large thing you're going to run. Have those threads dump results somewhere, then you have your main method free waiting to interrogate the results on the fly.
Related
I am working on a project where I have to send arguments by a command line to a python file (using system exec) and then visualize the results saved in a folder after the python file finishes executing. I need to have this by only clicking on one button, so my question is, if there is any way to realize this scenario or maybe if I can order the events.
Now I have included the flat sequence structure to the block Diagram so I can order the events, but I had an issue with making the program (the python file) running every time I press the Test button (it only runs in the first time I click on the Test button), I tried to use the while loop but I couldn't execute it again unless I restart the program.
The way you phrased your question makes me think that you want to wait until the command you call via system exec finished and then run some code. You could simply use a sequence structure for this.
However, if you need to do this asynchronously, i.e. launch the command and get an event when the command finished so you can draw the results, you will need to resort to asynchronous techniques like "Start Asynchronous Call" and "Wait On Asynchronous Call", or for example queues and a separate code area for the background-work.
Use "wait until completion?" input of System Exec function to make sure the script finished execution, then proceed with the results visualization part.
We have a QProcess that runs a bash script. The script finishes properly and produces expected output, but the finished signal takes a very long time (minutes) afterward to emit. Basically, our script is generating an encrypted tarball from a list of files fed as an argument. The final bundle is sitting there on disk, intact, but Process takes a very long time to return. This is preventing our UI from moving on to the next task, because we need to ensure the script has run to completion programatically, instead of through inspection. We're not doing anything other than
connect(myProcess, SIGNAL(finished()), mySlot, SLOT(tidyUp()));
myProcess.start();
We can monitor the size of the file with Qt, and we have an estimate of its final size based on the file list we feed the script, but the script hangs around for a very long time after the file has reached its estimated size. We've inserted sync statements, but that doesn't seem to have any effect. When the script is run on the command line, the file grows, and the script stops as soon as it reaches its final size.
Why is QProcess not sending it's finished signal immediately after the script completes?
We would very much like to attach a progress bar indicating percentage of file size produced, or give some other indication of progress, but we're stumped by this behavior. We've tried using both a worker thread moved to a QThread, and running the QProcess directly in a busy loop, calling processEvents(), to no avail.
Turns out this was a problem with the commands I had in my pipe. GPG plunks out the file fairly quickly (quite variable timing, though) but then often spends quite a lot of time idling/working after the file itself has reached its final size. I'm not sure what it's doing, or why it only does on some runs of the same content, but eventually it finishes, the script completes, and I get my finished() signal delivered. I may have to put a more elaborate progress bar in place that switches to busy wait if the file size hasn't changed for a while, but it appears that Qt is working as expected here after all.
I need to write a shell (bash) script that will be executing several Hive queries.
Each of the queries will produce a directory with a lot of files.
After all queries are finished I need to process all these files in a specific order.
I want to run Hive queries in parallel as background processes as each one might take couple of hours.
I would also like to parallelize resulting file processing but there are some culprits, that I don't know how to handle. I.e. I can start processing results of the first and second queries as soon as they are finished, but for the third, I need to hold until first two processors are done. Similarly for the fourth and fifth.
I won't have any problems writing such a program in Java, but how to do it in shell - beats me.
If someone can give me a hint on how can I monitor execution of these components in the shell script, I would appreciate it greatly.
Im writing a bash script that essentially fires off a python script that takes roughly 10 hours to complete, followed by an R script that checks the outputs of the python script for anything I need to be concerned about. Here is what I have:
ProdRun="python scripts/run_prod.py"
echo "Commencing Production Run"
$ProdRun #Runs python script
wait
DupCompare="R CMD BATCH --no-save ../dupCompareTD.R" #Runs R script
$DupCompare
Now my issues is that often the python script can generate a whole heap of different processes on our linux server depending on its input, with lots of different PIDs AND we have heaps of workers using the same server firing off scripts. As far as I can tell from reading, the 'wait' command must wait for all processes to finish or for a specific PID to finish, but when i cannot tell what or how many PIDs will be assigned/processes run, how exactly do I use it?
EDIT: Thank you to all that helped, here is what caused my dilemma for anyone google searching this. I broke up the ProdRun python script into its individual script that it was itself calling, but still had the issue, I think found that one of these scripts was also calling another smaller script that had a "&" at the end of it that was ignoring any commands to wait on it inside the python script itself. Simply removing this and inserting a line of "os.system()" allowed all the code to run sequentially.
It sounds like you are trying to implement a job scheduler with possibly some complex dependencies between different tasks. I recommend to use a job scheduler instead. It allows you to specify to run those jobs whilst also benefitting from features like monitoring, handling exceptional cases, errors, ...
Examples are: the open source rundeck https://github.com/rundeck/rundeck or the commercial one http://www.bmcsoftware.uk/it-solutions/control-m.html
Make your Python program wait on the children it spawns. That's the proper way to fix this scenario. Then you don't have to wait for Python after it finishes (sic).
(Also, don't put your commands in variables.)
With clojure.java.shell/sh it's possible to execute a shell command. After the invoked process is finished, the function returns a map containing it's exit code, std-out and std-err strings.
How can I capture stdout/-err of a spawned process from the moment it started? And: How can I terminate the process from within a clojure program/repl?
As far as I know it is not possible with clojure.java.shell/sh. You might take a look at Raynes/conch which provides features you ask for (getting output right after start etc.)
You can also DIY with java.lang.ProcessBuilder and java.lang.Process where you have full access to process's input stream or a method to terminate it.