I haven't been able to thoroughly concise my question, which means search engines can't give me a straight answer, and I've used this site in the past many times, so it's time I ask my own question..
So, I have a script (in bash) that runs through my list of pictures and determines if they need to be resized, which works well. I want to have a maximum of 3 running at a time. This is what I have.
if [ "$COUNT" = "$(echo "$(cat /proc/cpuinfo|grep processor|wc -l)+1"|bc)" ] ;then
wait
COUNT="0"
fi
Above, I set a COUNT variable to 0, and append 1 in a for loop (the same for loop that runs the convert statement). Once it hits (in this case) 3, (number of processors+1), it will 'wait' until the childs are done, then continue.
This is all fine and dandy, but let's say all my images are 48k, except for 1 which is 250mb (not true, but for explanation purposes). Somewhere down the line, the script would wait for that ONE picture when the rest could be running.
So finally, my question. Given the context and background info above, is there a way (using the 'wait' command or not) to have a script stop at 3 (processors+1) childs, and only execute 1 as one finishes? In context, is there a way to have 2 pictures running while the 250mb picture is doing its thing, then have 3 once it finishes? As opposed to what I do now, just wait for all 3, then execute 3 more.
Thank you for any and all suggestions!
Related
I need to lookup from a long list ("list_of_unique_ID") of shuffled ID ("shuffled_ID") in a huge file ("huge_file") those ID and extract them.
list_of_unique_IDhas been created from "huge_file" but only with its unique ID
The mono process scenario with AWK works fine.
I tried to multiprocess it with the following code : (bash, pseudocode because I can't extract the code from the machine it's running)
for sample in $list_of_samples; do
shuf -n lines_to_be_extracted list_of_unique_ID > shuffled_ID.txt
if (not_splitted = TRUE)
then
split --number=l/number_of_processes --numeric-suffixes huge_file huge_file_splitted_
not_splitted = FALSE
fi
done
for n in number_of_processes ; do
awk -v =shuffled_ID.txt [AWK function to lookup for the shuffled id] huge_file_splitted_${n} > output_${n} &
done
I'm running 5 processes for 3 parameters in parallel.
Which is fairly "simple". I'm just surprised as this script takes much time to run, that given the fact it's shuffled, I have first the output_00 and then output_02 and then output_05 each and every time which are filling first, in this order, before the other ones. I ran it multiple times and tried to swap the splitted huge_file but it seems that the order is given by what's outputed from the split function.
I don't think it's logical unless shuffle actually "jumps" over lines but keeps having the same order as the original file ? (But from what I have read, it seems not)
I might be wrong but I don't really the point of multiprocessing it if the processes are not parallel but subsequent ?
Is there something wrong in my code? Is there something I don't understand clearly?
Thanks !
Edit : thanks for the correction. I corrected my vocabulary :)
I've written a computer vision script that requires the use of a GPU, but my main webserver doesn't have a GPU (and can't have one).
As such my main webserver is box A, and I've bought a separate server, box B, with a GPU.
Workflow:
1) Users of my site upload photos to box A.
2) I process them for object-detection on box B (in batches as the object-detection code has a slow initial start-up)
3) And then I store the results on box A.
Currently I have 3 cron scripts:
1) box A: script makes filename list of most recent un-processed 1000 files.
(This runs every 5 minutes and typically takes a second.)
2) box B: using rsync, the script retrieves filename list from box A, then retrieves all of the files in the list, runs Yolo object detection on the list of images, and saves the results to results.txt
(This runs every 5 minutes, 1 minute after script 1 and typically takes 2 minutes.)
3) box A: another script connects to box B, retrieves results.txt, and processes them. Images are marked as processed so that the next time script (1) runs it doesn't do the same files again.
(This runs every 5 minutes, 1 minute before script 1 and typically takes a few seconds.)
What I'd like:
In theory I feel I could probably run all of this from a single script on box A, which would be cleaner and simpler. I'm concerned about what'll happen when one of the scripts gets delayed. I could write logic in to add some checks (touch script2_completed, for example), but I think I should consolidate all of the logic into one script, that executes code on both servers.
My ignorance:
I've never done this before though, and haven't a clue how best to go about it, and what's possible/isn't possible.
For instance, if I start running the remote object detection script on box 2, will the script running on box 1 wait until it's finished before it moves on to the next command, or would it end up being asynchronous, which would be undesirable?
Many thanks!
Turns out this is ridiculously simple.
To execute a script on the remote machine from the local machine, it's as simple as
ssh user#serverIP sh /run/this/script.sh
And as #Aaron says in a comment above, that runs as a blocking request so your local script won't resume until that remote script has finished.
I read this interesting question, that basically says that I should always avoid reaching PID of processes that aren't child processes. It's well explained and makes perfect sense.
BUT, while OP was trying to do something that cron isn't meant to, I'm in a very different situation :
I want to run a process say every 5 minutes, but once in a hundred times it takes a little more than 5 minutes to run (and I can't have two instances running at once).
I don't want to kill or manipulate other processes, I just want to end my process without doing anything if another instance of the process is running.
Is it ok to fetch PID of "not-child processes" in that case ? If so, how would I do it ?
I've tried doing if pgrep "myscript"; then ... or stuff like that, but the process finds its own PID. I need to detect if it finds more than one.
(Initially before being redirected I read this question, but the solution given doesn't work: it can give pid of the process using it)
EDIT: I should have mentioned it before, but if the script is already in use I still need to write something in a log file, at least : date>>script.log; echo "Script already in use">>script.log", I may be wrong but I think flock doesn't allow to do that.
Use lckdo or flock to avoid duplicated running.
DESCRIPTION
lckdo runs a program with a lock held, in order to prevent multiple
processes from running in parallel. Use just like nice or nohup.
Now that util-linux contains a similar command named flock, lckdo is
deprecated, and will be removed from some future version of moreutils.
Of course you can implement this primitive lockfile feature by yourself.
if [ ! -f /tmp/my.lock ];then
touch /tmp/my.lock
run prog
rm -f /tmp/my.lock
fi
I'm running a process that will take, optimistically, several hours, and in the worst case, probably a couple of days.
I've tried a couple of times to run it and it just never seems to complete (I should add, I didn't write the program, it's just a big dataset). I know my syntax for the command is correct as I use it all the time for smaller data and it works properly (I'll spare you the details as it is obscure for SO and I don't think that relevant to the question).
Consequently, I'd like to leave the program unattended running as a fork with &.
Now, I'm not totally sure whether the process is just grinding to a halt or is running but taking much longer than expected.
Is there any way to check the progress of the process other than ps and top + 1 (to check CPU use).
My only other thought was to get the process to output a logfile and periodically check to see if the logfile has grown in size/content.
As a sidebar, is it necessary to also use nohup with a forked command?
I would use screen for this purpose. see the man for more reference
Brief summary how to use:
screen -S some_session_name - starts a new screen session named session_name
Ctrl + a + d - detach session
screen -r some_session_name returns you to your session
I want to do a task every 5 mins. I want to control when i can start and when i can end.
One way is to use sleep in a while true loop, another way is to use cronjob. Which one is preferred performance-wise?
Thanks!
cron is almost always the best solution.
If you try to do it yourself with a simple script running in a while loop:
while true; do
task
sleep 300
done
you eventually find that nothing is happening because your task failed due to a transient error. Or the system rebooted. Or some such. Making your script robust enough to deal with all these eventualities is hard work, and unnecessary. That's what cron is for, after all.
Also, if the task takes some non-trivial amount of time, the above simple-minded while loop will slowly shift out of sync with the clock. That could be fixed:
while true; do
task
sleep $((300 - $(date +%s) % 300))
done
Again, it's hardly worth it since cron will do that for you, too. However, cron will not save you from starting the task before the previous invocation finished, if the previous invocation got stuck somehow. So it's not a completely free ride, but it still provides you with some additional robustness.
A simple approach to solving the stuck-task problem is to use the flock utility. For example, you could cron a script containing the following:
(
flock -n 8 || {
logger -p user.warning Task is taking too long
# You might want to kill the stuck task here. See pkill
exit 1
}
# Do the task here
) 8> /tmp/my_task.lck
Use a cron job. Cron is made for this type of use case. It frees you of having to to code the while loop yourself.
However, cron may be unsuitable if the run time of the script is unpredictable and exceeds the timer schedule.
Performance-wise It is hard to tell unless you share what the script does and how often it does it. But generally speaking, neither option should have a negative impact on performance.