The problem I want to tackle is as follows. I have a long(1 to 2 hours) running task that has to be run everyday. So the goto option was cron. But the catch is that I have to give a 24 hour gap between successive runs. So using cron now would involve rewriting the cron job file after every run. This might be clear after this example.
The long running job 'LR' starts at 6PM on Monday and finishes at 7:30PM sameday.
On Tuesday it's supposed to start at 7:30 PM and not 6PM (like it did on monday). This is because there has to be a 24hr gap between successive runs.
The obvious option here was to have a process running an infinite loop. start the LR job. Then sleep for 24hr and continue with the loop. This works perfectly too. In my setup there is a bash script which is running this loop.
while [ 1 == 1 ]; do
/bin/jobs/long_run.py
/bin/jobs/cleanup.sh
sleep 86400
done
So my question is what is the total amount of CPU resource spent and what is the RAM usage.
Not sure if this affects the answer in anyway; I'm running this on termux on an android phone.
Also please recommend other light weight options.
There is nothing to worry about resources, while a script executes sleep, it really sleeps. You should worry for if anything happens between two executions, like restart, downtime etc. This structure:
while true; do
sh script.sh
sleep 86400
done
does not resume and you don't save the time for the next execution anywhere. Similar to this structure is to have a wrapper, suppose f() is your jobs
f() {
echo working
}
wrapper() {
f
echo sleeping
sleep 86400
wrapper
}
wrapper
so now you call the wrapper, which works, sleeps and calls itself. You could use just this, if you are ok with what could go wrong, at least print the datetime somewhere.
You can replace the internal sleep and wrapper call with job scheduling with cron or at. Probably at is not a standard packet for all distributions (at least not for mine) while cron is. You could install it. For at the wrapper would be like this:
wrapper() {
f
at now +1 day wrapper
}
With cron, you could edit the crontab, like this but better use a crontab file like this, what you have to do is to parse date command, create the date prefix, update crontab.
Note: There may be other cron jobs for user, existing or added after that, this is considered in the last link.
I am watching my project dir for the file change and running sync script whenever files change.
Certainly I do not want to run the second synchronization before the first one is done. flock utility seems to be fit for preventing second sync from running as in
fswatch -0 ./myproject | xargs -0 -n 1 "flock /tmp/my.lock ./container_update.sh
However, it just puts the next request to the waiting queue, so if I change 20 small files, twenty synchronization will be run. That can be solved with flock -n that would quit immediately if lock cannot be obtained, but then I will lose changes performed while sync is in progress.
I tried building a naive single slot queue with a new item first requesting a "queue" lock, then proceeding to the "main" lock, leaving the "queue" one free for one more request only. It doesn't help; change requests continue to pile.
fswatch ./myproject | xargs -0 -n 1 "flock -n /tmp/my-queue.lock flock /tmp/my-main.lock flock -u /tmp/my-queue.lock ./container_update.sh"
What would be a way to let only one "next" request to be executed?
P.S.
If it matters I am running it on a Mac with this implementation of flock that is supposed to be identical to the Linux one - https://github.com/discoteq/flock
I have a simple bash script that runs some tasks which can take varying amounts of time to complete (from 15 mins to 5 hours). The script loops using a for loop, so that I can run it an arbitrary number of times, normally back-to-back.
However, I have been requested to have each iteration of the script start at the top of the hour. Normally, I would use cron and kick it off that way, every hour, but since the runtime of the script is highly variable, that becomes trickier.
It is not allowable for multiple instances of the script to be running at once.
So, I'd like to include the logic to wait for 'top of the hour' within the script, but I'm not sure of the best way to do that, or if there's some way to (ab)use 'at' or something more elegant like that. Any ideas?
You can still use cron. Just make your script use a lock file. With the flock utility you can do:
#!/bin/bash
exec 42> /tmp/myscriptname.lock
flock -n 42 || { echo "Previous instance still running"; exit 1; }
rest of your script here
Now, simply schedule your job every hour in cron, and the new instance will simply exit if the old one's still running. There is no need to clean up any lock files.
I have written a script to initiate multi-processing
for i in `seq 1 $1`
do
/usr/bin/php index.php name&
done
wait
A cron run every min - myscript.sh 3 now three background process get initiated and after some time I see list of process via ps command. I see all the processes are together in "Sleep" or "Running" mode...Now I wanted to achieve that when one goes to sleep other processes must process..how can I achieve it?. Or this is normal.
This is normal. A program that can run will be given time by the operating system... when possible. If all three are sleeping, then the system is most likely busy and time is being given to other processes.
I've been using the following Unix bash script:
#!/bin/bash
mkdir -p ~/Desktop/URLs
n=1
while read mp3; do
curl "$mp3" > ~/Desktop/URLs/$n.mp3
((n++))
done < ~/Desktop/URLs.txt
to download and rename a bunch mp3 files from URLs listed in "URLs.txt". It works well (thanks to StackOverflow users), but due to a suspected server quantity/time download limit, it's only allowing me to access a range of 40 - 50 files from my URL list.
Is there a way to work around this by adding a "timer" inside the while loop so it downloads 1 file per "X" seconds?
I found another related question, here:
How to include a timer in Bash Scripting?
but I'm not sure where to add the "sleep [number of seconds]"... or even if "sleep" is really what I need for my script...?
Any help enormously appreciated — as always.
Dave
curl has some pretty awesome command-line options (documentation), for example, --limit-rate will limit the amount of bandwidth that curl uses, which might completely solve your problem.
For example, replace the curl line with:
curl --limit-rate 200K "$mp3" > ~/Desktop/URLs/$n.mp3
would limit the transfers to an average of 200K per second, which would download a typical 5MB MP3 file in 25 seconds, and you could experiment with different values until you found the maximum speed that worked.
You could also try a combination of --retry and --retry-delay so that when and if a download fails, curl waits and then tries again after a certain amount of time.
For example, replace the curl line with:
curl --retry 30 "$mp3" > ~/Desktop/URLs/$n.mp3
This will transfer the file. If the transfer fails, it will wait a second and try again. If it fails again, it will wait two seconds. If it fails again, it will wait four seconds, and so on, doubling the waiting time until it succeeds. The "30" means it will retry up to 30 times, and it will never wait more than 10 minutes. You can learn this all at the documentation link I gave you.
#!/bin/bash
mkdir -p ~/Desktop/URLs
n=1
while read mp3; do
curl "$mp3" > ~/Desktop/URLs/$n.mp3 &
((n++))
if ! ((n % 4)); then
wait
sleep 5
fi
done < ~/Desktop/URLs.txt
This will spawn at most 4 instances of curl and then wait for them to complete before it spawns 4 more.
A timer?
Like your crontab?
man cron
You know what they let you download, just count the disk usage of your files that you did get.
There is the transfer you are allowed. You need that, and you will need the PID of your script.
ps aux | grep $progname | print awk '{print $1}'
or something like that. The secret sauce here is that you can suspend with
kill -SIGSTOP PID
and resume with
kill -SIGCONT PID
So the general method would be
Urls on an array or queue or whatever
bash lets you have
Process an url.
increment transfer counter
When transfer counter gets close
kill -SIGSTOP MYPID
You are suspended.
in your crontab foreground your script after a minute/hour/day whatever
Continue processing
Repeat until done.
just don't log out or you'll need to do the whole thing over again, although if you used perl it would be trivial.
Disclaimer, I am not sure if this is an exercise in bash or whatnot, I confess freely that I see the answer in perl, which is always my choice outside of a REPL. Code in Bash long enough , or heaven forbid, Zsh ( my shell ) and you will see why Perl was so popular. Ah memories...
Disclaimer 2: Untested, drive by , garbage methodology here only made possible because you've an idea what that transfer might be. Obviously, if you have ssh , use ssh -D PORT you#host and pull the mp3's out of the proxy half the time.
In my own defense, if you slow pull the urls with sleep you'll be connected for a while. Perhaps "they" might notice that. Suspend and resume and you only should be connected while grabbing tracks, and gone otherwise.
Not so much an answer as an optimization. If you can consistently get the first few URLs, but it times out on the later ones, perhaps you could trim your URL file as the mp3s were successfully received?
That is, as 1.mp3 is successfully downloaded, strip it from the list:
tail url.txt -n +2 > url2.txt; mv -f url2.txt url.txt
Then the next time the script runs, it'll begin from 2.mp3
If that works, you might just set up a cron job to periodically execute the script over and over, taking bites at a time.
It just occurred to me that you're programatically numbering the mp3s, and curl might clobber some of them on restart, since every time it runs it'll start counting at 1.mp3 again. Something to be aware of, if you go this route.