This is a strange one.
I have a bash script (let's call it fileChecker.sh) that loops through a directory of files. It checks each file, sends parameters for each to another bash script (uploadToS3.sh) which uploads them to an S3 bucket. The fileChecker.sh triggers the uploadToS3.sh and does not wait for it to finish (I believe this is called forking???). Snippet from the fileChecker.sh triggering the uploadToS3.sh:
sh ("/Users/Shared/Scripts/uploadToS3.sh" "$thefilepath" "$s3" "$thefilename" "$filenameExtension") &
The uploadToS3.sh script uses python and s3cmd to upload the file to the s3 bucket. Snippet from the script:
/usr/local/bin/s3cmd --access_key=$s3AccessKey --secret_key=$s3SecretKey --region=$s3Region --progress put "$thefilepath" "$s3Path""$thefilename"
The Problem: both scripts execute without issue when run manually from an IDE but I need it to run on time intervals every 30 seconds. When I run it with launchd, /Library/LaunchAgents/, the first script (fileChecker.sh) completes without issue. Each execution of the uploadToS3.sh is triggered successfully but never finishes. To be more specific, I check the output from each instance of the uploadToS3.sh and each file starts to upload to the s3 bucket then stops. There are no errors to be found in the stderr. The stdout just has the first details of the upload process.
Any thoughts? I'm happy to add more of the code and more detail if needed. Been stuck on this for a week now and could use any help I can get.
Thank you 🙏
I suspect you're tripping over a feature of launchd that's intended to be helpful, but sometimes causes problems like this: when a daemon or agent exits, launchd will "clean up" (i.e. kill) any leftover subprocesses. It sounds like when fileChecker.sh exits, launchd kills uploadToS3.sh before it has a chance to finish.
Fortunately, this is easy to fix. Add <key>AbandonProcessGroup</key><true/> to the launchd .plist file to disable this behavior.
Related
I may have inadvertently launched a bash script containing an infinite cycle whose exit condition may be met next century, if ever. The fact is that I launched the script, as I would do with a nohup program, with
bash [scriptname].sh &
so that (as I get it, which is most probably wrong) I can close the terminal and still keep the script running, as was my intention in developing it. The script should run calculation programmes in my absence and let me gather the results after some time.
Now I want to stop it, but nothing seems to do the trick: I killed the programmes the script had launched, I removed the input file the script was getting orders from and - last and most perfect of accomplishments - I accidentally closed the terminal trying to "exit" the script, which was still giving me error messages.
How can I check whether the script is running (as it does not appear in "top")? Is the '&' relevant? Should I just ask permission to reboot the pc, if that will work and kill everything?
Thank you.
[I put a "Hi everyone" at the beginning but the editor won't let me show it. Oh, well. It's that kind of day.]
Ok, I'll put it right here to prove my stupidity, as I wandered the internet shortly (after a long wandering before writing this post) and found that the line:
kill -9 $(pgrep -f [SCRIPTNAME].sh)
does the trick from any terminal window.
I write this answer to help anyone in the same situation, but feel free to remove the thread if unnecessary (and excuse me for disturbing).
Good you found it, here is another way if you do not use bash -c and run it in current shell not a separate shell.
# put a job in background
sleep 100 &
# save the last PID of background job
MY_PID=$!
# later
kill $MY_PID
I have a simple incron task setup to run a command whenever a particular .json file is written-to, then closed.
/var/www/html/api/private/resources/myfile.json IN_CLOSE_WRITE,IN NO LOOP /var/www/html/api/private/resources/run_service.sh
I can see that whenever the file to written to, there is a syslog entry for the event, and the command that was triggered - along the lines of <date> - incrond: CMD (/var/www/html/api/private/resources/run_service.sh).
But nothing seems to happen...
initially I thought this would be caused by an issue with the script, but replacing the script command to something simple such as echo "hello world" > /tmp/mylog.log still yields no output or results. I seem to have hit a brick wall with this one!
Update
Changing the incron command to read "/bin/bash /var/www/html/api/private/resources/run_service.sh" now seems to triggering the script correctly, as I can now get output from the script.
A simple mistake on my part, despite all examples online showing that using the script as the command should run it, for me it only works if I explicitly call bash to execute it
"<my directory/file to watch> <trigger condition> /bin/bash /var/www/html/api/private/resources/run_service.sh
I've been trying to generate and infinite loop in OpenWRT, and I've succeeded:
#!/bin/sh /etc/rc.common
while [ true ]
do
# Code to run
sleep 15
done
This code works as a charm if I execute it as ./script. However, I want this to start on its own when I turn on my router. I've placed the script in /etc/init.dand enabled it with chmod +x script.
Regardless, the program doesn't start running at all. My guess is that I shouldn't execute this script on boot up but have a script that calls this other script. I haven't been able to work this out.
Any help would be appreciated.
As I have messed with init scripts of OpenWRT in my previous projects. I would like contribute to Rich Alloway's answer (for the ones who will likely to drop here from google search). His answer only covers for "traditional SysV style init scripts" as it is mentioned in the page that he gave link Init Scripts.
There is new process management daemon, Procd that you might find in your OpenWRT version. Sadly documentation of it has not been completed yet; Procd Init Scripts.
There are minor differences like they have pointed out in their documentation :
procd expects services to run in the foreground,
Different shebang,
line: #!/bin/sh /etc/rc.common Explicitly use procd USE_PROCD=1
start_service() instead of start()
A simple init script for procd would look like :
#!/bin/sh /etc/rc.common
# it is run order of your script, make it high to not mess up with other init scripts
START=100
USE_PROCD=1
start_service() {
procd_open_instance
procd_set_param command /target/to/your/useless/command -some -useless -shit -here
}
I have posted some blog post about it while ago that might help.
You need to have a file in /etc/rc.d/ with an Sxx prefix in order for the system to execute the script at boot time. This is usually accomplished by having the script in /etc/init.d and a symlink in /etc/rc.d pointing to the script.
The S indicates that the script should run at startup while the xx dictates the order that the script will run. Scripts are executed in naturally increasing order: S10boot runs before S40network and S50cron runs before S50dropbear.
Keep in mind that the system may not continue to boot with the script that you have shown here!
/etc/init.d/rcS calls each script sequentially and waits for the current one to exit before calling the next script. Since your script is an infinite loop, it will never exit and rcS may not complete the boot process.
Including /etc/rc.common will be more useful if you use functions in your script like start(), stop(), restart(), etc and add START and STOP variables which describe when the script should be executed during boot/shutdown.
Your script can then be used to enable and disable itself at boot time by creating or removing the symlink: /etc/init.d/myscript enable
See also OpenWRT Boot Process and Init Scripts
-Rich Alloway (RogueWave)
I'm trying to write a simple shell script to start a program running in the background and redirect all its output to a log file, but I've come across a case where it silently fails and kills the program.
It happens if the file that's being redirected into is modified elsewhere. I've tried recreating the situation using the ping command so I can narrow the problem down to not being my program's code, but that fails in a different way - it stops writing to the file, but the terminal still acts as though the ping is running.
This is the essence of my code:
sh "$program" > "$file"
Is there a way to pause a Bash script, then resume it another time, such as after the computer has been rebooted?
The only way to do that AFAIK:
Save any variables, or other script context information in a temporary file to establish the state of the script just before the pause. This goes without saying that the script should include a mechanism to check this file to know if the previous execution was paused and, if it was, fetch all the context and resume accordingly.
After reboot, manually run the script again, OR, have the script automatically run from your startup profile script.
Try Ctrl-Z to pause the command. I don't think you can pause it and then resume after reboot unless you're keeping state somehow.
You can't pause and resume the same script after a reboot, but a script could arrange to have another script run at some later time. For example, it could create an init script (or a cron job, or a login script, etc) which contained the tasks you want to defer, and then removed itself.
Intriguing...
You can suspend a job in BASH with a CTRL-Z, but you can't resume after a reboot. A reboot initializes the machine and the process that was suspended is terminated.
However, it might be possible to force the process into a coredump via a 'kill -QUIT $pidand then usegdb` to restart the script. I tried for a while, but was unable to do it. Maybe someone else can point out the way.
If this applies to your script and the job it does, add checkpoints to it - that means places where all the state of the process is saved to disk before continuing. Then have each individual part check if the output they have to produce is already there, and skip running if it is. That should make a rerun of the script almost as efficient as resuming from the exact same place in execution.
Alternatively, run the script in a VM. Freeze the VM before shutting down the real system and resume it afterwards. It would probably take a really huge and complex shell script to make this worth it, though.