I have a Gearman worker in a shell script started with perp in the following way:
runuid -s gds \
/usr/bin/gearman -h 127.0.0.1 -t 1000 -w -f gds-rel \
-- xargs /home/gds/gds-rel-worker.sh < /dev/null 2>/dev/null
The worker only does some input validation and calls another shell script run.sh that invokes bash, curl, Terragrunt, Terraform, Ansible and gcloud to provision and update resources in GCP like this:
./run.sh --release 1.2.3 2>&1 >> /var/log/gds-release
The script is intended to run unattended. The problem I have is that after the job finishes successfully (that's both shell scripts run.sh and gds-rel-worker.sh) the Gearman job remains executing, because the child process becomes zombie (see last line below).
root 144748 1 0 Apr29 ? 00:00:00 perpboot -d /etc/perp
root 144749 144748 0 Apr29 ? 00:00:00 \_ tinylog -k 8 -s 100000 -t -z /var/log/perp/perpd-root
root 144750 144748 0 Apr29 ? 00:00:00 \_ perpd /etc/perp
root 2492482 144750 0 May14 ? 00:00:00 \_ tinylog (gearmand) -k 10 -s 100000000 -t -z /var/log/perp/gearmand
gearmand 2492483 144750 0 May14 ? 00:00:08 \_ /usr/sbin/gearmand -L 127.0.0.1 -p 4730 --verbose INFO --log-file stderr --keepalive --keepalive-idle 120 --keepalive-interval 120 --keepalive-count 3 --round-robin --threads 36 --worker-wakeup 3 --job-retries 1
root 2531800 144750 0 May14 ? 00:00:00 \_ tinylog (gds-rel-worker) -k 10 -s 100000000 -t -z /var/log/perp/gds-rel-worker
gds 2531801 144750 0 May14 ? 00:00:00 \_ /usr/bin/gearman -h 127.0.0.1 -t 1000 -w -f gds-rel -- xargs /home/gds/gds-rel-worker.sh
gds 2531880 2531801 0 May14 ? 00:00:00 \_ [xargs] <defunct>
So far I have traced the problem to run.sh, because if I replace its call with something simpler (e.g. echo "Hello"; sleep 5) the worker does not hang. Unfortunately, I have no clue what is causing the problem. The script run.sh is rather long and complex, but has been working without a problem so far. Tracing the worker process I see this:
getpid() = 2531801
write(2, "gearman: ", 9) = 9
write(2, "gearman_worker_work", 19) = 19
write(2, " : ", 3) = 3
write(2, "gearman_wait(GEARMAN_TIMEOUT) ti"..., 151) = 151
write(2, "\n", 1) = 1
sendto(5, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
recvfrom(5, "\0RES\0\0\0\n\0\0\0\0", 8192, MSG_NOSIGNAL, NULL, NULL) = 12
sendto(5, "\0REQ\0\0\0\4\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
poll([{fd=5, events=POLLIN}, {fd=3, events=POLLIN}], 2, 1000) = 1 ([{fd=5, revents=POLLIN}])
sendto(5, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
recvfrom(5, "\0RES\0\0\0\6\0\0\0\0\0RES\0\0\0(\0\0\0QH:terra-"..., 8192, MSG_NOSIGNAL, NULL, NULL) = 105
pipe([6, 7]) = 0
pipe([8, 9]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fea38480a50) = 2531880
close(6) = 0
close(9) = 0
write(7, "1.2.3\n", 18) = 6
close(7) = 0
read(8, "which: no terraform-0.14 in (/us"..., 1024) = 80
read(8, "Identity added: /home/gds/.ssh/i"..., 1024) = 54
read(8, 0x7fff6251f5b0, 1024) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2531880, si_uid=1006, si_status=0, si_utime=0, si_stime=0} ---
read(8,
So the worker continues reading standard output even though the child has finished successfully and presumably closed it. Any ideas how to catch what causes this problem?
I was able to solve it. The script run.sh was starting ssh-agent, which opens a socket and since Gearman redirects all outputs the worker continued reading the open file descriptor even after the script successfully completed.
I found it by examining the open file descriptors for the Gearman worker process after it hang:
# ls -l /proc/2531801/fd/*
lr-x------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/0 -> /dev/null
l-wx------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/1 -> 'pipe:[9356665]'
l-wx------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/2 -> 'pipe:[9356665]'
lr-x------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/3 -> 'pipe:[9357481]'
l-wx------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/4 -> 'pipe:[9357481]'
lrwx------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/5 -> 'socket:[9357482]'
lr-x------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/8 -> 'pipe:[9369888]'
Then identified the processes using file node for the pipe in file descriptor 8 that German worker continued reading:
# lsof | grep 9369888
gearman 2531801 gds 8r FIFO 0,13 0t0 9369888 pipe
ssh-agent 2531899 gds 9w FIFO 0,13 0t0 9369888 pipe
And finally listed files opened by ssh-agent and found what stands behind file descriptor 3:
# ls -l /proc/2531899/fd/*
lrwx------. 1 root root 64 May 17 11:14 /proc/2531899/fd/0 -> /dev/null
lrwx------. 1 root root 64 May 17 11:14 /proc/2531899/fd/1 -> /dev/null
lrwx------. 1 root root 64 May 17 11:14 /proc/2531899/fd/2 -> /dev/null
lrwx------. 1 root root 64 May 17 11:14 /proc/2531899/fd/3 -> 'socket:[9346577]'
# lsof | grep 9346577
ssh-agent 2531899 gds 3u unix 0xffff89016fd34000 0t0 9346577 /tmp/ssh-0b14coFWhy40/agent.2531898 type=STREAM
As a solution I added kill of the ssh-agent before exit from run.sh script and now there are no jobs hanging due to zombie process.
I can't seem to find support for the CRON_TZ settings for crontab. How would I configure ansible to run a cron job for a specific timezone?
This is what I want my crontab file to look like. This cron config is currently running on a CentOS 7 box.
/etc/crontab:
# Default Timezone
30 9 * * * bobr /home/bobr/crontest.sh LOCAL "`date -R`"
CRON_TZ=Canada/Eastern
30 11 * * * bobr /home/bobr/crontest.sh Eastern "`date -R`"
45 11 * * * bobr/home/bobr/crontest.sh Eastern "`date -R`"
CRON_TZ=Canada/Pacific
30 8 * * * bobr/home/bobr/crontest.sh Pacific "`date -R`"
44 8 * * * bobr/home/bobr/crontest.sh Pacific "`date -R`"
For reference crontest.sh just sends an email with the 2 parameters:
#!/bin/bash
echo "this is the cron test. $1 $2" | mail -s "Cron Test" bobr#example.com
Didnt find any attribute for timezone set in ansible CRON module.. May be you can try out other way.. using shell..
- name: configure cron using shell
shell: crontab cron.conf
and conf file is
cat cron.conf
# Default Timezone
30 9 * * * bobr /home/bobr/crontest.sh LOCAL "`date -R`"
CRON_TZ=Canada/Eastern
30 11 * * * bobr /home/bobr/crontest.sh Eastern "`date -R`"
45 11 * * * bobr/home/bobr/crontest.sh Eastern "`date -R`"
CRON_TZ=Canada/Pacific
30 8 * * * bobr/home/bobr/crontest.sh Pacific "`date -R`"
44 8 * * * bobr/home/bobr/crontest.sh Pacific "`date -R`"
I have the following command in my crontab:
* * * * * root /home/amith/m.sh >dev/null 2>&1
where m.sh consists of:
#!/bin/sh
curl -0 "http://www.google.com" > /home/amith/Desktop/h2
but the command in shell script is not executing at all.
Can anyone please tell me the solution?
First be sure that your script /home/amith/m.sh is running correctly!
Your crontab entry is wrong you dont need root before script. Also your redirection to /dev/null is not good you are missing / before dev
You can set env varibale for SHELL in crontab with this
crontab -e
SHELL=/bin/sh
Then add your script:
* * * * * /home/amith/m.sh >/dev/null 2>&1
* * * * * cd /full/path; sh m.sh;
First, change the directory into your file location (use cd). Then, bash it with sh command.
Change * * * * * with the time schedule. To run the file every minute * * * * * To run every hour (1:00, 2:00, 3:00 and so on) use 00 * * * * To run everyday at 6 AM use 00 06 * * * To run the file on 9th March at 6 AM use 00 06 09 03 * The structure is minute hour date month day of week If you want to schedule it every Monday at 06 AM use this 00 06 * * 1 See this link http://www.pantz.org/software/cron/croninfo.html
I setup a cronjob to call myscript.sh every 5 min which then calls a php file between 30 sec and 3 in time and I don't get it why the average Interval is 05:09.
I want to call cron2_.php every 4-8 min but no chance to achieve that.
Tank you.
Cron Job: */5 * * * * myscript.sh
Shell script:
#!/bin/sh
# Grab a random value between 60-180 or ( between 30sec and 3 minutes )
value=$RANDOM
while [ $value -gt 180 ] || [ $value -lt 30 ] ;
do
value=$RANDOM
done
# Sleep for that time.
sleep $value
# Exectue Cron.
echo "Exectued on:$(date)" >> public_html/log_file.txt
exec php -f public_html/cron2_.php
Here is the exectuion time for 2 hours:
Average Interval -> 05:09
Execution Time Interval Min:Sec
13:02:52 00:00
13:07:06 04:14
13:11:35 04:29
13:17:34 05:59
13:21:55 04:21
13:26:54 04:59
13:32:00 05:06
13:35:50 03:50
13:42:44 06:54
13:47:03 04:19
13:51:26 04:23
13:56:48 05:22
14:01:53 05:05
14:07:42 05:49
14:12:15 04:33
14:16:22 04:07
14:23:01 06:39
14:27:17 04:16
14:32:21 05:04
14:35:57 03:36
14:42:14 06:17
14:45:44 03:30
14:52:52 07:08
14:56:50 03:58
15:02:57 06:07
15:06:43 03:46
15:12:26 05:43
15:16:29 04:03
15:22:00 05:31
15:25:35 03:35
15:31:51 06:16
15:37:51 06:00
15:42:56 05:05
15:47:32 04:36
15:50:36 03:04
15:55:45 05:09
16:02:15 06:30
16:06:10 03:55
16:11:11 05:01
16:15:56 04:45
16:21:58 06:02
16:25:56 03:58
16:31:09 05:13
16:37:06 05:57
16:42:30 05:24
16:45:36 03:06
You want your script to run every 4 to 8 minutes. Let's say then that we want, on average, one execution every 6 minutes. In that case, set the crontab line to:
*/6 * * * * myscript.sh
Next, in your script, put a random delay of zero to two minutes:
sleep $(($RANDOM % 120))
Consider two extreme cases. First, suppose that one job waits the maximum 2 minutes and the next waits the minimum of 0 minutes. The time between their executions is 4 minutes. For the second case, consider the opposite: the first job waits the minimum of 0 minutes and the second waits the maximum of 2 minutes. In this case, the time between their executions is 8 minutes. Thus, this approach achieves a wait of 4 to 8 minutes with an average wait of 6 minutes.
I'm trying to create a bash script that builds up a command to execute (that includes arguments). The name of the command executable is ms (which lives in the ms directory) and it takes a bunch of parameters that I compute and store in a string. When it comes time to execute the command, I try:
GENETREES=$(../ms/${x:1})
but am getting the error message:
./simulate.sh: line 21: ../ms/ms 6 1 -T -I 6 1 1 1 1 1 1 -ej 0.059851352500000010170566611122922040522098541259765625 1 4 -es 0.059851352500000010170566611122922040522098541259765625 4 0.3457841801761454281205487859551794826984405517578125 -ej 0.059851352500000010170566611122922040522098541259765625 2 3 -es 0.059851352500000010170566611122922040522098541259765625 3 0.54870128110803395582451003065216355025768280029296875 -ej 0.0897770262499999471828004971030168235301971435546875 7 9 -es 0.089777026250000002693951728360843844711780548095703125 3 0.8097582153199012200417428175569511950016021728515625 -ej 0.119702699999999995217336845598765648901462554931640625 3 4 -ej 0.125827642499999947656164067666395567357540130615234375 9 10 -es 0.1258276425000000031673152989242225885391235351562500 4 0.28069295861466903030390085405088029801845550537109375 -ej 0.13195258499999995560614252099185250699520111083984375 8 10 -ej 0.1817980399999999663318561715641408227384090423583984375 6 5 -ej 0.2525933399999999717788767839010688476264476776123046875 10 4 -ej 0.41145434999999996872332985731191001832485198974609375 4 5 : File name too long
I think bash thinks that I intended all those command parameters to be a part of the executable name. But this is not my intent.
What am I doing wrong?
Thanks.
Update - More info requested on how x is constructed
MSOUT=$(java -jar ./NetworkSearchGen.jar ms $3 $TRUENETWORK $4)
OIFS=$IFS
IFS='}'
first="true"
for x in $MSOUT
do
if [ $first = "true" ]; then
echo "$x"
else
GENETREES=$(../ms/${x:1})
fi
first="false"
done
It was setting IFS that screwed things up. Making IFS=" " just before executing got the script working again.