I've run into a problem with sending emails via CURL using shell script.
I have a script that do:
Download dashboards from our Tableau server (private) via tabcmd command then save these files into a directory.
Send these files to receivers using a SMTP sever (private) via CURL. Both Tableau Server and SMTP server are installed on a container CENTOS 7. I have CURL version 7.29.
I have a list of dashboards and list of emails. For each dashboard (that has a unique URL), the script downloads the dashboard files and then send the files to receiver.
My problem is that for some receivers they received their attached files but others didn't. I checked the stdout of CURL command, it returned that
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to services.xxxx.xx.xxxx.xxx port 25 (#0)
* Trying xxx.xxx.xxxx.xxx...
* Connected to services.xxxx.xx.xxxx.xxx (xxx.xxx.xxx.xxx) port 25 (#0)
< 220 xxxxx.xxxxx.xxxx.xxxx ESMTP Postfix
> EHLO tableau
< xxxxx.xxxxx.xxxx.xxxx
< 250-PIPELINING
< 250-SIZE 10240000
< 250-VRFY
< 250-ETRN
< 250-STARTTLS
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250-DSN
< 250-SMTPUTF8
< 250 CHUNKING
> MAIL FROM:<xxxxx#yyyy.email.com>
< 250 2.1.0 Ok
> RCPT TO:<xxxx#yyyy.hotmail.com>
< 250 2.1.5 Ok
> DATA
< 354 End data with <CR><LF>.<CR><LF>
} [data not shown]
< 250 2.0.0 Ok: queued as 2xxxxx
0 0 0 0 0 130k 0 5735k --:--:-- --:--:-- --:--:-- 5932k
* Connection #0 to host services.xxxx.xx.xxxx.xxx left intact
This means that the email is sent successfully with attached files. But the receiver didn't receive the attached files. I also checked that the files were in the directory.
The code to send email is something like that:
send_pdf(){
receiver="$1"
subject="$2"
message="$3"
num_file_attaches=$(expr $# - 3)
eval body=$(cat parameters/data.txt)
for file in "${#:4}"; do
add_file "$body" "$file"
done
body+="--MULTIPART-MIXED-BOUNDARY--"
echo -e "$body" | curl -s "$SMTP_SERVER" \
--mail-from "$SENDER" \
--mail-rcpt "$receiver"\
-T -
}
and the function add_file
add_file() {
body="$1"
file_attach="$2"
name_file_attach=$(basename $file_attach)
file_base64=$(cat $file_attach | base64)
MIMEType=`file --mime-type "$file_attach" | sed 's/.*: //'`
body+="--MULTIPART-MIXED-BOUNDARY\nContent-Type: $MIMEType;name=${name_file_attach}\nContent-Transfer-Encoding: base64\nContent-Disposition: attachment;filename=${name_file_attach}\n\n\n${file_base64}\n\n\n"
}
For each time I test with my emails, the script works but when I replace my email by those of clients, there are always some clients who didn't receive attached files. However, these clients received the message.
I can't do some tests with emails of clients :( So, I can't reproduce the problem.
Does anyone have any idea where the problem could come from or how can I reproduce the problem?
Many thanks,
I've got an unexpected error, I guess it's about timeout.
Task takes time quite long. generally less than 1minute. There is no problem when it fininsh before 1 minute. But sometimes task takes more than 1 minute. Error occurs when it takes more than 1minute. Please see below.
[ec2-user#ip-000-00-0-00 ~]$ curl --location --request POST 'userid.ap-northeast-2.elb.amazonaws.com:port/service' -d "video.mp4" -o output.json -v
Note: Unnecessary use of -X or --request, POST is already inferred.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.31.74.35:5000...
* Connected to loadbalancer.ap-northeast-2.elb.amazonaws.com (000.00.00.00) port 0000 (#0)
> POST /service HTTP/1.1
> Host: userid.ap-northeast-2.elb.amazonaws.com:port
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Length: 37
> Content-Type: application/x-www-form-urlencoded
>
} [37 bytes data]
100 37 0 0 0 37 0 0 --:--:-- 0:00:59 --:--:-- 0* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Date: Tue, 19 Apr 2022 01:45:23 GMT
< Content-Type: application/octet-stream
< Content-Length: 0
< Connection: keep-alive
< Server: Python/3.7 aiohttp/3.7.4.post0
<
100 37 0 0 0 37 0 0 --:--:-- 0:01:00 --:--:-- 0
* Connection #0 to host loadbalancer.ap-northeast-2.elb.amazonaws.com left intactm left intact
With curl verbose option, I get 500 Internal Server Error at 0:00:59. How can I finish my task which takes more than 1minutes?
I've tried
increasing Health check grace period for ECS Service
increasing idle timeout of Load Balancer
increasing timeout and interval of Target group
curl options (like keep-alive, max-time)
My EC2 Instance
type : t2.micro
Amazon Linux
My Service
Service type : REPLICA
Launch type : FARGATE
My task in service
Network mode : awsvpc
Compatibilities : EC2, FARGATE
Requries compatibilities : FARGATE
EFS mounted
Docker
Appreciate,
I have a Gearman worker in a shell script started with perp in the following way:
runuid -s gds \
/usr/bin/gearman -h 127.0.0.1 -t 1000 -w -f gds-rel \
-- xargs /home/gds/gds-rel-worker.sh < /dev/null 2>/dev/null
The worker only does some input validation and calls another shell script run.sh that invokes bash, curl, Terragrunt, Terraform, Ansible and gcloud to provision and update resources in GCP like this:
./run.sh --release 1.2.3 2>&1 >> /var/log/gds-release
The script is intended to run unattended. The problem I have is that after the job finishes successfully (that's both shell scripts run.sh and gds-rel-worker.sh) the Gearman job remains executing, because the child process becomes zombie (see last line below).
root 144748 1 0 Apr29 ? 00:00:00 perpboot -d /etc/perp
root 144749 144748 0 Apr29 ? 00:00:00 \_ tinylog -k 8 -s 100000 -t -z /var/log/perp/perpd-root
root 144750 144748 0 Apr29 ? 00:00:00 \_ perpd /etc/perp
root 2492482 144750 0 May14 ? 00:00:00 \_ tinylog (gearmand) -k 10 -s 100000000 -t -z /var/log/perp/gearmand
gearmand 2492483 144750 0 May14 ? 00:00:08 \_ /usr/sbin/gearmand -L 127.0.0.1 -p 4730 --verbose INFO --log-file stderr --keepalive --keepalive-idle 120 --keepalive-interval 120 --keepalive-count 3 --round-robin --threads 36 --worker-wakeup 3 --job-retries 1
root 2531800 144750 0 May14 ? 00:00:00 \_ tinylog (gds-rel-worker) -k 10 -s 100000000 -t -z /var/log/perp/gds-rel-worker
gds 2531801 144750 0 May14 ? 00:00:00 \_ /usr/bin/gearman -h 127.0.0.1 -t 1000 -w -f gds-rel -- xargs /home/gds/gds-rel-worker.sh
gds 2531880 2531801 0 May14 ? 00:00:00 \_ [xargs] <defunct>
So far I have traced the problem to run.sh, because if I replace its call with something simpler (e.g. echo "Hello"; sleep 5) the worker does not hang. Unfortunately, I have no clue what is causing the problem. The script run.sh is rather long and complex, but has been working without a problem so far. Tracing the worker process I see this:
getpid() = 2531801
write(2, "gearman: ", 9) = 9
write(2, "gearman_worker_work", 19) = 19
write(2, " : ", 3) = 3
write(2, "gearman_wait(GEARMAN_TIMEOUT) ti"..., 151) = 151
write(2, "\n", 1) = 1
sendto(5, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
recvfrom(5, "\0RES\0\0\0\n\0\0\0\0", 8192, MSG_NOSIGNAL, NULL, NULL) = 12
sendto(5, "\0REQ\0\0\0\4\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
poll([{fd=5, events=POLLIN}, {fd=3, events=POLLIN}], 2, 1000) = 1 ([{fd=5, revents=POLLIN}])
sendto(5, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
recvfrom(5, "\0RES\0\0\0\6\0\0\0\0\0RES\0\0\0(\0\0\0QH:terra-"..., 8192, MSG_NOSIGNAL, NULL, NULL) = 105
pipe([6, 7]) = 0
pipe([8, 9]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fea38480a50) = 2531880
close(6) = 0
close(9) = 0
write(7, "1.2.3\n", 18) = 6
close(7) = 0
read(8, "which: no terraform-0.14 in (/us"..., 1024) = 80
read(8, "Identity added: /home/gds/.ssh/i"..., 1024) = 54
read(8, 0x7fff6251f5b0, 1024) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2531880, si_uid=1006, si_status=0, si_utime=0, si_stime=0} ---
read(8,
So the worker continues reading standard output even though the child has finished successfully and presumably closed it. Any ideas how to catch what causes this problem?
I was able to solve it. The script run.sh was starting ssh-agent, which opens a socket and since Gearman redirects all outputs the worker continued reading the open file descriptor even after the script successfully completed.
I found it by examining the open file descriptors for the Gearman worker process after it hang:
# ls -l /proc/2531801/fd/*
lr-x------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/0 -> /dev/null
l-wx------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/1 -> 'pipe:[9356665]'
l-wx------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/2 -> 'pipe:[9356665]'
lr-x------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/3 -> 'pipe:[9357481]'
l-wx------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/4 -> 'pipe:[9357481]'
lrwx------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/5 -> 'socket:[9357482]'
lr-x------. 1 gds devops 64 May 17 11:26 /proc/2531801/fd/8 -> 'pipe:[9369888]'
Then identified the processes using file node for the pipe in file descriptor 8 that German worker continued reading:
# lsof | grep 9369888
gearman 2531801 gds 8r FIFO 0,13 0t0 9369888 pipe
ssh-agent 2531899 gds 9w FIFO 0,13 0t0 9369888 pipe
And finally listed files opened by ssh-agent and found what stands behind file descriptor 3:
# ls -l /proc/2531899/fd/*
lrwx------. 1 root root 64 May 17 11:14 /proc/2531899/fd/0 -> /dev/null
lrwx------. 1 root root 64 May 17 11:14 /proc/2531899/fd/1 -> /dev/null
lrwx------. 1 root root 64 May 17 11:14 /proc/2531899/fd/2 -> /dev/null
lrwx------. 1 root root 64 May 17 11:14 /proc/2531899/fd/3 -> 'socket:[9346577]'
# lsof | grep 9346577
ssh-agent 2531899 gds 3u unix 0xffff89016fd34000 0t0 9346577 /tmp/ssh-0b14coFWhy40/agent.2531898 type=STREAM
As a solution I added kill of the ssh-agent before exit from run.sh script and now there are no jobs hanging due to zombie process.
I have the following timey.cpp code in RedHat 2.6.32 x86_64:
using namespace std ;
int main()
{
while( 1 ){
char x[64]={0} ;
strcpy( x,"1234567890") ;
std::string s = x ;
std::cout << "(" << x << ")" << std::endl ;
struct timeval localtimex ;
long secs,usecs ;
gettimeofday(&localtimex,0x00) ;
secs = localtimex.tv_sec ;
usecs = localtimex.tv_usec ;
//long mills = (time.tv_sec * 1000) + (time.tv_usec / 1000 ) ;
printf("secs=(%d),usecs=(%d)\n",secs,usecs) ;
sleep( 1 ) ;
} //while
}
in /home/informix/test, compiled by g++ --std=c++0x timey.cpp -o timey.exe,
and the shell timey.sh:
source /etc/bashrc
nohup /home/informix/test/timey.exe &
Then I run timey.sh by:
/home/informix/test/timey.sh
and take a look if timey.exe runs by:
ps -ef | grep timey
It seems timey.exe runs as expected:
informix 41340 1 0 10:32 pts/10 00:00:00 /home/informix/test/timey.exe
What confuses me is that I add this shell to crontab:
38 10 * * 1-5 informix /home/informix/test/timey.sh
and restart crond:
/etc/init.d/crond restart
What surprises me is I see 4 copies of timey.exe running:
ps -ef | grep timey
informix 41498 1 0 10:38 ? 00:00:00 /home/informix/test/timey.exe
informix 41499 1 0 10:38 ? 00:00:00 /home/informix/test/timey.exe
informix 41529 1 0 10:38 ? 00:00:00 /home/informix/test/timey.exe
informix 41561 1 0 10:38 ? 00:00:00 /home/informix/test/timey.exe
What did I do wrong so that 4 copies of timey.exe are running?
I see the following in /var/log/cron:
Jul 2 10:38:01 localhost CROND[41440]: (informix) CMD (/home/informix/test/timey.sh )
Jul 2 10:38:01 localhost CROND[41439]: (informix) CMD (/home/informix/test/timey.sh )
Jul 2 10:38:01 localhost CROND[41491]: (informix) CMD (/home/informix/test/timey.sh )
Jul 2 10:38:01 localhost CROND[41533]: (informix) CMD (/home/informix/test/timey.sh )
It seems that crond really ran timey.sh 4 times, but why?
Also:
In RedHat 2.6.32-279.el6.x86_64, it works!
In RedHat 2.6.32-358.el6.x86_64, it does not work!
In command below I enable file /dev/tcp/10.10.10.1/80 both for reading and writing and associate it with file descriptor 3:
$ time exec 3<>/dev/tcp/10.10.10.1/80
bash: connect: Operation timed out
bash: /dev/tcp/10.10.10.1/80: Operation timed out
real 1m15.151s
user 0m0.000s
sys 0m0.000s
This automatically tries to perform TCP three-way handshake. If 10.10.10.1 is not reachable as in example above, then connect system call tries to connect for 75 seconds. Is this 75 second timeout determined by bash? Or is this system default? Last but not least, is there a way to decrease this timeout value?
It's not possible in Bash without modifying the source as already mentioned, although here is the workaround by using timeout command, e.g.:
$ timeout 1 bash -c "</dev/tcp/stackoverflow.com/80" && echo Port open. || echo Port closed.
Port open.
$ timeout 1 bash -c "</dev/tcp/stackoverflow.com/81" && echo Port open. || echo Port closed.
Port closed.
Using this syntax, the timeout command will kill the process after the given time.
See: timeout --help for more options.
It is determined by TCP. It can be decreased on a per-socket basis by application code.
NB The timeout only takes effect if there is no response at all. If there is a connection refusal, the error occurs immediately.
No: there is no way of changing timeout by using /dev/tcp/
Yes, you could change default timeout for TCP connection in any programming language.
But, bash is not a programming language!
You could have a look into source code (see: Bash Homepage), you may find lib/sh/netopen.c file where you could read in _netopen4 function:
s = socket(AF_INET, (typ == 't') ? SOCK_STREAM : SOCK_DGRAM, 0);
You could read this file carefully, there are no consideration of connection timeout.
Without patching bash sources, there is no way of changing connection timeout by a bash script.
Simple HTTP client using netcat (near pure bash)
There is a little sample HTTP client written in pure bash, but using netcat:
#!/bin/bash
tmpfile=$(mktemp -p $HOME .netbash-XXXXXX)
exec 7> >(nc -w 3 -q 0 stackoverflow.com 80 >$tmpfile)
exec 6<$tmpfile
rm $tmpfile
printf >&7 "GET %s HTTP/1.0\r\nHost: stackoverflow.com\r\n\r\n" \
/questions/24317341/how-to-decrease-tcp-connect-system-call-timeout
timeout=100;
while ! read -t .001 -u 6 status ; do read -t .001 foo;done
echo STATUS: $status
[ "$status" ] && [ -z "${status//HTTP*200 OK*}" ] || exit 1
echo HEADER:
while read -u 6 -a head && [ "${head//$'\r'}" ]; do
printf "%-20s : %s\n" ${head%:} "${head[*]:1}"
done
echo TITLE:
sed '/<title>/s/<[^>]*>//gp;d' <&6
exec 7>&-
exec 6<&-
This could render:
STATUS: HTTP/1.1 200 OK
HEADER:
Cache-Control : private
Content-Type : text/html; charset=utf-8
X-Frame-Options : SAMEORIGIN
X-Request-Guid : 46d55dc9-f7fe-425f-a560-fc49d885a5e5
Content-Length : 91642
Accept-Ranges : bytes
Date : Wed, 19 Oct 2016 13:24:35 GMT
Via : 1.1 varnish
Age : 0
Connection : close
X-Served-By : cache-fra1243-FRA
X-Cache : MISS
X-Cache-Hits : 0
X-Timer : S1476883475.343528,VS0,VE100
X-DNS-Prefetch-Control : off
Set-Cookie : prov=ff1129e3-7de5-9375-58ee-5f739eb73449; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
TITLE:
bash - How to decrease TCP connect() system call timeout? - Stack Overflow
Some explanations:
We create first a temporary file (under private directory for security reason), bind and delete before using them.
$ tmpfile=$(mktemp -p $HOME .netbash-XXXXXX)
$ exec 7> >(nc -w 3 -q 0 stackoverflow.com 80 >$tmpfile)
$ exec 6<$tmpfile
$ rm $tmpfile
$ ls $tmpfile
ls: cannot access /home/user/.netbash-rKvpZW: No such file or directory
$ ls -l /proc/self/fd
lrwx------ 1 user user 64 Oct 19 15:20 0 -> /dev/pts/1
lrwx------ 1 user user 64 Oct 19 15:20 1 -> /dev/pts/1
lrwx------ 1 user user 64 Oct 19 15:20 2 -> /dev/pts/1
lr-x------ 1 user user 64 Oct 19 15:20 3 -> /proc/30237/fd
lr-x------ 1 user user 64 Oct 19 15:20 6 -> /home/user/.netbash-rKvpZW (deleted)
l-wx------ 1 user user 64 Oct 19 15:20 7 -> pipe:[2097453]
$ echo GET / HTTP/1.0$'\r\n\r' >&7
$ read -u 6 foo
$ echo $foo
HTTP/1.1 500 Domain Not Found
$ exec 7>&-
$ exec 6>&-