Aws S3 wget vs chunked download speeds - performance

My application is trying to read a file in chunks of 2 MB from S3 using S3 java SDK. Its a multi threaded app that downloads distinct chunks. I am running this app on a m5.xlarge instance (4 cores, 16 Gig RAM).
The maximum download bandwidth I am getting using chunked download is ~30 MBps whereas if I use wget on the same file after making it public, I get speeds up to 100 MBps. I looked at iftop and tcptrack's network monitoring and it looks like even with the chunked parallel download the same connections are being reused.(same ports are being used) What can be the possible reasons for this and how can I improve my read performance?
This is what I get from my application
Downloaded 500 chunks in 34seconds bandwidthInMBps: 29.41176470588235
ChunkSizeInMB: 2, threadCount: 16
This is from wget
rksupport#bolt:~$ wget https://s3-us-west-1.amazonaws.com/aXXXXXXXXXX/upload.tar.gz
--2018-09-19 22:36:40-- https://s3-us-west-1.amazonaws.com/XXXXXXXXX/upload.tar.gz
Resolving s3-us-west-1.amazonaws.com (s3-us-west-1.amazonaws.com)... 54.231.236.29
Connecting to s3-us-west-1.amazonaws.com (s3-us-west-1.amazonaws.com)|54.231.236.29|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3122206720 (2.9G) [application/x-gzip]
Saving to: 'upload.tar.gz'
upload.tar.gz 100%[==========================================================================================================================================================================>] 2.91G 91.6MB/s in 31s
2018-09-19 22:37:11 (95.1 MB/s) - 'upload.tar.gz' saved [3122206720/3122206720]

Have you tried configuring S3 Transfer acceleration?
Do you have the S3 Endpoint in your VPC?
Both of those should make your downloads faster.

Related

TFTP larger files fails to be sent

I am sending a set of .bin files via TFTP from Windows server to Linux client. When I send files < 50KB, the file is being sent successfully but any larger files is not being sent.
I use Python socket.py module to send and receive files and acknowledgments respectively.
I am thinking in the following directions :
(1) MTU - buffer size - (currently changed to 9000)
(2) Firewall preventing larger files ?
(3) Duplex settings mismatch - (currently set to 100 Mbps FULL Duplex - does not work under autonegotiation)
(4) Any configurations specific to Windows (the same file is sent successfully from Linux machine tftp server)
What could be the possible problems? Please help me narrow down the scope of the issue.

Connection reset on heavy load using jmeter

I'm trying to perform load 15000 concurrent requests on my application server(linux red hat) which has 16 core and 32 GB RAM through 2 jmeter server(windows server 2012) with configuration of 16 core and 32 GB RAM and getting this error "Non HTTP response code: java.net.SocketException/Non HTTP response message: Connection reset"
I've already followed this link related to configuration but couldn't find any solution Java.net.SocketException connection reset error in .net application
First i was using single Jmeter server(Window server 2012) for 15000 request and i split this into two servers 7500 concurrent request each with configuration of 16 core and 32 GB RAM but getting same issue on "Non HTTP response code: java.net.SocketException/Non HTTP response message: Connection reset"
It is a socket issue on your server end.
Go to /etc/security/limits.conf
and update soft limit to a higher value.
For example:
vi /etc/security/limits.conf
set ulimit of soft and hard to maximum count

lsass.exe, Https, IIS and High Concurrent Connections

I have Windows Server 2012 r2, Hosting multiple websites on http and https protocols.
when the current connections reach 26000 the https is not served although the http is serving normally .
After the 36000 connections the http also is not serving any website.
also the lsass process takes about 20 to 25 percent of the cpu staring from 26000 connections.
is there a way to increase the number of connections served or do I need another server behind load balance.
I changed the SSL certificate RSA key size from 4048 to 2024 and it works just fine .
The reason that the processor take too much calculations to encrypt and decrypt bigger key size.

TCP vs UDP bandwidth contention

Consider this scenario:
I've 10 Mbps internet connection. One user is downloading 1GB file from internet over TCP called TCP.file and another 1GB file from internet using UDP called UDP.file.
Assume there are no issues with the server (or upstream) serving files over TCP & UDP. There is no packet inspection or anything, just plain router.
Which download will complete first? Will it be TCP.file or UDP.file? And why?
I've two arguments:
1) TCP.file will compete first. Because TCP has flow control and can gobble up entire 10 Mbps BW.
2) UDP.file will compete first. Since UDP has less overhead and no flow control, this can also send consume entire 10 Mbps BW.
Can anyone tell which is correct?

web server running inside a docker container running inside an EC2 instance responses very slowly

I have a web server running inside a docker container in AWS EC2 Ubuntu instance. When I send requests to the web server, I get the response very slowly (20+ seconds most of the times, although the response time varies). It does not time-out though. The web server is a very lightweight. It is only to test, so almost does not do anything.
docker version 17.03.0-ce
docker-compose version 1.12.0-rc1
How I debugged so far
When sending requests to the web server running in the docker container from within the EC2 instance (url = ' http:// localhost:xxxx/api ') it is still very slow. So should not be related to sending requests from outside.
I run another web server inside the EC2 directly (not in a docker container), and it is not slow. It responses very fastly.
I run another web server inside another docker container in EC2, and it is also very slow!
When I send the request from inside the docker container to the web server that is running in it (at its localhost), it is also very slow!
I run the containers with the same command on my mac computer and the get response is not slow!
Here is one of the containers stats:
CPU %: 0.28%
MEM USAGE / LIMIT: 27.49 MiB / 992.5 MiB
MEM %: 2.77%
NET I/O: 53.7 kB / 30.5 kB
BLOCK I/O: 2.24 MB / 0 B
I understand it might be very hard to know the issue. My question is the steps to debug the cause and finally find the solution. I appreciate if you could explain your approach in detail.
This sounds like a name resolution problem.
To debug this, you can do different things.
You can first start a simple tcp server with nc -l -p 8000 within the docker container (which is started -p 8000:8000), and the on the host launch: nc 127.0.0.1 8000, type some charachter, see if the TCP communication works, they should appear within the container.
Next, you can do the same as before but using "localhost" instead of 127.0.0.1
After this, you can do the same HTTP request you did, but using 127.0.0.1 instead of localhost (this will set the request's Host: header to the same value, which the webserver might not check , or might resolve more easily).
You can also have have a look at the generated /etc/hosts and /etc/resolv.conf within the container. Maybe they do not make sense in the network context of your container.
Also, you can precisely measure the time needed for your requests, if they are near a precise second, this sounds once more like a DNS timeout (if it's 5.003, 10.200, 20.030 seconds, it's like a X seconds timeout, plus the real time needed to respond).

Resources