I provisioned two i2.8xlarge VMs on AWS, which should have 10G network as it said on the portal.
But when I run iperf3 test (with "-P 64", which means run with 64 TCP connections) on the VMs, it just gave me ~5Gbps throughput. At the same time when running iperf3, the icmp ping latency is relatively high (about 19ms).
//by the way, iperf3 with 1 TCP connection, gave me the best throughput number, which is about 7Gbps.
Do I miss any configuration/setting to my AWS deployment so that I am not able to get the high throughput and reasonable low latency?
Thanks!
Related
In K6, I'm observing more fail request in my performance test execution with dial tcp : I/O timeout. Please suggest any fine tuning if I missed at K6.
With low concurrent let’s with 225 users no issues but when increase user to 300 am facing this issue and I'm using MacBook for the test execution
This error indicates that your server under test isn't able to keep up with TCP connection attempts from k6, which is usually a hint that you're reaching the performance limits of what your server is able to deliver.
At this point you would have to tweak your server settings or increase app performance to reach the levels you're aiming for. One sanity check you can do on the server side is confirm if the maximum number of file descriptors (which includes network sockets) is sufficient for your test. See ulimit.
It is a SpringBoot website and deployed in one Linux server. We use Jmeter to do the load test.
We mock 500 users to visit the webiste index page simultaneously. The index page is very simple html, no database connection,so it is a quite short connection.
After about 2 minutes, Jmeter starts to throw timeout exception as bleow
I guess this is because of website reaching its capacity and running out of connection.
I get one quesiton here, why does website reach its capacity 2 minutes later after Jemter starts. If its TCP connection capacity for this website is 1000, I guess it will reach 1000 very soon after the Jmeter starts, not 2 minutes.
Besides, I see many TCP connections are in TIME_WAIT status in Linux server. I guess this may be related with the connection timeout?
Edit: Someone thinks it is running of port. Someone thinks it is running out of connection. And someone thinks it is running out of processing thread(eg. What does this messge java.net.ConnectException/Connection timed out mean in log.jtl file of Jmeter?). I don't know which one is the exact reason...
Most probably this is due to underlying Linux TCP/IP kernel stack configuration, as per Linux TCP/IP tuning for scalability article:
By default, a connection is supposed to stay in the TIME_WAIT state for twice the msl. Its purpose is to make sure any lost packets that arrive after a connection is closed do not confuse the TCP subsystem (the full details of this are beyond the scope of this article, but ask me if you’d like details). The default msl is 60 seconds, which puts the default TIME_WAIT timeout value at 2 minutes. Which means you’ll run out of available ports if you receive more than about 400 requests a second, or if we look back to how nginx does proxies, this actually translates to 200 requests per second. Not good for scaling.
SO double check timeouts along with maximum number of ports/sockets/files on the Linux server - my expectation is that the aforementioned parameters need to be tuned for high loads.
It's also a good practice to have monitoring of baseline OS health metrics in place (CPU, RAM, Network, Disk, swap usage, etc.). You can use i.e. JMeter PerfMon Plugin or JMeter SSHMon Listener for this.
I have a high load application which spins in IIS in Windows Server 2012. How to determine how many simultaneius tcp connections can windows handle and how to increase it? I already change TcpNumConnections and MaxUserPort to maximum allowed values, and TcpTimedWaitDelay to minimum. It helps to increase throughput of VM, but is it possible to configure something for better perfomance? Does it depend on SynAttackProtection or something else? And what should be tcp autotuninglevel for maximum performance?
I am trying to measure latency to a server that I don't control. This is in a colocated environment, so the latency is on the order of 500 us (.5 ms).
I understand that Cisco gear frequently deprioritizes ICMP traffic, making ping times unreliable. Is there a way for me to tell if this is the case on the gear I am traversing?
Can I use TCP acknowledgements to determine the minimum latency to the remote server? To do this, I would somehow need to force the remote server to send a TCP ack immediately on receiving my data.
Try hping. You can send acks and measure the latency:
hping -A -p 80 host
or with a SYN:
hping -S -p 80 host
Also note, deprioritization on a layer-2 link is unlikely (but possible). In addition, seeing ARP being slower than ICMP doesn't necessarily mean ICMP isn't deprioritized---it might mean bandwidth is insufficient to hit the limiting threshold.
ARP will almost always be slower because it broadcasts and may suffer port-queuing at the switch. You could unicast ARP, but that might look suspicious if anyone is looking for it.
You could try using arping, which does a ping using ARPs.
I need to test the performance of application running on localhost as if it were in the online environment. I mean the performance test conducted by the network traffic simulation, limited bandwidth simulation, or other parameter as if it were online.
Could Apache Ab do the simulation?
We've used Charles and Firefox Throttle in the past to simulate slow networks.
Why can't you connect to a different PC, or even use a virtual machine and rate limit the virtual network connection?
Yes, but you will need to connect to your application by IP address, not "localhost" or 127.0.0.1. Typically for web applications (HTTP) I use Fiddler which can simulate limited bandwidth, but only if you connect as I have noted. Other bandwidth limiters for non-HTTP I'm not sure/aware of.
Ab won't simulate low level network errors. If you're on linux, you can simulate some networking with 'tc'. See http://www.kdedevelopers.org/node/1878 for a small example.
You can set up a local tunnel to expose your localhost to the world, using Ngrok. From there you can use any number of online performance tools
You can throttle bandwidth via Chrome Dev Tools