I am setting up load test for SaaS platform.
I want to benchmark 20 clients connect to SaaS Platform and push some data.
Each client can send maximum of 2 MB and average of 200 Bytes data to SaaS endpoint.
Which one testing is better setting up 20 Clients on single machine or setting up 20 Clients on 5 different machines?
I want to know from TCP stack point of view?
Now when we run 20 clients in a single machine, it will create connection to same destination address and destination port but 20 different Source Ports.
However in background will it use same TCP connection to push data of 20 clients?
From "TCP stack" point of view one "client" == one "connection". If the server doesn't have any background logic to check source IP address in order to restrict requests rate - you can go for a single machine.
See Connection establishment for more information.
In general you need to mimic real life usage of the SaaS platform by end users (or upstream/downstream systems) as close as possible so carefully choose a load testing tool which can produce the same network footprint in terms of creating connection, re-using it and keeping it alive
Related
I am building TCP Proxy: client <-> proxy <-> Vertica
I have a net.TCPListener, which takes incoming requests by AcceptTCP() and creating connections, then, making connection to destination socket by net.DialTCP("tcp", nil, raddr). Looks like a bridge. Default proxy model.
Firstly, at first version, i have a trouble: if i have 59 parallel incoming request, everything is fine. But if i have one more (60), i have a trouble: 1-59 connections are OK, but 60 and newer are fault. I cant catch error properly. Looks like some socket unexpectedly closes
Secondly, i tried to set queue for listener. It helps me a lot: but if i have more than 258 requests, i get error again.
My question: is there any limit of connections in net package? May be it is system limitation?
For external info: Vertica running in docker container, hw/system: macbook, vertica limit connection pool: 5, but pool logic implemented into proxy.
I also tried set "raw" proxy without pool logic (thats why i set queue for listener: i must not exceed threshold of Vertica User's pool), result is 258 requests..
UPDATED: (05.04.2020)
Looks like it is system limitations fault. Did I mention anywhere that I trying to run the whole system on one PC?
So, what I had:
300 parallel processes as requests (making by multiprocessing.Pool
Python) (300 sockets)
Listener that creates 300 connections (once
more 300 sockets)
And series of rapidly creating/closing sockets in
deep of proxy (according to queue and Vertica pool)
What I have now:
300 python requests making from another PC in my local network (on Windows)
Proxy works fine
But I have several errors on Windows PC, which creating requests to my proxy. Errors like low memory in "swap file".
I still need to make some stress test for proxy. Adding less memory for swap file didn't solve my problem on Windows PC. I will be grateful for any suggestions and ideas. Thanks!
How does the proxy connect to Vertica?
There is by default a maximum of 50 ordinary mortal users to be connected to one Vertica node at any one time. The superuser "dbadmin" always has 5 connections in addition to that.
So if I try to connect 60 times as dbadmin, I get this on a default Vertica configuration:
Connection attempt failed: FATAL 4060: New session rejected due to limit, already 55 sessions active
You can increase the Vertica config item MaxClientSessions from its default of 50 per node.
Command is : ALTER NODE <_node_name_> SET MaxClientSessions = 100, for example.
I suppose you are always connecting to the same Vertica node, and that you have set ConnectionLoadBalancing to FALSE. So you always connect to the same node, and soon reach the default maximum of 50.
Hope that's the reason found ....
It is a SpringBoot website and deployed in one Linux server. We use Jmeter to do the load test.
We mock 500 users to visit the webiste index page simultaneously. The index page is very simple html, no database connection,so it is a quite short connection.
After about 2 minutes, Jmeter starts to throw timeout exception as bleow
I guess this is because of website reaching its capacity and running out of connection.
I get one quesiton here, why does website reach its capacity 2 minutes later after Jemter starts. If its TCP connection capacity for this website is 1000, I guess it will reach 1000 very soon after the Jmeter starts, not 2 minutes.
Besides, I see many TCP connections are in TIME_WAIT status in Linux server. I guess this may be related with the connection timeout?
Edit: Someone thinks it is running of port. Someone thinks it is running out of connection. And someone thinks it is running out of processing thread(eg. What does this messge java.net.ConnectException/Connection timed out mean in log.jtl file of Jmeter?). I don't know which one is the exact reason...
Most probably this is due to underlying Linux TCP/IP kernel stack configuration, as per Linux TCP/IP tuning for scalability article:
By default, a connection is supposed to stay in the TIME_WAIT state for twice the msl. Its purpose is to make sure any lost packets that arrive after a connection is closed do not confuse the TCP subsystem (the full details of this are beyond the scope of this article, but ask me if you’d like details). The default msl is 60 seconds, which puts the default TIME_WAIT timeout value at 2 minutes. Which means you’ll run out of available ports if you receive more than about 400 requests a second, or if we look back to how nginx does proxies, this actually translates to 200 requests per second. Not good for scaling.
SO double check timeouts along with maximum number of ports/sockets/files on the Linux server - my expectation is that the aforementioned parameters need to be tuned for high loads.
It's also a good practice to have monitoring of baseline OS health metrics in place (CPU, RAM, Network, Disk, swap usage, etc.). You can use i.e. JMeter PerfMon Plugin or JMeter SSHMon Listener for this.
alright i am trying to understand this approach , lets say i have run 2 servers
Server A is on Ip 1.1.1.1 and port 36663
Server B is on ip 2.2.2.2 and port 54223
i am asking to be able to understand this approach
can i make clients on server A be able to communicate with clients on Server B ?
as example a client is connecting on Server A and want to send some data to a client who connecting to server B is this can be done using indy tcp server ?
if the answer is yes an example will be much helpful to fully understand this approach .
i have 2 servers on different machine
1 machine have some slow network issue and the other have good network.
the logic here is when the client to serverA that takes more than 20 seconds to connect, during this 20 seconds try to reconnect to other server ip and be able to communicate with the client that already coonnected on serverA
TIdTCPServer has a Bindings property, which is a collection of IP/Port pairs that the server listens on. You can have a single TIdTCPServer object listening on multiple IP/Port pairs, or you can use multiple TIdTCPServer objects listening on different pairs, on the same machine.
Either way, the connected clients are stored in the TIdTCPServer.Contexts property.
When a client wants to send data to another client, regardless of which server IP/Port it is connected to, all you have to do is iterate through the Contexts list of the appropriate TIdTCPServer object until you find the TIdContext object of the target client, and then you will have access to its Connection.IOHandler property.
On the other hand, if you have separate TIdTCPServer objects running on different machines, clients cannot directly communicate with clients on another server. You would have to establish a connection between the two servers and then you can proxy any client-to-client data through that connection as needed.
I am testing my application using jmeter tool. I am having 2 EC2 m1.small instances behind an ELB (not autoscaling group), 2 caching nodes huge RDS db (Multi-AZ). My Apache (peforked) is configured with default values like 256 maxclients. Both can handle 256-256 request. Now when jmeter throws 500 request I see connection time out error in one of sampler of the jmeter. Can anyone figure out what the problem is?
Thanks in advance.
What does JMeter throw back?
There are couple of things that could happen.
Connection reset: If this error is thrown out from your JMeter then it means that the server has maxed out and cannot support any more concurrent requests. i.e. the 256 threads allotted are all in use serving other requests and this basically means you have hit your limit on the server.
"Address already in use" exception: These kinds of errors are what you must not get when you do load tests. This basically means that there are no available ports on your system make more requests and all ports are busy. This could happen for a variety of reasons but you could try tweaking system settings like ulimit for linux or if your using a windows box you may want to look at the tcpTimedWaitDelay and corresponding settings to see the average turn around time for the ports to be handed over back into the active pool to be reused for the next connection. This condition is called tcp port exhaustion (http://www.outsystems.com/NetworkForums/ViewTopic.aspx?TopicId=6956&Topic=How-to-tune-the-TCP%2FIP-stack-for-high-volume-of-web-requests)
TO get around this you could also try distributed load testing and/or use timers to ensure that you always have the ports to make new connections.
Is there a limit to the number of HTTP ports in a machine. I have a windows application that uses .NET Remoting. Each instance of the application, exposes a Remote object on load, through a HTTP Channel with port 0 (so that port can be decided dynamically). In a Multi user environment, will there be a limit to the number of HTTP Ports.
Thanks in Advance!
Yes there will be a limit to the number of ports available which is 65535 minus the number of ports already in use for existing services (for example, SMTP [25], HTTPS [443], SQL Server [1433], etc).
So on a typical Windows server, a finger in the air calculation would be 65535 - 1024 (the well know service ports <= 1024 which are considered out of bounds) - another 10-20 or so possible other application (SQL Server, MySQL, Oracle, etc). This would leave around 64490 post available.
However will you really be running 64000 of instances of your server?