We're having a connection timeout issue from an API pooling connections to an informix connection manager which forwards the queries to the appropriate informix database server.
Recently, I've set up the mail service and realized that we're having delays in receiving the mail send and after troubleshooting I saw that the database server is not syncronized at all with the API ( 2+ minutes difference ).
I've read somewhere that time sync is important when using jdbc pooling but I can't find to much information regarding this on internet. The timeout kinda makes sense because of the tcp keepalive.
Had anyone experienced or know about this ?
Thank you,
Mihai.
It is common to intermix database timestamps and local timestamps. This causes issues when the server times are different. If the mail server is looking for records before the current time, there could be a two minute delay before mail is sent.
Email may be delayed in transit between servers. Check the Received headers to see if there are any unexpected delays. (You will need to compensate for time variances on the servers.
Normally, you would use NTP to ensure the time is the same on all servers. Within a data center it should be able to synchronize times to a millisecond or so.
Related
Experimented with server sent events and all the examples i saw is to store the connection in-memory, in this way spring can track the active connections and will keep the http connection open.
Is there an alternative of persisting this connections? This is due to the following reasons:
We are targeting 100 thousands of users, and keeping this number of connections might impact greatly on memory of the services.
When the application server crash, connection will not be impacted since they will be stored externally. I know this is not the right approach since connection is not meant to be keep long and should have a certain timeout applied.
I am struggling to find information on how to gauge the scalability of websockets. A scenario -
Let's say from client wants to establish socket connection from a browser, and the client application and service layer (Micronaut) both have two instances behind an elb - service layer will sit us-east region and can expect anyone from around the world can access the frontend app from browser and can expect an open connection for an avg of 2-5 min, no longer than 30 minutes.
Is there a ballpark number on how many concurrent websocket connections a couple servers can handle? Or if there are certain factors that I didn't mention that are vital to handling websocket connections in general?
Thank you in advance.
I'm assuming you want to know the scalability of the implementation of WS in Micronaut and not WS in general. Of course, the scalability of WS is dependent on the specific implementation and WS itself. You probably already know this, but wanted to state it for the record. You may also want to be sure you increase your file descriptors for your server process to the max number (you may have to adjust your kernel to increase the FDs).
Btw, don't forget to handle retries and reconnects as you would for a low-level TCP connection
We have a node instance that has about 2500 client socket connections, everything runs fine except occasionally then something happens to the service (restart or failover event in azure), when the node instances comes back up and all socket connections try to reconnect the service comes to a halt and the log just shows repeated socket connect/disconnects. Even if we stop the service and start it the same thing happens, we currently send out a package to our on premise servers to kill the users chrome sessions then everything works fine as users begin logging in again. We have the clients currently connecting with 'forceNew' and force web sockets only and not the default long polling than upgrade. Any one ever see this or have ideas?
In your socket.io client code, you can force the reconnects to be spread out in time more. The two configuration variables that appear to be most relevant here are:
reconnectionDelay
Determines how long socket.io will initially wait before attempting a reconnect (it should back off from there if the server is down awhile). You can increase this to make it less likely they are all trying to reconnect at the same time.
randomizationFactor
This is a number between 0 and 1.0 and defaults to 0.5. It determines how much the above delay is randomly modified to try to make client reconnects be more random and not all at the same time. You can increase this value to increase the randomness of the reconnect timing.
See client doc here for more details.
You may also want to explore your server configuration to see if it is as scalable as possible with moderate numbers of incoming socket requests. While nobody expects a server to be able to handle 2500 simultaneous connections all at once, the server should be able to queue up these connection requests and serve them as it gets time without immediately failing any incoming connection that can't immediately be handled. There is a desirable middle ground of some number of connections held in a queue (usually controllable by server-side TCP configuration parameters) and then when the queue gets too large connections are failed immediately and then socket.io should back-off and try again a little later. Adjusting the above variables will tell it to wait longer before retrying.
Also, I'm curious why you are using forceNew. That does not seem like it would help you. Forcing webSockets only (no initial polling) is a good thing.
We had issues with lot of Applications connecting to MQ server without properly doing a disconnect. Hence we introduced DISCINT on our server connection channels with a value 1800 sec which we found ideal for our transactions. But our Keep Alive interval is pretty high with 900 sec. We would like to reduce that less than 300 as suggested by mqconfig util. But before doing that I would like to know if this is going to affect our disconnect interval value and whether it is going to override our disconnect interval value and make more frequent disconnects which will be a performance hit for us.
How does both these values work and how they are related?
Thanks
TCP KeepAlive works below the application layer in the protocol stack, so it does not affect the disconnecting of the channel configured by the DISCINT.
However lowering the value can result in more frequent disconnects, if your network is unreliable, for example has intermittent very short (shorter then the current KeepAlive, but longer then the new) periods when packets are not flowing.
I think the main difference is, that DISCINT is for disconnecting a technically working channel, which is not used for a given period, while KeepAlive is for detecting a not working TCP connection.
And MQ provides means to detect not working connections in the application layer too, configured by the heartbeat interval.
These may help:
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q015650_.htm
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ref.con.doc/q081900_.htm
http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ref.con.doc/q081860_.htm
I have the following system:
A Windows 2003 server running WebSphere Application Server, listening on port 8080.
A lot of clients of this server.
I tried a loads test - making clients connect to the server and asking for services. This didn't end well: Many clients were denied service and the server started reporting it was unable to create new sockets.
My question is which parameters should I change in my Windows?
I thought about number of connections, but I am not sure this exists on 2003 (from what I have read). Instead, there is a number of userPorts, which I don't think is what I need, since I am only using one port (8080) on the server side.
Am I wrong assuming that I am only using one port in the server side?
Are there parameters for number of connections per port, per system, or perhaps this is affected by the amount of data transferred. I pass a lot of data, so a reference to amount of data (if there is such a parameter that might limit, I am glad to hear it).
Should I also reduce the amount of wait each connection waits after tear down? This may allow the pool of connections to be more available. If so which Parameter is this?
Any other parameters that are consistent with this problem?