I am supervising the Apache prefork server on Cent OS.
Apache hosts several ASP.NET Core services using proxy.
Usually apache serves 20 requests per second and server limit is 920 workers.
In case of busyness of one hosted service, incomming requests occupy all apache workers very fast.
In consequence of this, a request that are dedicated for another service i.e. virtualhost has no free worker.
Is it possible to reserve defined amount of workers for specific virtualhost?
Something like this:
subdomain1.myserver.com:443 - 200 Workers
subdomain2.myserver.com:443 - 200 Workers
subdomain3.myserver.com:443 - 200 Workers
subdomain4.myserver.com:443 - 200 Workers
myserver.com:4800 - 70 Workers
myserver.com:443 - 50 Workers
Thank you
Related
hi there i am trying to deploy laravel project on my deticated server which has two processors about 32 core and 128 giga ram
i am using apache longue as web server and using mpm winnt (this is the allowed mpm for windows
my problem is when i testing using jmeter i get about 90% of requests quicly but the rest requests take too long time to response and i can not understand why ??
it seems like those requests hold in queue but i really dont no why
my winnt conf is
<IfModule mpm_winnt_module>
ThreadsPerChild 1920
MaxConnectionsPerChild 1000000
</IfModule>
i am trying to get 2000 request per second
We have this Ignite cluster configuration: there are several servers + 1 client, which acts as a balancer. For example, IP addresses are:
server1 - 192.168.100.1
server2 - 192.168.100.2
server3 - 192.168.100.3
client - 192.168.100.100
So requests go to the client - for example http://192.168.100.100:8082/request1
Then the client sends the Distributed Computing Task to the cluster - the calculations are performed on one of the servers, so on 192.168.100.1 or 192.168.100.2 or 192.168.100.3. Results of calculations then return to client, then client finally sends response to the request.
There is no problem if one of the servers crashes - client would be knowing about that and wouldn't send Task on that server. But there is a problem if the client crashes - all servers work, but the address http://192.168.100.100:8080/request1 is not available.
What can I do about it? Can client be failover? Does Ignite have something for that? If not, what other technology/software can I use?
As far as I understand, Jelastic utilizes some kind of load balancer which is deployed behind the nodes.
So if I have two nodes and one node is down, how this load balancer will act?
I'm thinking about blue-green deployment without traffic distributor add-on.
By default, all the incoming traffic at Jelastic Platform is routed between containers via Shared Load Balancer. Herewith, this node performs constant servers’ health checkups, utilizing the NGINX upstream check module with the following settings for that:
interval=15000 rise=2 fall=3 timeout=2000 default_down=false;
In such a way, all containers are considered “up” from the startup, whilst the system verifies their availability each 15 seconds. If no response is received from a container within 2 seconds, such checkup is regarded as failed. Three consecutive fails will mark a node as “down”, whilst two successful checks in a row - as “up”.
As for the traffic distribution within a separate environment, a dedicated load balancer node is automatically added to its topology when the number of application server instances is set more than one (i.e. it’s scaled out horizontally). Jelastic PaaS provides 4 load balancer stacks you can choose between, each of which has some health check configuration specifics:
NGINX - runs a simple tcp check (i.e. verifies the required server port availability) right before routing a user request to it; if the check fails, the next node within a layer will be tried
HAProxy - performs regular tcp checks (every 2 seconds by default), storing the results in a table of backends state and keeping it constantly up-to-date
Apache Balancer - no implemented health check procedure by default
Varnish - all backends are assigned the probe = { .url = "/"; .timeout = 30s; .interval = 60s; .window = 5; .threshold = 2; } parameter within balancer configs, so that the health checks are performed once per minute with a 30 seconds timeout (see the official documentation for more information)
Obviously, the default health check settings can be manually adjusted up to your needs (through either Jelastic File Manager GUI or via SSH) according to the appropriate load balancer stack specification - refer to the official NGINX, HAProxy, Apache Balancer or Varnish documentation to see the details on possible settings.
If you have multiple web application nodes (e.g. Apache servers) you should also add your own load balancer node (in which case you have control over the health check parameters in Nginx or Apache configs).
It's also highly recommended to use public IPs for production use. Only use the Jelastic shared resolvers / load balancers for proxying requests to your environment during dev/test (they're shared infrastructure, so can be a bottleneck for your application as well as introducing other bugs / unwanted behaviours re. headers, protocols, ciphers etc.).
Windows Server 2003R2/2008R2/2012, Openfire 3.8.1, Hazelcast 1.0.4, MySQL 5.5.30-ndb-7.2.12-cluster-gpl-log
We've set up 5 servers in Openfire Cluster. Each of them in a different subnet, subnets are located in different cities and interconnected with each other through VPN routers (2-8 Mbps):
192.168.0.1 - node0
192.168.1.1 - node1
192.168.2.1 - node2
192.168.3.1 - node3
192.168.4.1 - node4
Openfire configured to use MySQL database which is successfully replicating from the master node0 to all slave nodes (each node uses it's own local database server, functioning as slave).
In Openfire Web Admin > Server Manager > Clustering we are able to see all cluster nodes.
Openfire custom settings for Hazelcast:
hazelcast.max.execution.seconds - 30
hazelcast.startup.delay.seconds - 3
hazelcast.startup.retry.count - 3
hazelcast.startup.retry.seconds - 10
Hazelcast config for node0 (similar on other nodes except for interface section) (%PROGRAMFILES%\Openfire\plugins\hazelcast\classes\hazelcast-cache-config.xml):
<join>
<multicast enabled="false" />
<tcp-ip enabled="true">
<hostname>192.168.0.1:5701</hostname>
<hostname>192.168.1.1:5701</hostname>
<hostname>192.168.2.1:5701</hostname>
<hostname>192.168.3.1:5701</hostname>
<hostname>192.168.4.1:5701</hostname>
</tcp-ip>
<aws enabled="false" />
</join>
<interfaces enabled="true">
<interface>192.168.0.1</interface>
</interfaces>
These are the only settings changed from default ones.
The problem is that XMPP clients are authorizing too long, about 3-4 minutes, after authorization other users in roster are inactive for 5-7 minutes, during this time logged in user in Openfire Web Admin > Sessions is marked as Offline. Even after user is able to see other logged in users as active, messages are not delivered, or delivered after 5-10 minutes or after few Openfire restarts...
We appreciate any help. We spent about 5 days trying to set up this monster, and are out of any ideas... :(
Thanks a lot in advance!
UPD 1: Installed Openfire 3.8.2 alpha with Hazelcast 2.5.1 Build 20130427 same problem
UPD 2: Tried starting the cluster on two servers that are in the same city, separated by probably 1-2 hops # 1-5ms ping. Everything works perfectly! Then we stopped one of those servers and started one in another city (3-4 hops # 80-100 ms ping) the problem occured again... Slow authorizations, logged off users in roster, messages are not delivered on time etc.
UPD 3: Installed Openfire 3.8.2 without JRE, and Java SDK 1.70_25.
Here are JMX screenshots:
node 0:
node 1:
Red line is the first client connection (after Openfire restart). Tested on two users. Same thing... First user (node0) connected instantly, second user (node1) spent 5 seconds on connection.
Rosters have been showing offline users on both sides for 20-30 seconds, then online users start appearing in them.
First user sends message to second user. Second user waits for 20 seconds, then receives first message. Reply and all other messages are transfered instantly.
UPD 4:
Durring the diggin through JConsole "Threads" tab we've discovered these various states:
For example hz.openfire.cached.thread-3:
WAITING on java.util.concurrent.SynchronousQueue$TransferStack#8a5325
Total blocked: 0 Total waited: 449
Maybe this could help... We actually don't know where to look for.
Thanks!
[UPDATE] Note per the Hazelcast documentation - WAN replication is supported in their enterprise version only, not in the community version that is shipped with Openfire. You must obtain an enterprise license key from Hazelcast if you would like to use this feature.
You may opt to setup multiple LAN-based Openfire clusters and then federate them using the S2S integration across separate XMPP domains. This is the preferred approach for scaling up Openfire for a very large user base.
[Original post follows]
My guess is that the longer network latency in your remote cluster configuration might be tying up the Hazelcast executor threads (for queries and events). Some of these events and queries are invoked synchronously within an Openfire cluster. Try tuning the following properties:
hazelcast.executor.query.thread.count (default: 8)
hazelcast.executor.event.thread.count (default: 16)
I would start by setting these values to 40/80 (5x) respectively to see if there is any improvement in the overall application responsiveness, and potentially even higher based on your expected load. Additional Hazelcast settings (including other thread pools) plus instructions for adding these properties into the configuration XML can be found here:
Hazelcast configuration properties
Hope that helps ... and good luck!
I am using JBoss AS 4.2.3 along with the seam framework. My CPU usage increases as the number of users increase and it hits 99% for just 80 users. We also use Hibernate, EJB3 and Apache with mod_jk for loadbalancing.
When I took the thread dump all the runnable threads are doing a same activity with the following trace:
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1012)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1091)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:384)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:662)
I am not able to interpret this with the stack trace. Also I find that even when the users have logged out, the CPU utilization still continues to be the same with threads in the same state.
These threads are attempting to read from a Socket connection. In this case they are waiting for the next request to be sent to the server from mod_jk in Apache. This is quite normal and they probably are not the reason for your CPU usage.
At this point you really need to go and run your application through a profiler.
If you are unable to run a profiler on the system (i.e. it's a production box) the next best thing is to start to take many stack dumps each a couple of seconds apart and then go though them by hand matching up the thread IDs. You need to look for the threads that are running your code and don't seem to have changed between dumps.
It is a very tedious task and doesn't always get clear results, but without a profiler or some sort of instrumentation you won't be able to find where all that CPU is going.
Review your AJP configuration between Apache and Jboss, as described in https://developer.jboss.org/wiki/OptimalModjk12Configuration
The problem
JBoss Web's (Tomcat) server.xml AJP snippet:
<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3"
emptySessionPath="true" enableLookups="false" redirectPort="8443" ></Connector> Apache's httpd.conf:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
The above configuration, under load, may cause mod_jk to be very slow
and unresponsive, cause http errors, and cause half-closed
connections. These problems can arise because there are no
connection timeouts specified to take care of orphaned connections, no
error handling properties defined in workers.properties, and no
connection limits set in Apache and Tomcat.
But this high number of threads could be from another source. As described here:
the most common scenario for a hanging Socket.read() is a high
processing time or unhealthy state of your remote service provider.
This means that you will need to communicate with the service provider
support team right away in order to confirm if they are facing some
slowdown condition on their system.
Your applicaton server Threads should be released once the remote
service provider system problem is resolved but quite often you will
need to restart your server instances (Java VM) to clear all the
hanging Threads; especially if you are lacking proper timeout
implementation.
Other less common causes include:
Huge response data causing increased elapsed time to read / consume the Socket Inputstream e.g. such as very large XML data. This can be
proven easily by analysing the size of the response data
Network latency causing increased elapsed time in data transfer from the service provider to your Java EE production system. This can
be proven by running some network sniffer between your production
server and the service provider and determine any major lag/latency
problem
Whatever was your problem, the first thing to do is review your timeout configuration!
What you can do?
You need to do some configuration for Jboss and Apache.
JBoss side
The main concern with server.xml is setting the connectionTimeout
which sets the SO_TIMEOUT of the underlying socket. So when a connection in
Tomcat hasn't had a request in the amount of time specified by
connectionTimeout, then the connection dies off. This is necessary
because if the connection hasn't been used for a certain period of
time then there is the chance that it is half-close on the mod_jk end.
If the connection isn't closed there will be an inflation of threads
which can over time hit the maxThreads count in Tomcat then Tomcat will
not be able to accept any new connections. A connectionTimeout of
600000 (10 minutes) is a good number to start out with. There may be
a situation where the connections are not being recycled fast enough,
in this instance the connectionTimeout could be lowered to 60000 or 1
minute.
When setting connectionTimeout in Tomcat, mod_jk should also have
connect_timeout/prepost_timeout set, which allows detection that the
Tomcat connection has been closed and preventing a retry request.
The recommended value of maxThreads is 200 per CPU, so here we assume
the server is a single core machine. If it has been quad core, we
could push that value to 800, and more depending on RAM and other
machine specs.
<Connector port="8009"
address="${jboss.bind.address}"
emptySessionPath="true"
enableLookups="false"
redirectPort="8443"
protocol="AJP/1.3"
maxThreads="200"
connectionTimeout="600000"></Connector>
Apache side
worker.properties file
See comments inline.
worker.list=loadbalancer,status
worker.template.port=8009
worker.template.type=ajp13
worker.template.lbfactor=1
#ping_timeout was introduced in 1.2.27
worker.template.ping_timeout=1000
#ping_mode was introduced in 1.2.27, if not
#using 1.2.27 please specify connect_timeout=10000
#and prepost_timeout=10000 as an alternative
worker.template.ping_mode=A
worker.template.socket_timeout=10
#It is not necessary to specify connection_pool_timeout if you are running the worker mpm
worker.template.connection_pool_timeout=600
#Referencing the template worker properties makes the workers.properties shorter and more concise
worker.node1.reference=worker.template
worker.node1.host=192.168.1.2
worker.node2.reference=worker.template
worker.node2.host=192.168.1.3
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=True
worker.status.type=status
The key points in the above workers.properties is we've added limits
for the connections mod_jk makes. With the base configuration, socket
timeouts default to infinite. The other important properties are
ping_mode and ping_timeout which handle probing a connection for
errors and connection_pool_timeout which must be set to equal
server.xml's connectionTimeout when using the prefork mpm. When these
two values are the same, after a connection has been inactive for x
amount of time, the connection in mod_jk and Tomcat will be closed at
the same time, preventing a half-closed connection.
Apache configuration
Make note that maxThreads for the AJP connection should coincide with
the MaxClients set in Apache's httpd.conf. MaxClients needs to be set
in the correct module in Apache.
This can be determined by running httpd -V:
# httpd -V
Server version: Apache/2.2.3
Server built: Sep 11 2006 09:43:05
Server's Module Magic Number: 20051115:3
Server loaded: APR 1.2.7, APR-Util 1.2.8
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
Which tells me the Server MPM is Prefork. This is not always 100%
accurate so you should also view the output of /etc/sysconfig/httpd to
see if the following line is there: HTTPD=/usr/sbin/httpd.worker. If
it is commented out you are running prefork, otherwise if uncommented
worker.
httpd.conf:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
MaxClients 200
MaxRequestsPerChild 0
</IfModule>
Or if Apache is using worker, it is
<IfModule worker.c>
StartServers 2
MaxClients 200
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
MaxRequestsPerChild is 0, this is the recommended value when using
mod_jk as mod_jk keeps open persistent connections. The key values in
the above configuration are MaxClients and MaxRequestsPerChild, the
rest of the values are left as default. Note that MaxRequestsPerChild
is recommended to be 0 however the value may need to be greater than 0
depending on if Apache is used for other modules also, especially in
the case of resource leakage.
In the link you can find another configuration to optimize even more this scenario.