502 bad gateway on wildfly-9.01 and apache - performance

we are using wildfly-9.01 on azure environment and using application gateway and WAF services.
We observe 502 in apache log in underload and application became unresponsive.
We are using following configuration.
OS: centos-8(32 vcpus, 256 GiB memory)
web server: apache with modjk
application server: wildfly-9.01(40 GB memory is allocated)
Below is the Wildfly configuration:
<subsystem xmlns="urn:jboss:domain:io:1.1">
<worker name="default" task-max-threads="500"/>
<buffer-pool name="default"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:undertow:2.0">
<buffer-cache name="default"/>
<server name="default-server">
<ajp-listener name="ajpListener" scheme="http" socket-binding="ajp" max-header-size="50000" max-post-size="5368709120" max-parameters="5000000" allow-equals-in-cookie-value="true"/>
<http-listener name="default" socket-binding="http" redirect-socket="https" max-header-size="50000" max-post-size="5368709120" max-parameters="5000000" allow-equals-in-cookie-value="true"/>
<host name="default-host" alias="localhost"/>
</server>
<servlet-container name="default">
<jsp-config x-powered-by="false"/>
<websockets/>
</servlet-container>
<filters>
<response-header name="server-header" header-name="Server" header-value="My company"/>
<response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="My company Application"/>
</filters>
</subsystem>
Below is the Apache Keep alive config:
KeepAlive On
MaxKeepAliveRequests 300
KeepAliveTimeout 600
TimeOut 1200
any suggestions for things to look at or try to avoid 502?

Related

Two services with different host but same alias not working wildfly

We're using Wildfly-17.0.1
and we have 2 spring-boot microservices which are being deployed as war on wildfly
service-a.war
service-b.war
We have jboss-web.xml in both services under src/main/webapp/WEB-INF
service-a.war
<jboss-web>
<context-root>/</context-root>
<virtual-host>default-host</virtual-host>
</jboss-web>
service-b.war
<jboss-web>
<context-root>/</context-root>
<virtual-host>b-host</virtual-host>
</jboss-web>
standalone.xml
<subsystem xmlns="urn:jboss:domain:undertow:9.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other" statistics-enabled="${wildfly.undertow.statistics-enabled:${wildfly.statistics-enabled:false}}">
<buffer-cache name="default" />
<server name="default-server">
<http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true" />
<https-listener name="https" socket-binding="https" security-realm="UndertowRealm" enable-http2="true" />
<host name="default-host" alias="alias" default-web-module="service-a.war" />
<host name="b-host" alias="alias" default-web-module="service-b.war" />
</server>
...
</subsystem>
Above alias is the machine name
In the server.log we see below message when both applications are deployed
...
Registered web context: '/' for server 'default-server'
...
Registered web context: '/' for server 'default-server'
But when we try to access both services only one of the service seems to be working, other gives
<html><head><title>Error</title></head><body>404 - Not Found</body></html>
We tried giving different context-root, and we saw this in log
...
Registered web context: '/a' for server 'default-server'
...
Registered web context: '/b' for server 'default-server'
but still only one works and other gives 404.
It works when we give different alias for both the hosts
Why giving the same alias for different hosts doesn't work?
I think the string you put as virtualhost in jboss-web.xml must match the alias (not the name) in standalone conf
Instead of
<host name="b-host" alias="alias" default-web-module="service-b.war" />
Try with:
<host name="vhost2" alias="b-host" default-web-module="service-b.war" />

Wildfly Undertow directory listing - sorting file

I use undertow to serve some app logs, but I can't figure out how to sort files, for example for modification time... is there a way?
This is the subsystem conf:
<subsystem xmlns="urn:jboss:domain:undertow:3.0">
...
<server name="default-server">
...
<host name="default-host" alias="localhost">
...
<location name="/logs" handler="logs"/>
...
</host>
...
</server>
...
<handlers>
...
<file name="logs" path="/home/app/logs" directory-listing="true"/>
</handlers>
...
</subsystem>

JGroups ec2 cluster fails to connect (times out) with Hibernate Search / Infinispan setup

I'm trying to set up a distributed Hibernate Search (5.5.4) cluster on my Elastic Beanstalk (Tomcat8) environment, using Infinispan (8.2.4) and JGroups.
I'm currently stuck on an issue where a node can't connect to an existing cluster, and it times out trying to connect.
Starting JGroups channel ISPN
variable "${jgroups.s3.pre_signed_delete_url}" in S3_PING could not be substituted; pre_signed_delete_url is removed from properties
variable "${jgroups.s3.prefix}" in S3_PING could not be substituted; prefix is removed from properties
variable "${jgroups.s3.pre_signed_put_url}" in S3_PING could not be substituted; pre_signed_put_url is removed from properties
ip-172-31-24-216-1799: JOIN(ip-172-31-24-216-1799) sent to ip-172-31-14-33-238 timed out (after 5000 ms), on try 1
ip-172-31-24-216-1799: JOIN(ip-172-31-24-216-1799) sent to ip-172-31-14-33-238 timed out (after 5000 ms), on try 2
...
ip-172-31-24-216-1799: JOIN(ip-172-31-24-216-1799) sent to ip-172-31-14-33-238 timed out (after 5000 ms), on try 10
ip-172-31-24-216-1799: too many JOIN attempts (10): becoming singleton
ISPN000094: Received new cluster view for channel ISPN: [ip-172-31-24-216-Channel ISPN local address is ip-172-31-24-216-1799, physical addresses are [127.0.0.1:7800]
I have enabled all types of inbound traffic within the elastic beanstalk security group, and can successfully ping the other nodes in the group using the internal IP addresses.
This is my infinispan.xml file
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:8.2 http://infinispan.org/schemas/infinispan-config-8.2.xsd"
xmlns="urn:infinispan:config:8.2">
<jgroups>
<stack-file name="default-jgroups-ec2" path="default-configs/default-jgroups-ec2.xml"/>
</jgroups>
<cache-container name="HibernateSearch" default-cache="default" statistics="false" shutdown-hook="DONT_REGISTER">
<transport stack="default-jgroups-ec2"/>
<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<jmx duplicate-domains="true"/>
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<replicated-cache name="LuceneIndexesMetadata" mode="SYNC" remote-timeout="25000">
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<transaction mode="NONE"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence>
<file-store path="LuceneIndexes/Metadata" preload="true" />
</persistence>
<indexing index="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
</replicated-cache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<distributed-cache name="LuceneIndexesData" mode="SYNC" remote-timeout="25000">
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<transaction mode="NONE"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence>
<file-store path="LuceneIndexes/Data" />
</persistence>
<indexing index="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
</distributed-cache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<replicated-cache name="LuceneIndexesLocking" mode="SYNC" remote-timeout="25000">
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<transaction mode="NONE"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence>
<file-store path="LuceneIndexes/Locking" />
</persistence>
<indexing index="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
</replicated-cache>
</cache-container>
</infinispan>
And the jgroups config file is the default ec2 config packaged with Infinispan default-jgroups-ec2.xml
Does anyone have any idea of where I may have gone wrong, or what exactly I need to do to get this working?
Your local address is 127.0.0.1:7800, which is the default. This will definitely not work if you need to talk to other nodes.
Also you can see those messages in the logs:
variable "${jgroups.s3.pre_signed_delete_url}" in S3_PING could not be substituted; pre_signed_delete_url is removed from properties
variable "${jgroups.s3.prefix}" in S3_PING could not be substituted; prefix is removed from properties
variable "${jgroups.s3.pre_signed_put_url}" in S3_PING could not be substituted; pre_signed_put_url is removed from properties
You should probably define those variables.

JBoss AS7 rewrite issue

I'm trying to redirect http connections to https in JBoss AS 7.1.1 using the rewrite valve inside a virtual host. This is my current configuration in standalone.xml:
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" redirect-port="8443"/>
<connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
<ssl name="my-ssl" key-alias="my-alias" password="pass" certificate-key-file="/path/to/keystore"/>
</connector>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="localhost"/>
<alias name="example.com"/>
<rewrite pattern=".*" substitution="https://myserver.com:8443" flags="R,L">
<condition test="%{HTTPS}" pattern="off" flags="NC"/>
</rewrite>
</virtual-server>
</subsystem>
The http port is 8080 and the https is 8443. Access to http://myserver.com:8080 are redirected correctly to https://myserver.com:8443, but (this is the issue I'm trying to resolve) access to http://myserver.com:8443 return an empty HTTP 200 OK response.
It is correct that JBoss apparently accepts http connections in the https port, or there is an error in the server configuration? If there isn't a error, how can I redirect http://myserver.com:8443 to https://myserver.com:8443?

How do I set the number of "webcontainer worker-threads" in jboss as 7?

Most app. servers provide a way of tuning the number of WebContainer worker-threads when it goes down to tuning. Is it possible to do that in JBoss AS 7.x?
Thanks.
you can tune the HTTP Conector of the AS7 web subsystem. The available attributes you can tune for the HTTP Connector are described here The Http Connector. To define the max-connections for this connector you need change it in $JBOSS_HOME/standalone/configuration/standalone.xml or $JBOSS_HOME/domain/configuration/domain.xml
See this piece of configuration:
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
<connector name="http"
protocol="HTTP/1.1"
scheme="http"
socket-binding="http"
max-connections="250"/>
...
</subsystem>
To define a thread pool specific for the HTTP Connector you need to use the AS7 threads subsystem like this one:
<subsystem xmlns="urn:jboss:domain:threads:1.0">
<bounded-queue-thread-pool name="http-executor" blocking="true">
<core-threads count="10" per-cpu="20" />
<queue-length count="10" per-cpu="20" />
<max-threads count="10" per-cpu="20" />
<keepalive-time time="10" unit="seconds" />
</bounded-queue-thread-pool>
</subsystem>
and then you need to reference it in the executor attribute of the HTTP Connector. See this piece of config:
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
<connector name="http"
protocol="HTTP/1.1"
scheme="http"
socket-binding="http"
max-connections="250"
executor="http-executor"/>
...
</subsystem>
For more details about tuning the AS7 see this post JBoss AS 7 Performance tuning - Tuning Web server thread pool on the masterjboss.com.

Resources