I have Amazon's out of the box instance with Tomcat7 (ami-518e4c38), deployed a war file to it, but keep getting 503 error.
I've set the connector to listen on port 80 in server.xml, in the default security group I got 80 (HTTP) set to 0.0.0.0/0
I'm assuming that I don't have to start/stop tomcat manually, should start when the instance is launched. Am I correct on this one?
When I ping localhost (while ssh'ed into the instance ) 2 times on port 80 I get:
2 packets transmitted, 2 received, 0% packet loss, time 999ms
Any help will be most appreciated.
Seriously, this isn't an answer, you should describe your initial situation and what you did in your answer and accept it.
I just want to clear up what happened, and I would need your feedback to understand more about it.
Initially you had a 503 error, which means service unavailable, normally Tomcat is configured behind Apache, and with this I mean that the Apache server gets (all) the requests and may forward (some of) them to Tomcat, if configured to do so.
If you get a 503 error, it means there is some server up and running to reply to you and that should be Apache. If I shut down Tomcat and try to request an URL that would be forwarded to Tomcat I get:
Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.2.17 (Ubuntu) Server at private.it Port 80
That's why my first thought was to check if Tomcat was running. Then you did:
$ ps fax | grep tomcat
And you got that tomcat was actually running:
1127 pts/0 S+ 0:00 _ grep tomcat 987 ? Sl 0:42 /usr/lib/jvm/jre/bin/java -classpath /opt/tomcat7/bin/bootstrap.jar:/opt/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat7 -Dcatalina.home=/opt/tomcat7 -Djava.awt.headless=true -Djava.endorsed.dirs=/opt/tomcat7/endorsed -Djava.io.tmpdir=/opt/tomcat7/temp -Djava.util.logging.config.file=/opt/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
Another reason could be that the connection Tomcat-Apache is not configured correctly. However then you posted that Tomcat was trying to bind port 80 and failing:
Sep 30, 2011 4:07:51 AM org.apache.coyote.AbstractProtocol init SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-80"] java.net.BindException: Address already in use :80
I tried to replicate that error, I went in my conf/server.xml file and change the connector to listen on port 80 instead of 8080:
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL HTTP/1.1 Connector on port 8080
-->
<Connector port="80" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
I start tomcat and monitor the log:
bin/startup.sh && tail -f logs/catalina.out
And I get an exception that is similar to what you get, but not exactly the same:
SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-80"]
java.net.BindException: Permission denied <null>:80
Now if I check whether Tomcat is running I find that is indeed running. While without checking I'd have supposed that in a case like that the server would have just gave up and quit. In my case the correction would be, either shut down Apache and use Tomcat directly, which is not advisable for high traffic sites or move Tomcat back to port 8080 and install mod_jk.
The installation of mod_jk enables the communication between Apache and Tomcat. Then in the virtual host configuration I'd mount a folder or the root folder and do some URL rewriting if necessary:
JkMount /webappname/* ajp13_worker
RewriteEngine on
RewriteRule ^/nametoshow/(.*)$ /webappname/$1 [PT,QSA]
And everything should work. However, I am not sure how you solved the bind error, you didn't say.
Related
Short introduction
I have Apache Superset and Apache2 server located on the same EC2 instance. Apache2 is acting as a proxy server. It accepts HTTPS requests and transfers them to Apache Superset. Apache Superset is run using gunicorn.
Problem
Requests to Apache Dremio data engine could take some time (< 60 seconds). When accessing dashboards on Superset, using DNS name with SSL, with proxy setup some dashboards parts (requests) are failing with the following error:
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
Reason: Error reading from remote server
Strangely, these errors can appear in a matter of seconds despite that default value for ProxyTimeout is quite high.
The problem doesn't occur if Superset is accessed by IP address.
Error message in apache2/error.log:
(20014) Internal error (specific information not available): [client 10.4.26.3:6969] AH01102: error reading status line from remote server localhost:8088, referer: ...
What was tried to solve a problem
Problem can be with proxy server timeout or with Superset web server dropping some connections. My Apache2 config:
<VirtualHost *:443>
ProxyPreserveHost On
ProxyRequests Off
ServerName dash.domain.com
ServerAlias dash.domain.com
SSLEngine on
SSLCertificateFile /etc/ssl/private/cert.crt
SSLCertificateChainFile /etc/ssl/certs/cert2.crt
SSLCertificateKeyFile /etc/ssl/private/key.key
ProxyPass / http://localhost:8088/ connectiontimeout=3600 timeout=3600
ProxyPassReverse / http://localhost:8088/
# things tried
# SetEnv force-proxy-request-1.0 1
# SetEnv proxy-nokeepalive 1
# SetEnv proxy-initial-not-pooled 1
# ProxyTimeout 3600
# TimeOut 3600
</VirtualHost>
Things tested (and not working):
Timeout and ProxyTimeout
connectiontimeout and timeout (as seen above)
Keepalive=On for ProxyPass
different SetEnv
superset_config.py -> ENABLE_PROXY_FIX, SUPERSET_WEBSERVER_TIMEOUT
In addition, similar proxy setup was build using nginx, error is similar to what is described here.
Any help or ideas would be appreciated. Thank you very much!
Useful information
Apache Superset version: 0.37.2
Apache Dremio version: 4.1.0
Apache2 server version: 2.4.29
EC2 instance type: t3.medium
OS version: Ubuntu 18.04
The problem was in dying gunicorn async workers. Too many requests were coming from the charts and workers were not able to handle them. Changing worker type from async to sync (default gunicorn type) solved the proxy problem.
I still don't know why direct access by IP was not producing the 502 proxy error.
Sorry for not including information about gunicorn in the question.
P.S Recommended type of workers for Apache Superset from their docs is async, but, for my case, sync were the better solution. In theory, sync workers are slower compare to async (in Superset context).
Following this detailed article: https://www.tessian.com/blog/how-to-fix-http-502-errors/
We have tried the suggested fix (based on AWS ALB default connection idle timeout = 60s setting ):
Gunicorn (Python)
As command line arguments:
--keep-alive 65
Works like a charm!
And to explain "why direct access by IP was not producing the 502 proxy error", check this Gunicorn settings doc:
https://docs.gunicorn.org/en/stable/settings.html#keepalive
Generally set in the 1-5 seconds range for servers with direct connection to the client (e.g. when you don’t have separate load balancer).
Since the default keepalive setting is 2 seconds, it works well on direct access by IP.
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-12-28 09:49:48.561 ERROR 482 --- [ main] o.s.boot.SpringApplication : Application run failed
when I try to start server with command:
java -jar jin-alpha-1.0.jar
under AWS Ubuntu linux server.
In case of trying to local start server was success.
The Changed port is suspicious the reason why before changed It work.
I changed Server port from 8080 to 80.(The reason why AWS http allowed only 80 port)
Local Case : The port of Tomcat server ==> 8080 and 80, Both of them are work. and Stared
AWS Ubuntu : Start with 8080 is work but 80 is not work and Error displayed Above.
I got solution.
I Changed Tomcat port from 80 to 8080. so Server started under Aws ubuntu.
(I don't still know why 80 port is not work under Aws Ubutu)
and Add "Custom TCP Rule" with 8080 port Range.
These are called Priviliged ports on *nix systems. The TCP/IP port numbers below 1024 are special in that normal users are not allowed to run servers on them. This is a security feaure, in that if you connect to a service on one of these ports you can be fairly sure that you have the real thing, and not a fake which some hacker has put up for you.
The normal port number for W3 servers is port 80. This number has been assigned to WWW by the Internet Assigned Numbers Authority, IANA.
When you run a server as a test from a non-priviliged account, you will normally test it on other ports, such as 2784, 5000, 8001 or 8080.
Hope this explains why you cannot run on 80. There are workarounds like running as root. Which I dont recommend as if some hacker gets access to the service then they get root privileges on the box. You need to be careful not running services on such ports.
We have Spring Boot service running in Kubernetes.
This service has endpoint:
- GET /healthz
We have liveness probe that uses this endpoint. Probe runs successfully.
It means that the endpoint is reachable from the service pod (localhost).
When I run in the service pod :
wget https://localhost:8080/healthz
I get an answer (OK)
When I try to call this endpoint outside the pod wget https://myhost:8080/healthz, I get response 400 without body.
I don't see any logs of Sprint. It seems that it does not reach the Sprint .
When I added flag -Djavax.net.debug=all I see in log that TLS handshake finished and then:
GET /healthz HTTP/1.1
host: myhost:8080
accept: application/json
Connection: close
and immediately
HTTP/1.1 400
Transfer-Encoding: chunked
Date: Mon, 25 Jun 201 8 08:43:43 GMT
Connection: close
When I try wget https://myhost:8080/blahblah (non existing endpoint),
I still get 400, not 404!
When I try wget https://myWronghost:8080/healthz (wrong host), I get an error Bad address. It means that host 'myhost' is correct (otherwise I would get this error).
Docker file:
FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENV JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8
ENTRYPOINT ["java","-Djavax.net.debug=all", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
EXPOSE 8080
Summing up:
The service endpoints are accessible from within service pod, but not accessible from outside the pod.
Any idea why?
Update:
The problem was solved by calling the service with fully qualified domain name : serviceName.namespaceName.svc.cluster.local
Tomcat didn't accept calls with short domain serviceName.namespaceName, it responded 400.
Your issue can be caused by https://github.com/spring-projects/spring-boot/issues/13205.
All you have to do is upgrade Tomcat version to 8.5.32. You can do that by adding the version in pom.xml file.
<properties>
<!-- your properties -->
<tomcat.version>8.5.32</tomcat.version>
</properties>
If you are using Spring boot 2 this may be due to bug in Tomcat 8.5.31 that doesnt allow '-' in last part of FQDN
Update Tomcat to 8.5.32 fixes this.
Reference:
https://github.com/spring-projects/spring-boot/issues/13205
https://bz.apache.org/bugzilla/show_bug.cgi?id=62383
https://bz.apache.org/bugzilla/show_bug.cgi?id=62371
Not sure if it has any influence here, but you're trying everything with https. Can you try with http instead? Your spring app probably doesn't support https on port 8080.
The problem was solved by calling the service with fully qualified domain name :
service-name.namespace-name.svc.cluster.local
The service didn't accept calls with service-name.namespace-name, responded 400.
I have an apache/2.4.18 ubuntu server and I want to host my spring application on it. I generated a JAR file and can run it on the server. It starts an embedded tomcat server on port 8090.
However when i navigate to 'my-site-ip:8090' the connection times out.
I have zero experience deploying web applications so any help would be appreciated.
I've created a TCP rule for port 8090 and still no joy.
The solution was adding a proxy to the Myapp.conf file as below:
ProxyRequests off
ProxyPreserveHost On
ProxyPass / http://localhost:8090/
ProxyPassReverse / http://localhost:8090/
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
It´s very hard to explain all the steps in one answer but you can follow these steps to get into the full configuration by your own. I did the same on my 1&1 cloud server.
First of all you need root access to your server.
Normally, on your server the port 80 and 443 should already be open. Else you can define that in the 1&1 Admin Portal. If your Server already has the apache configuration you should be able to see the apache site if you go to your server address. You can find details and the full setup if you dont have an apache installed for this step here:
How To Install the Apache Web Server on Ubuntu
The second step would be to configure a virtual host on your apache webserver.
This is cool because you can define multiple domains and there applications on your server. So http://yourServer.com(port 80 or 443 from extern) goes to yourApp1. (port 8090 from intern).
In this step you will tell apache if your enter your url to go to your app with port 8090
How To Set Up Apache Virtual Hosts on Ubuntu
The last step would be to install your spring-boot app as a service on your machine. The docs of Spring describes it very well.
Installation as an init.d Service
If you install the app as a service you are able to start and stop the app with the service command.
service myapp start
And dont forget to add the plugin for maven or gradle to your pom.xml. This is necessary to run the app as a service.
If you follow these Steps you should be able to reach you app without specify a port and be ready to go with your app in production if necessary.
The best approach for this would be to use the apache proxy. This should get it done.
https://blog.marcnuri.com/running-apache-tomcat-and-apache-httpd-on-port-80-simultaneously/
I have installed wakanda server on an Amazon EC2 server running ubuntu by following this utube video: https://www.youtube.com/watch?v=uSQODnB7wRU .
Now the video is for an older version but I have followed along successfully until I actually launch wakanda on the server. This is what I get in the console:
Welcome to Wakanda Server 10 build 10.187175
Publishing "DefaultSolution" solution
The solution's log file will be stored in the "/home/ubuntu/.Wakanda Server/UserCache/Wakanda Server/DefaultSolution-1882/Logs/" folder
The Administration Web Server cannot listen for connections on port 8080 or secure port 4433 on all IP addresses
You can customize the Administration Web Server's ports with the "--admin-port" and "--admin-ssl-port" options
, then when I try to log into it via the browser it says the connection dropped! Any help would be much appreciated, it seams I need to restrict the IP addresses which can access, but how?
Your wakanda server tried and failed to listen on 8080 and/or 4433
Check the following things:
Are the ports 8080 and/or 4433 used by other processes? (sudo netstat -tapen | grep :8080, if a result is found, then yes another process uses 8080. Check 4433 also)
You may found that wakanda server is already running as a service:
yes you should use this service (create and edit /etc/default/wakanda, add WAKANDA_SOLUTION_AT_STARTUP=your_path and restart with sudo /etc/init.d/wakanda restart)
or to continue starting it manually, stop the service first (sudo /etc/init.d/wakanda stop)
Has the current user the right to listen on those ports? (try running the server with sudo just to check, then use authbind or equivalent)
Can you use alternative ports? (use --admin-port and --admin-ssl-port wakanda server options)
wakanda-server --help will give you the list of options available, especially --solution=VALUE to provide the path to your solution.