We have Spring Boot service running in Kubernetes.
This service has endpoint:
- GET /healthz
We have liveness probe that uses this endpoint. Probe runs successfully.
It means that the endpoint is reachable from the service pod (localhost).
When I run in the service pod :
wget https://localhost:8080/healthz
I get an answer (OK)
When I try to call this endpoint outside the pod wget https://myhost:8080/healthz, I get response 400 without body.
I don't see any logs of Sprint. It seems that it does not reach the Sprint .
When I added flag -Djavax.net.debug=all I see in log that TLS handshake finished and then:
GET /healthz HTTP/1.1
host: myhost:8080
accept: application/json
Connection: close
and immediately
HTTP/1.1 400
Transfer-Encoding: chunked
Date: Mon, 25 Jun 201 8 08:43:43 GMT
Connection: close
When I try wget https://myhost:8080/blahblah (non existing endpoint),
I still get 400, not 404!
When I try wget https://myWronghost:8080/healthz (wrong host), I get an error Bad address. It means that host 'myhost' is correct (otherwise I would get this error).
Docker file:
FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENV JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8
ENTRYPOINT ["java","-Djavax.net.debug=all", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
EXPOSE 8080
Summing up:
The service endpoints are accessible from within service pod, but not accessible from outside the pod.
Any idea why?
Update:
The problem was solved by calling the service with fully qualified domain name : serviceName.namespaceName.svc.cluster.local
Tomcat didn't accept calls with short domain serviceName.namespaceName, it responded 400.
Your issue can be caused by https://github.com/spring-projects/spring-boot/issues/13205.
All you have to do is upgrade Tomcat version to 8.5.32. You can do that by adding the version in pom.xml file.
<properties>
<!-- your properties -->
<tomcat.version>8.5.32</tomcat.version>
</properties>
If you are using Spring boot 2 this may be due to bug in Tomcat 8.5.31 that doesnt allow '-' in last part of FQDN
Update Tomcat to 8.5.32 fixes this.
Reference:
https://github.com/spring-projects/spring-boot/issues/13205
https://bz.apache.org/bugzilla/show_bug.cgi?id=62383
https://bz.apache.org/bugzilla/show_bug.cgi?id=62371
Not sure if it has any influence here, but you're trying everything with https. Can you try with http instead? Your spring app probably doesn't support https on port 8080.
The problem was solved by calling the service with fully qualified domain name :
service-name.namespace-name.svc.cluster.local
The service didn't accept calls with service-name.namespace-name, responded 400.
Related
I have an application (spring boot 2.5.6) where the user is able to request a REST resource with a callback uri as query parameter eg. https://someservice/callback or http://localhost:5555/callback. This works well if a user specifies a port which is not equal to the internal tomcat port (8080 in this case). However if the user defines an uri like http://localhost:8080 he's not redirected to http://localhost:8080/callback but to the uri which was specified in the request Host-Header eg. https://myservice.com/callback:
Working as expected
GET https://myservice.com/foobar?callback=https://someotherservice/callback HTTP/1.1
Host: myservice.com
...
HTTP/1.1 303
Location: https://someotherservice/callback?...
...
Working as expected
GET https://myservice.com/foobar?callback=http://localhost:5555/callback HTTP/1.1
Host: myservice.com
...
HTTP/1.1 303
Location: http://localhost:5555/callback?...
...
Not working as expected
GET https://myservice.com/foobar?callback=http://localhost:8080/callback HTTP/1.1
Host: myservice.com
...
HTTP/1.1 303
Location: https://myservice.com/callback?...
...
I'm not able to reproduce this behavior on my local machine (windows 10). This happens on our dev environment (debian buster). Any ideas who is rewriting the url (spring, tomcat?) and how I'm able to prevent this behavior? I tested it on a single instance to be sure it's not our loadbalancer.
Thank you in advance!
The config server is up and the client is able to fetch the properties from config server.
HTTP GET http://localhost:8888/account-service/dev/config
Accept=[application/json, application/*+json]
Response 200 OK
But the problem is, client service is trying to start on config server port 8888. I have set server.port=8080 in the client but still not working.
***************************
APPLICATION FAILED TO START
***************************
Description:
Web server failed to start. Port 8888 was already in use.
Am I missing any configuration here? Highly appreciate any help.
Config server application.properties,
spring.application.name=config-server
spring.cloud.config.profiles=dev
spring.cloud.config.server.git.uri=REPO_URL
server.port=8888
Finally, I was able to find the solution. When I was trying to start client service, server.port was also getting overridden with config server port. I added server.port=8080 in the properties file for the client service profile in config repo (account-service.properties) and it worked.
Hi openshift community,
I am currently migrating an app to Openshift and has encountered failed health probes due to connection refused. What I find strange is that if I ssh into the pod and use
curl localhost:10080/xxx-service/info
It returns HTTP 200 but if I use the IP address then it fails with
This is the details:
POD status
Logs in Openshift saying Spring boot started successfully
Openshift events saying probes failed due to connection refused
Tried SSH to pod to check using localhost which works
Not sure why the IP address is not resolving at the POD level.... Does anyone know the answer or have encountered it?
It is hard to say in your case what exactly the issue is, as it is environment specific.
In general, you should avoid using IP addresses when working with Kubernetes, as these will change whenever a Pod is restarted (which may be the root cause for the issue you are seeing).
When defining readiness and liveness probes for your container, I would recommend that you always use the following syntax to define your checks (note that it does not specify the host):
...
readinessProbe:
httpGet:
path: /xxx-service/info
port: 10080
initialDelaySeconds: 15
timeoutSeconds: 1
...
See also the Kubernetes or OpenShift documentation for more information:
https://docs.openshift.com/container-platform/3.11/dev_guide/application_health.html
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-http-request
I found the root cause, it turns out it was Spring related.
It was a Spring Boot app that was being packaged as WAR file and deployed to Tomcat server. In the application.properties, it had this field:
server.address=127.0.0.1
Removing it has fixed this issue.
https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#common-application-properties-server
I want to deploy my spring boot app in a docker component to gcp App Engine
When I run the docker componet local I get access to the web site.
When I deploy the component to the gcp app engine with the command gcloud app deploy
I get a http error 502 Bad Gateway nginx
The Docker file look like this
FROM adoptopenjdk/openjdk14
MAINTAINER steinko
VOLUME /tmp
COPY build/libs/atm.jar ./
ENTRYPOINT ["java"]
CMD ["-jar", "/atm.jar"]
EXPOSE 4001
The app.yaml files looks like this
runtime: custom
env: flex
handlers:
- url: /.*
script: this field is required, but ignored
service: atm
How do I fix this error?
According to this document: The App Engine front end will route incoming requests to the appropriate module on port 8080. You must be sure that your application code is listening on 8080. Also, it looks like the FROM should be one of Google's base image, also in that document.
I have Amazon's out of the box instance with Tomcat7 (ami-518e4c38), deployed a war file to it, but keep getting 503 error.
I've set the connector to listen on port 80 in server.xml, in the default security group I got 80 (HTTP) set to 0.0.0.0/0
I'm assuming that I don't have to start/stop tomcat manually, should start when the instance is launched. Am I correct on this one?
When I ping localhost (while ssh'ed into the instance ) 2 times on port 80 I get:
2 packets transmitted, 2 received, 0% packet loss, time 999ms
Any help will be most appreciated.
Seriously, this isn't an answer, you should describe your initial situation and what you did in your answer and accept it.
I just want to clear up what happened, and I would need your feedback to understand more about it.
Initially you had a 503 error, which means service unavailable, normally Tomcat is configured behind Apache, and with this I mean that the Apache server gets (all) the requests and may forward (some of) them to Tomcat, if configured to do so.
If you get a 503 error, it means there is some server up and running to reply to you and that should be Apache. If I shut down Tomcat and try to request an URL that would be forwarded to Tomcat I get:
Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.2.17 (Ubuntu) Server at private.it Port 80
That's why my first thought was to check if Tomcat was running. Then you did:
$ ps fax | grep tomcat
And you got that tomcat was actually running:
1127 pts/0 S+ 0:00 _ grep tomcat 987 ? Sl 0:42 /usr/lib/jvm/jre/bin/java -classpath /opt/tomcat7/bin/bootstrap.jar:/opt/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat7 -Dcatalina.home=/opt/tomcat7 -Djava.awt.headless=true -Djava.endorsed.dirs=/opt/tomcat7/endorsed -Djava.io.tmpdir=/opt/tomcat7/temp -Djava.util.logging.config.file=/opt/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
Another reason could be that the connection Tomcat-Apache is not configured correctly. However then you posted that Tomcat was trying to bind port 80 and failing:
Sep 30, 2011 4:07:51 AM org.apache.coyote.AbstractProtocol init SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-80"] java.net.BindException: Address already in use :80
I tried to replicate that error, I went in my conf/server.xml file and change the connector to listen on port 80 instead of 8080:
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL HTTP/1.1 Connector on port 8080
-->
<Connector port="80" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
I start tomcat and monitor the log:
bin/startup.sh && tail -f logs/catalina.out
And I get an exception that is similar to what you get, but not exactly the same:
SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-80"]
java.net.BindException: Permission denied <null>:80
Now if I check whether Tomcat is running I find that is indeed running. While without checking I'd have supposed that in a case like that the server would have just gave up and quit. In my case the correction would be, either shut down Apache and use Tomcat directly, which is not advisable for high traffic sites or move Tomcat back to port 8080 and install mod_jk.
The installation of mod_jk enables the communication between Apache and Tomcat. Then in the virtual host configuration I'd mount a folder or the root folder and do some URL rewriting if necessary:
JkMount /webappname/* ajp13_worker
RewriteEngine on
RewriteRule ^/nametoshow/(.*)$ /webappname/$1 [PT,QSA]
And everything should work. However, I am not sure how you solved the bind error, you didn't say.