We are running spring boot admin console inside Istio and trying to connect to pods with actuator end points. When SBAC tries to connect to a pod, it gives 502. We checked the logs and it sounds that SBAC is duplicating the IP address in the request URL
boot-admin-console 2023-01-12 19:26:54.480 DEBUG 1 --- [or-http-epoll-3] r.n.http.client.HttpClientOperations
: [db6578e5-10, L:/172.30.208.27:34026 - R:172.30.198.29/172.30.198.29:8080] Received response (auto-read:false) : RE
SPONSE(decodeResult: success, version: HTTP/1.1)
spring-boot-admin-console HTTP/1.1 200 OK
spring-boot-admin-console x-content-type-options:
spring-boot-admin-console x-xss-protection:
spring-boot-admin-console cache-control:
spring-boot-admin-console pragma:
spring-boot-admin-console expires:
spring-boot-admin-console x-frame-options:
spring-boot-admin-console content-type:
spring-boot-admin-console date:
spring-boot-admin-console x-envoy-upstream-service-time:
spring-boot-admin-console server:
spring-boot-admin-console transfer-encoding:
spring-boot-admin-console 2023-01-12 19:26:54.480 DEBUG 1 --- [or-http-epoll-3] r.n.r.DefaultPooledConnectionProvider
: [db6578e5-10, L:/172.30.208.27:34026 - R:172.30.198.29/172.30.198.29:8080] onStateChange(GET{uri=/actuator/health,
connection=PooledConnection{channel=[id: 0xdb6578e5, L:/172.30.208.27:34026 - R:172.30.198.29/172.30.198.29:8080]}}, [r
esponse_received])
Any ideas?
We have the headless service for the target service to allow connectivity between SBAC and the application pod. I curled the target pod actuator endpoint from the sbac pod and it is accessible but from the UI is giving 502
I have a third party webapplication having websocket connections. It is expecting Upgrade, Connection header names to be case sensitive. I am using spring cloud gateway as a reverse proxy to this webapp. All the http calls are success but for websocket requests i am getting the following error
io.netty.handler.codec.http.websocketx.WebSocketClientHandshakeException:
Invalid handshake response getStatus: 404 Not Found at
io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:272)
~[netty-codec-http-4.1.65.Final.jar:4.1.65
WebSocketClientHandshaker is setting the upgrade and connection headers and it is in lower case. How do I override these header values ?
application.yml configuration is as below
spring:
main:
web-application-type: reactive
profiles:
active: dev
cloud:
gateway:
globalcors:
corsConfigurations:
'[/**]':
allowedOrigins: "*"
allowedMethods: "*"
allowedHeaders: "*"
httpclient:
pool:
max-idle-time: 10s
ssl:
useInsecureTrustManager: true
filter:
remove-non-proxy-headers:
headers:
- Keep-Alive
- TE
- Trailer
- Transfer-Encoding
routes:
- id : no_op
uri: no://op
predicates:
- Path=/falcon/tunneling/**
- id: reroute_ws
uri: ws://localhost:5555
predicates:
- Path=/websocket/**
- id: reroute_2
uri: http://localhost:5555
predicates:
- Path=/**
I'm playing with Minikube to deploy a simple spring-boot application you can find here: https://github.com/shopKubernetesSpringboot
minikube v1.9.2 on Ubuntu 18.04
Kubernetes v1.17.0 on Docker 19.03.5 ...
I'm using cors, session, basic auth. It works outside the Minikube cluster (with the embedded spring-boot netty server), but I can't make it work with nginx-ingress.
It is basically two microservices products and cart. Product & cart list (GET) works. But I can't make cart add (POST) work, it always returns the following in the browser:
XHRPOST http://192.168.39.94/cart/add
[HTTP/1.1 403 Forbidden 37ms]
CSRF Token has been associated to this client
The back-end is receiving the X-XSRF-TOKEN:
[or-http-epoll-3] o.s.w.s.adapter.HttpWebHandlerAdapter : [e8b45b1f-2] HTTP POST "/cart/add", headers=[Host:"192.168.39.94", X-Request-ID:"dc741d2765b9d0c585b7f73d0e2bab95", X-Real-IP:"192.168.39.1", X-Forwarded-For:"192.168.39.1", X-Forwarded-Host:"192.168.39.94", X-Forwarded-Port:"80", X-Forwarded-Proto:"http", X-Scheme:"http", Content-Length:"120", User-Agent:"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0", Accept:"application/json, text/plain, /", Accept-Language:"en-GB,en;q=0.5", Accept-Encoding:"gzip, deflate", Content-Type:"application/json;charset=utf-8", Authorization:"Basic dXNlcjp1c2Vy", X-XSRF-TOKEN:"17e1b013-98b3-4064-8173-bf3af4ce8bc7", Origin:"http://localhost:3000", Referer:"http://localhost:3000/"]
[or-http-epoll-3] o.s.w.s.adapter.HttpWebHandlerAdapter : [e8b45b1f-2] Completed 403 FORBIDDEN, headers=[Vary:"Origin", "Access-Control-Request-Method", "Access-Control-Request-Headers", Access-Control-Allow-Origin:"http://localhost:3000", Access-Control-Allow-Credentials:"true", Content-Type:"text/plain", Cache-Control:"no-cache, no-store, max-age=0, must-revalidate", Pragma:"no-cache", Expires:"0", X-Content-Type-Options:"nosniff", X-Frame-Options:"DENY", X-XSS-Protection:"1 ; mode=block", Referrer-Policy:"no-referrer", content-length:"45"]
My ingress looks like that:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: shop-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/session-cookie-name: session
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "http://localhost:3000"
nginx.ingress.kubernetes.io/cors-allow-headers: X-XSRF-TOKEN,Accept,Accept-Encoding,Accept-Language,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,Origin,Connection,Content-Length,Content-Type,Host,Referer,User-Agent
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/session-cookie-path: /cart
spec:
rules:
- http:
paths:
- path: /product
backend:
serviceName: product-svc
servicePort: 80
- path: /cart
backend:
serviceName: cart-svc
servicePort: 80
Cart service & pod
apiVersion: v1
kind: Service
metadata:
name: cart-svc
labels:
app: shop
name: cart
tier: backend
spec:
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 120
selector:
app: shop
name: cart
tier: backend
ports:
- protocol: TCP
port: 80
targetPort: rest-api-port
---
apiVersion: v1
kind: Pod
metadata:
name: cart-pod
labels:
app: shop
name: cart
tier: backend
spec:
containers:
- name: cart
image: davidgfolch/shop-cart:latest
ports:
- containerPort: 8080
name: rest-api-port
env:
- name: SPRING_PROFILES_ACTIVE
value: "prod"
SpringSecurityConfig.java
#Configuration
#Slf4j
public class SecurityConfig {
#Value("${com.dgf.shopCart.cors.origins}")
private String corsOrigins;
#Bean
SecurityWebFilterChain springWebFilterChain(ServerHttpSecurity http) {
return http
.csrf().csrfTokenRepository(CookieServerCsrfTokenRepository.withHttpOnlyFalse()).and()
.cors(c -> c.configurationSource(cors()))
.authorizeExchange()
.anyExchange().authenticated()
.and()
.httpBasic()
.and()
.build();
}
private UrlBasedCorsConfigurationSource cors() {
CorsConfiguration config = new CorsConfiguration();
config.setAllowCredentials(true);
config.addAllowedOrigin(corsOrigins);
config.addAllowedHeader("*");
config.addAllowedMethod("*");
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
source.registerCorsConfiguration("/**", config);
return source;
}
}
Comparing logs with and without Kubernetes
With Kubernetes SESSION cookie is not arriving to the back-end (as shown in the logs):
HTTP POST "/cart/add", headers=[Host:"192.168.39.94", X-Request-ID:"aba957e6cc3c8803c1734a2724a75fbf", X-Real-IP:"192.168.39.1", X-Forwarded-For:"192.168.39.1", X-Forwarded-Host:"192.168.39.94", X-Forwarded-Port:"80", X-Forwarded-Proto:"http", X-Scheme:"http", Content-Length:"120", User-Agent:"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0", Accept:"application/json, text/plain, /", Accept-Language:"en-GB,en;q=0.5", Accept-Encoding:"gzip, deflate", Content-Type:"application/json;charset=utf-8", Authorization:"Basic dXNlcjp1c2Vy", X-XSRF-TOKEN:"17e1b013-98b3-4064-8173-bf3af4ce8bc7", Origin:"http://localhost:3000", Referer:"http://localhost:3000/"]
Without Kubernetes they do:
HTTP POST "/cart/add", headers=[Host:"localhost:8080", User-Agent:"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0", Accept:"application/json, text/plain, /", Accept-Language:"en-GB,en;q=0.5", Accept-Encoding:"gzip, deflate", Content-Type:"application/json;charset=utf-8", Authorization:"Basic dXNlcjp1c2Vy", X-XSRF-TOKEN:"17e1b013-98b3-4064-8173-bf3af4ce8bc7", Content-Length:"120", Origin:"http://localhost:3000", Connection:"keep-alive", Referer:"http://localhost:3000/", Cookie:"Idea-c3de9a37=a5f5270f-e0e1-4807-92e2-79c126b768fc; XSRF-TOKEN=17e1b013-98b3-4064-8173-bf3af4ce8bc7; SESSION=eddb22e9-15f7-4b28-a422-36faa8a7f285"]
Any advice? What I'm missing?
I finally found that sticky sessions don't work if not specified spec.host for nginx-ingress versions prior to v.0.27.0. See pull request.
So it should be updated nginx-ingress to version 0.27.0 to make the above configuration work or set spec.host and (I hope) enable DNS addon to make it work. Accessing via the host dns name:
spec:
host: myapp.example.com
rules:
- http:
paths:
- path: /product
backend:
serviceName: product-svc
servicePort: 80
- path: /cart
backend:
serviceName: cart-svc
servicePort: 80
Here you can find some related links:
Sticky session official example
Session affinity does not work when no "rules" are defined
nginx-controller troubleshooting
I will be fine-tuning this answer as soon as I get a working version.
update minikube docker ingress controller to 0.27.0 (NOTE: not working yet):
#minikube docker-env
#eval $(minikube -p minikube docker-env)
#docker images | grep ingress
#docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
#minikube stop && minikube start
#eval $(minikube -p minikube docker-env)
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.27.1/deploy/static/mandatory.yaml
minikube stop && minikube start
All static resources return GET http://www.gogogo.com:8888/js/jquery/2.0.0/jquery.min.js net::ERR_ABORTED 404
and then cannot go to the main page without port number.
I uploaded the spring-boot jar file to Google Cloud server (centos7)
and followed the instruction to set all environment include Nginx. The website is available to open use www.gogogo.com:8888 or port number 8080 but no CSS/JS.
I think the problem is the server didn't go to the correct place to find static resources and after a lot of search on google. I still can't fix the problem. The web on my local page works well and the direction is shown on chrome like "localhost:8888/js". All the static resources are under the webapp file.
This is how I link JS in HTML: <script src="js/jquery/2.0.0/jquery.min.js"></script>
The address in the project is gogogo/src/main/webapp/js.
include /etc/nginx/conf.d/*.conf;
server {
listen 8080 default_server;
listen [::]:8080 default_server;
server_name www.gogogo.com;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location ~ .*\.(js|css|img|jpg|svg|gif|png|bmp|swf|ttf|woff|svg|map|eot)$ {
root /gogogo/;
}
location / {
proxy_pass http://127.0.0.1:8888;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
Another problem is I cannot connect to the website using www.gogogo.com directly.
//////////////////error log///////////////////////////////
"Same Url as before" == www.gogogo.com
2019/02/10 14:23:46 [error] 23153#0: *26 open() "/gogogo/css/nav.css" failed (2: No such file or directory), cl
ient: 31.205.233.115, server: www.gogogo.com, request: "GET /css/nav.css HTTP/1.1", host: "www.gogogo.com:8080", referrer: "http://"www.gogogo.com:8080/home"
2019/02/10 14:23:46 [error] 23153#0: *27 open() "/gogogo/js/moment/2.22.2/moment.js" failed (2: No such file or
directory), client: 31.205.233.115, server: "Same Url as before", request: "GET /js/moment/2.22.2/moment.js HTTP/1.1
", host: ""Same Url as before":8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:23:46 [error] 23153#0: *17 open() "/gogogo/css/home.css" failed (2: No such file or directory), c
lient: 31.205.233.115, server: "Same Url as before", request: "GET /css/home.css HTTP/1.1", host: ""Same Url as before"
:8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:23:46 [error] 23153#0: *20 open() "/gogogo/img/utility/white.png" failed (2: No such file or dire
ctory), client: 31.205.233.115, server: "Same Url as before", request: "GET /img/utility/white.png HTTP/1.1", host: "
"Same Url as before":8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:23:53 [notice] 23259#0: signal process started
2019/02/10 14:24:06 [error] 23264#0: *2 open() "/gogogo/js/jquery/2.0.0/jquery.min.js" failed (2: No such file
or directory), client: 31.205.233.115, server: "Same Url as before", request: "GET /js/jquery/2.0.0/jquery.min.js HTT
P/1.1", host: ""Same Url as before":8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:24:06 [error] 23264#0: *3 open() "/gogogo/js/vue/2.5.16/vue.min.js" failed (2: No such file or di
rectory), client: 31.205.233.115, server: "Same Url as before", request: "GET /js/vue/2.5.16/vue.min.js HTTP/1.1", ho
st: ""Same Url as before":8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:24:06 [error] 23264#0: *4 open() "/gogogo/js/axios/0.17.1/axios.min.js" failed (2: No such file o
r directory), client: 31.205.233.115, server: "Same Url as before", request: "GET /js/axios/0.17.1/axios.min.js HTTP/
1.1", host: ""Same Url as before":8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:24:06 [error] 23264#0: *9 open() "/gogogo/js/moment/2.22.2/moment.js" failed (2: No such file or
directory), client: 31.205.233.115, server: "Same Url as before", request: "GET /js/moment/2.22.2/moment.js HTTP/1.1"
, host: ""Same Url as before":8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:24:06 [error] 23264#0: *10 open() "/gogogo/css/nav.css" failed (2: No such file or directory), cl
ient: 31.205.233.115, server: "Same Url as before", request: "GET /css/nav.css HTTP/1.1", host: ""Same Url as before":8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:24:07 [error] 23264#0: *4 open() "/gogogo/css/footer.css" failed (2: No such file or directory),
client: 31.205.233.115, server: "Same Url as before", request: "GET /css/footer.css HTTP/1.1", host: "www.gogogo.
com:8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:24:07 [error] 23264#0: *3 open() "/gogogo/img/utility/white.png" failed (2: No such file or direc
tory), client: 31.205.233.115, server: "Same Url as before", request: "GET /img/utility/white.png HTTP/1.1", host: "w
ww.gogogo.com:8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:24:07 [error] 23264#0: *2 open() "/gogogo/css/home.css" failed (2: No such file or directory), cl
ient: 31.205.233.115, server: "Same Url as before", request: "GET /css/home.css HTTP/1.1", host: ""Same Url as before":
8080", referrer: "http://"Same Url as before":8080/home"
2019/02/10 14:24:07 [error] 23264#0: *1 open() "/gogogo/img/utility/white.png" failed (2: No such file or direc
tory), client: 31.205.233.115, server: "Same Url as before", request: "GET /img/utility/white.png HTTP/1.1", host: "w
ww.gogogo.com:8080", referrer: "http://"Same Url as before":8080/home"
The request fetching in on wensocket server is :
Upgrade: websocket
Connection: Upgrade
Host: 10.1.5.20:5555
Origin: http://localhost:8080
Sec-WebSocket-Protocol: sip
Pragma: no-cache
Cache-Control: no-cache
Sec-WebSocket-Key: T3jkd1s0pRceQbgdTLoaiQ==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: x-webkit-deflate-frame
The response I am trying to send back for the handshake is :
HTTP/1.1 101 WebSocket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
Sec-WebSocket-Origin: http://localhost:8080
Sec-WebSocket-Location: ws://10.1.5.20:5555/
But the error I receive is :
**Exception in thread "main" java.io.IOException: Handshake failed
at websocket4j.AbstractWebSocket.<init>(AbstractWebSocket.java:123)
at websocket4j.server.WebSocket.<init>(WebSocket.java:73)
at websocket4j.server.WebServerSocket.accept(WebServerSocket.java:119)
at websocket4j.examples.EchoServer.main(EchoServer.java:51)
Caused by: java.io.IOException: End of stream
at websocket4j.AbstractWebSocket.readBytes(AbstractWebSocket.java:230)
at websocket4j.server.WebSocket.handshake(WebSocket.java:203)
at websocket4j.AbstractWebSocket$HandshakeRunner.run(AbstractWebSocket.java:79)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)**
shouldn't you respond with the key?
try to add Sec-WebSocket-Accept: sha1(Sec-WebSocket-Key + magic string ) on your response