Kubernetes: Tomcat throws Exception when enabling proxy protocol - spring-boot

Question

I am kind of lost right now. I set up a Kubernetes Cluster, deployed a Spring Boot API and a LoadBalancer which worked fine. Now I want to enable proxy protocol on the LoadBalancer to preserve the real clients IP, but once I do this my Spring Boot API always returns with a 400 Bad Request and an IllegalArgumentException is thrown.
Here is the short stack trace (I masked the ip addresses):
2020-09-29 20:05:58.382 INFO 1 --- [nio-8080-exec-1] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol [255.255.255.253 255.255.255.254]
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:560) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
I am using the hcloud-cloud-controller-manager from Hetzner.
Here is my LoadBalancer:
apiVersion: v1
kind: Service
metadata:
labels:
service: auth-service
name: auth-service-service
annotations:
load-balancer.hetzner.cloud/name: "lb-backend"
load-balancer.hetzner.cloud/health-check-port: "80"
load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
spec:
ports:
- name: http
port: 80
targetPort: 8080
selector:
service: auth-service
externalTrafficPolicy: Local
type: LoadBalancer
Here is my Spring Config:
spring:
datasource:
platform: postgres
url: ${DATABASE_CS}
username: ${DATABASE_USERNAME}
password: ${DATABASE_PASSWORD}
driver-class-name: org.postgresql.Driver
flyway:
schemas: authservice
jpa:
show-sql: false
properties:
hibernate:
dialect: org.hibernate.dialect.PostgreSQLDialect
jdbc:
lob:
non_contextual_creation: true
hibernate:
ddl-auto: validate
security:
jwt:
secret-key: ${JWT_SECRET_KEY}
expires: ${JWT_EXPIRES:300000}
mail:
from: ${MAIL_FROM}
fromName: ${MAIL_FROM_NAME}
smtp:
host: ${SMTP_HOST}
username: ${SMTP_USERNAME}
password: ${SMTP_PASSWORD}
port: ${SMTP_PORT:25}
mjml:
app-id: ${MJML_APP_ID}
app-secret: ${MJML_SECRET_KEY}
stripe:
keys:
secret: ${STRIPE_SECRET_KEY}
public: ${STRIPE_PUBLIC_KEY}
server:
forward-headers-strategy: native
As you may have noticed I already tried to enable the forward-headers based on this issue.
Thanks for your help!

Answer 1

You enabled the PROXY protocol, which is not HTTP, but a different protocol to tunnel TCP connections to downstream servers, keeping as much information as possible.
https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
I am quite sure you want to disable this
load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
and instead rely on the forward headers to the the remote-address to the correct value for the client.
To be honest I am not aware that tomcat supports the PROXY protocol. (Edit: Currently is does not, see https://bz.apache.org/bugzilla/show_bug.cgi?id=57830)