Spring Webflux and Proxy Protocol on Kubernetes - spring

Question

I have a Spring Webflux application that is deployed on Digital Ocean Kubernetes with Proxy Protocol enabled because the app needs to get access to the client IP.
Here are Kubernetes resource descriptions:
---
apiVersion: v1
kind: Service
metadata:
name: myapp-loadbalancer
annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
spec:
type: LoadBalancer
selector:
app: myapp
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ipregistry-api
spec:
replicas: 1
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp/myapp:1553679009241
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /actuator/health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
imagePullSecrets:
- name: regcred
The deployment works great. Unfortunately, when I perform a GET request to any app endpoint I get an HTTP 400 response.
By looking at the logs I noticed that Spring Webflux and the underlying Netty library that is used does not parse the request properly when proxy protocol is enabled.
Please note that if the proxy protocol is disabled by setting service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol to false then everything works but I cannot get the real client IP.
Here is the error I get in the logs when debugging is enabled:
2019-03-27 09:33:45.916 DEBUG 1 --- [or-http-epoll-1] r.n.http.server.HttpServerOperations : [id: 0x3c2ee440, L:/10.244.2.230:8080 - R:/10.131.95.63:27204] New http connection, requesting read 2019-03-27 09:33:45.916 DEBUG 1 --- [or-http-epoll-1] reactor.netty.channel.BootstrapHandlers : [id: 0x3c2ee440, L:/10.244.2.230:8080 - R:/10.131.95.63:27204] Initialized pipeline DefaultChannelPipeline{(BootstrapHandlers$BootstrapInitializerHandler#0
= reactor.netty.channel.BootstrapHandlers$BootstrapInitializerHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)} 2019-03-27 09:33:45.917 DEBUG 1 --- [or-http-epoll-1] r.n.http.server.HttpServerOperations : [id: 0x3c2ee440, L:/10.244.2.230:8080 - R:/10.131.95.63:27204] Decoding failed: DefaultFullHttpRequest(decodeResult: failure(java.lang.IllegalArgumentException: invalid version format:
83.252.136.179 10.16.5.223 3379 80), version: HTTP/1.0, content: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 0, cap: 0)) GET /bad-request HTTP/1.0 :
java.lang.IllegalArgumentException: invalid version format:
83.252.136.179 10.16.5.223 3379 80 at io.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:121) ~[netty-codec-http-4.1.33.Final.jar:4.1.33.Final] at io.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:76) ~[netty-codec-http-4.1.33.Final.jar:4.1.33.Final] at io.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:87) ~[netty-codec-http-4.1.33.Final.jar:4.1.33.Final] at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:219) ~[netty-codec-http-4.1.33.Final.jar:4.1.33.Final] at io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:101) ~[netty-codec-http-4.1.33.Final.jar:4.1.33.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.33.Final.jar:4.1.33.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.33.Final.jar:4.1.33.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) ~[netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar:4.1.33.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) ~[netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar:4.1.33.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) ~[netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar:4.1.33.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
I tried to use the property server.use-forward-headers=true in my application properties but it has no effect.
Is there a way to support Proxy Protocol with Spring Webflux? How to proceed? An example would help a lot.