First request for Okta authentication taking a lot of time - spring-boot

Question

I am using Okta authorisation server for securing spring boot app.
Have configured the below 3 property
okta.oauth2.issuer={URI}
okta.oauth2.client-secret={SECRET}
okta.oauth2.client-id={CLIENTID}
Have protected the endpoint using the below code in Spring boot.
#Bean
public SecurityWebFilterChain securityWebFilterChain(ServerHttpSecurity http) {
Okta.configureResourceServer401ResponseBody(http);
return http
.csrf(spec -> spec.disable())
.authorizeExchange().pathMatchers("/api/**").permitAll()
.anyExchange().authenticated().and()
.oauth2Login()
.and()
.oauth2ResourceServer()
.jwt().and().and().build();
}
However when I issue the first request to any protected endpoint by passing bearer token it takes a lot of time almost 1min. The subsequent request taken Milliseconds.
From the log I can see the DNS resolution (io.netty.resolver.dns.DnsNameResolver) in the first request are taking a lot of time almost 40 Seconds. The application is of type Web in Okta.
The application is running on Netty server.
Any suggestion to fix this?
I was able to debug and find that on windows 10, JDK 11 the Netty DnsQueryContext takes 5 seconds to resolve DNS whereas the same in macOS or linux happens in milliseconds.
I tried to disable Netty as default server and use tomcat but even then the Netty DNS resolver starts when a incoming request is sent. I need to have Netty on my class path for WebClient to work.
It looks like a issue with Netty as I was able to reproduce the issue with the below code on windows but the same code works fine on macOS.
#SpringBootApplication
public class GqlApplicationStarter {
public static void main(String[] args) throws UnknownHostException, ExecutionException, InterruptedException {
// SpringApplication.run(GqlApplicationStarter.class, args);
NioEventLoopGroup group = new NioEventLoopGroup(1);
DnsNameResolver resolver = new DnsNameResolverBuilder(group.next())
.channelFactory(NioDatagramChannel::new)
.optResourceEnabled(false)
.build();
try {
System.out.println("2" + resolver.resolveAll("dev-542348.okta.com").get());
}
finally {
resolver.close();
group.shutdownGracefully();
}
}
Below are the logs from Windows.
13:11:27.823 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[53282: /10.150.1.252:53], DefaultDnsQuestion(dev-542348.okta.com. IN
A) 13:11:27.830 [nioEventLoopGroup-2-1] DEBUG io.netty.util.Recycler -
-Dio.netty.recycler.maxCapacityPerThread: 4096 13:11:27.831 [nioEventLoopGroup-2-1] DEBUG io.netty.util.Recycler -
-Dio.netty.recycler.ratio: 8 13:11:27.831 [nioEventLoopGroup-2-1] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.chunkSize: 32
13:11:27.831 [nioEventLoopGroup-2-1] DEBUG io.netty.util.Recycler -
-Dio.netty.recycler.blocking: false 13:11:27.834 [nioEventLoopGroup-2-1] DEBUG io.netty.buffer.AbstractByteBuf -
-Dio.netty.buffer.checkAccessible: true 13:11:27.834 [nioEventLoopGroup-2-1] DEBUG io.netty.buffer.AbstractByteBuf -
-Dio.netty.buffer.checkBounds: true 13:11:27.834 [nioEventLoopGroup-2-1] DEBUG
io.netty.util.ResourceLeakDetectorFactory - Loaded default
ResourceLeakDetector: io.netty.util.ResourceLeakDetector#14896267
13:11:32.859 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[10561: /10.202.1.252:53], DefaultDnsQuestion(dev-542348.okta.com. IN
A) 13:11:37.868 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[40341: /10.180.1.20:53], DefaultDnsQuestion(dev-542348.okta.com. IN
A) 13:11:42.880 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[10043: /10.180.1.201:53], DefaultDnsQuestion(dev-542348.okta.com. IN
A) 13:11:47.892 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[28012: /10.180.1.202:53], DefaultDnsQuestion(dev-542348.okta.com. IN
A) 13:11:52.898 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[8689: /180.235.155.185:53], DefaultDnsQuestion(dev-542348.okta.com.
IN A) 13:11:53.045 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsNameResolver - [id: 0xc4d4fa95] RECEIVED: UDP
[8689: /180.235.155.185:53], DatagramDnsResponse(from:
/180.235.155.185:53, to: /0.0.0.0:56937, 8689, QUERY(0), Refused(5),
RD) DefaultDnsQuestion(dev-542348.okta.com. IN A) 13:11:53.045
[nioEventLoopGroup-2-1] DEBUG io.netty.resolver.dns.DnsQueryContext -
[id: 0xc4d4fa95] WRITE: UDP, [43413: /192.168.1.1:53],
DefaultDnsQuestion(dev-542348.okta.com. IN A) 13:11:53.171
[nioEventLoopGroup-2-1] DEBUG io.netty.resolver.dns.DnsNameResolver -
[id: 0xc4d4fa95] RECEIVED: UDP [43413: /192.168.1.1:53],
DatagramDnsResponse(from: /192.168.1.1:53, to: /0.0.0.0:56937, 43413,
QUERY(0), NoError(0), RD RA) DefaultDnsQuestion(dev-542348.okta.com.
IN A) DefaultDnsRawRecord(dev-542348.okta.com. 300 IN CNAME 25B)
DefaultDnsRawRecord(ok11-crtrs.tng.okta.com. 202 IN CNAME 66B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41957 IN
NS 23B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41957 IN NS
25B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41957 IN NS 21B)
DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41957 IN NS 22B)
13:11:53.175 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[61505: /10.150.1.252:53],
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:11:58.185 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[46866: /10.202.1.252:53],
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:12:03.187 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[39078: /10.180.1.20:53],
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:12:08.196 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[50281: /10.180.1.201:53],
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:12:13.208 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[53416: /10.180.1.202:53],
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:12:18.214 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[19383: /180.235.155.185:53],
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:12:18.320 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsNameResolver - [id: 0xc4d4fa95] RECEIVED: UDP
[19383: /180.235.155.185:53], DatagramDnsResponse(from:
/180.235.155.185:53, to: /0.0.0.0:56937, 19383, QUERY(0), Refused(5),
RD)
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:12:18.320 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0xc4d4fa95] WRITE: UDP,
[39446: /192.168.1.1:53],
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:12:18.367 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsNameResolver - [id: 0xc4d4fa95] RECEIVED: UDP
[39446: /192.168.1.1:53], DatagramDnsResponse(from: /192.168.1.1:53,
to: /0.0.0.0:56937, 39446, QUERY(0), NoError(0), RD RA)
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
34 IN A 4B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
34 IN A 4B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
34 IN A 4B) 2[dev-542348.okta.com/3.15.36.194,
dev-542348.okta.com/3.15.36.192, dev-542348.okta.com/3.15.36.193]
And below are the logs from MACOS
13:23:20.937 [nioEventLoopGroup-2-1] DEBUG io.netty.util.Recycler -
-Dio.netty.recycler.maxCapacityPerThread: 4096 13:23:20.937 [nioEventLoopGroup-2-1] DEBUG io.netty.util.Recycler -
-Dio.netty.recycler.ratio: 8 13:23:20.937 [nioEventLoopGroup-2-1] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.chunkSize: 32
13:23:20.937 [nioEventLoopGroup-2-1] DEBUG io.netty.util.Recycler -
-Dio.netty.recycler.blocking: false 13:23:20.941 [nioEventLoopGroup-2-1] DEBUG io.netty.buffer.AbstractByteBuf -
-Dio.netty.buffer.checkAccessible: true 13:23:20.941 [nioEventLoopGroup-2-1] DEBUG io.netty.buffer.AbstractByteBuf -
-Dio.netty.buffer.checkBounds: true 13:23:20.941 [nioEventLoopGroup-2-1] DEBUG
io.netty.util.ResourceLeakDetectorFactory - Loaded default
ResourceLeakDetector: io.netty.util.ResourceLeakDetector#41e61236
13:23:21.046 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsNameResolver - [id: 0x0ea5f4b3] RECEIVED: UDP
[52950: /192.168.1.1:53], DatagramDnsResponse(from: /192.168.1.1:53,
to: /0.0.0.0:49776, 52950, QUERY(0), NoError(0), RD RA)
DefaultDnsQuestion(dev-542348.okta.com. IN A)
DefaultDnsRawRecord(dev-542348.okta.com. 300 IN CNAME 25B)
DefaultDnsRawRecord(ok11-crtrs.tng.okta.com. 217 IN CNAME 66B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41273 IN
NS 23B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41273 IN NS
25B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41273 IN NS 21B)
DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41273 IN NS 22B)
13:23:21.048 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsQueryContext - [id: 0x0ea5f4b3] WRITE: UDP,
[2623: /192.168.1.1:53],
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A) 13:23:21.084 [nioEventLoopGroup-2-1] DEBUG
io.netty.resolver.dns.DnsNameResolver - [id: 0x0ea5f4b3] RECEIVED: UDP
[2623: /192.168.1.1:53], DatagramDnsResponse(from: /192.168.1.1:53,
to: /0.0.0.0:49776, 2623, QUERY(0), NoError(0), RD RA)
DefaultDnsQuestion(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
IN A)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B)
DefaultDnsRawRecord(ok11-crtr-tls12-nlb-7429507134c4aa04.elb.us-east-2.amazonaws.com.
60 IN A 4B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41273 IN
NS 23B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41273 IN NS
25B) DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41273 IN NS 21B)
DefaultDnsRawRecord(elb.us-east-2.amazonaws.com. 41273 IN NS 22B)
2[dev-542348.okta.com/3.15.36.192, dev-542348.okta.com/3.15.36.194,
dev-542348.okta.com/3.15.36.193] 13:23:23.184 [nioEventLoopGroup-2-1]
DEBUG io.netty.buffer.PoolThreadCache - Freed 8 thread-local buffer(s)
from thread: nioEventLoopGroup-2-1
As the reactor-netty code is also giving the same issue with windows but working fine on MACOS this looks like an issue with Netty.
Any suggestion to fix this