How to set netty log level on selected spring boot Webclient instances - spring-boot

We have 2 different instances of spring boot Webclients. We want one of them to log all requests/responses and the other to log nothing. Settiing:
logging.level.reactor.netty.http.client=debug
in the application.properties file causes both instances to log requests/responses. Is there a way to programmatically set the log level on one of the instances to log debug levels and the other to not?
We create the first webclient like this:
WebClient.create();
For this one we don't want debug logging.
The second one, for which we want logging, we create like this:
return WebClient.builder()
.clientConnector(createWebClientWithTimeout())
// Add base url to all requests (callers only need to add the path and query params)
.baseUrl(baseUrl)
// Increase the buffer size from 256K to 1M
.codecs(configurer -> configurer
.defaultCodecs()
.maxInMemorySize(1024 * 1024))
// Filter to add bearer token to auth header before sending request
.filter((request, next) -> getToken(custId).map(setAuthHeader(request)).flatMap(next::exchange))
// Filter to send the request, and try again if it has an auth error
.filter((request, next) -> next.exchange(request).flatMap(clientResponse -> {
:::
return Mono.just(clientResponse);
})).build();
and
private ClientHttpConnector createWebClientWithTimeout() {
// create reactor netty HTTP client
HttpClient httpClient = HttpClient.newConnection()
.wiretap("reactor.netty.http.client.HttpClient",
LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL)
.compress(true)
.tcpConfiguration(tcpClient -> {
tcpClient = tcpClient.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, linkedinConnectTimeoutInSeconds * 1000)
.doOnConnected(conn -> conn
.addHandlerLast(new ReadTimeoutHandler(linkedinReadTimeoutInSeconds, TimeUnit.SECONDS))
.addHandlerLast(new WriteTimeoutHandler(linkedinWriteTimeoutInSeconds, TimeUnit.SECONDS)));
return tcpClient;
});
// create a client http connector using above http client
return new ReactorClientHttpConnector(httpClient);
}
If we don't set logging.level.reactor.netty.http.client=debug, neither of these webclients log netty debug statements. If we set the debug parameter, both log the requests, the second one also logs request/response headers and bodies.

Related

Handling Mappings when migrating Get Index call from Transport client to Rest High Level Client

We are migrating from Transport Client to High Level Rest Client. How do we reconcile the difference in the mappings field of GetIndexResponse object returned from Transport client vs Rest High Level Client?
For transport client, we use this code to obtain the index information:
GetIndexResponse response = client.get()
.admin()
.indices()
.prepareGetIndex()
.setFeatures(GetIndexRequest.Feature.MAPPINGS, GetIndexRequest.Feature.SETTINGS)
.setIndices(indices)
.get();
The response mappings field is a ImmutableOpenMap<String, ImmutableOpenMap<String, MappingMetaData>>
But for rest high level client, the mappings field is just a Map<String, MappingMetadata>.
Here is the code for High Level Rest Client:
GetIndexRequest request = new GetIndexRequest(indices);
request.addFeatures(GetIndexRequest.Feature.MAPPINGS, GetIndexRequest.Feature.SETTINGS);
try {
GetIndexResponse response = esClient.indices().get(request, RequestOptions.DEFAULT);
return
} catch (IOException ex) {
throw new SafeRuntimeException(ex);
}
}
These are the response object classes:
Transport Client: org.elasticsearch.action.admin.indices.get.GetIndexResponse
Rest High Level Client: org.elasticsearch.client.indices.GetIndexResponse
for mappings use the GetMappings request:
ImmutableOpenMap<String, ?> mappings = esclient.admin().indices().getMappings(new GetMappingsRequest()).actionGet().getMappings();
and you can also specify the indices you want.
EDIT: ah i see, you mean this:
client.indices().getMapping(new GetMappingsRequest().indices("index_name1", "index_name2"), RequestOptions.DEFAULT);
client.indices().getSettings(new GetSettingsRequest().indices("index_name1", "index_name2"), RequestOptions.DEFAULT);
when "client" is an instance of the High level REST client.

Apache Http Components - How to timeout CONNECT request to a proxy?

Timeout Without Using Proxy
I start netcat in my local as follows, which basically listens to connections on port 9090:
netcat -l -p 9090
And using Apache HttpComponents, I create a connection to it with a timeout of 4 seconds..
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(4000)
.setConnectTimeout(4000)
.setConnectionRequestTimeout(4000)
.build();
HttpGet httpget = new HttpGet("http://127.0.0.1:9090");
httpget.setConfig(requestConfig);
try (CloseableHttpResponse response = HttpClients.createDefault().execute(httpget)) {}
In terminal (where I have netcat running) I see:
??]?D???;#???9?Mۡ?NR?w?{)?V?$?(=?&?*kj?
?5??98?#?'<?%?)g#? ?/??32?,?+?0??.?2???/??-?1???D
<!-- 4 seconds later -->
read(net): Connection reset by peer
In client side what I see is:
Exception in thread "main" org.apache.http.conn.ConnectTimeoutException:
Connect to 127.0.0.1:9090 [/127.0.0.1] failed: Read timed out
This is all expected.
Timeout Using Proxy
I change the client code slightly and configure a proxy, following the docs here.
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(4000)
.setConnectTimeout(4000)
.setConnectionRequestTimeout(4000)
.build();
HttpHost proxy = new HttpHost("127.0.0.1", 9090);
DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy);
CloseableHttpClient httpclient = HttpClients.custom()
.setRoutePlanner(routePlanner)
.build();
HttpGet httpget = new HttpGet("https://127.0.0.1:9090");
httpget.setConfig(requestConfig);
try (CloseableHttpResponse response = httpclient.execute(httpget)) {}
And again start netcat, and this time on serverside
CONNECT 127.0.0.1:9090 HTTP/1.1
Host: 127.0.0.1:9090
User-Agent: Apache-HttpClient/4.4.1 (Java/1.8.0_212)
But timeout is not working for CONNECT. I just wait forever..
How can I configure the httpclient to timeout for 4 seconds just like in the first case I described?
RequestConfig only take effect once a connection to the target via the specific route has been fully established . They do not apply to the SSL handshake or any CONNECT requests that take place prior to the main message exchange.
Configure socket timeout at the ConnectionManager level to ensure connection level operations time out after a certain period of inactivity.
One possibility:
// This part is the same..
httpget.setConfig(requestConfig);
ExecutorService executorService = Executors.newSingleThreadExecutor();
Callable<CloseableHttpResponse> callable = () -> {
try (CloseableHttpResponse response = httpclient.execute(httpget)) {
return response;
}
};
Future<CloseableHttpResponse> future = executorService.submit(callable);
try {
future.get(4, TimeUnit.SECONDS);
} catch (InterruptedException | ExecutionException | TimeoutException e) {
httpget.abort();
executorService.shutdownNow();
}
But I am open to other suggestions..

Redirect HTTP requests to HTTPS in netty

I am modifying elasticsearch code to configure HTTPS without x-pack and reverse proxies.
I modified initchannel() method in the netty4HttpServerTransport file , https is working fine,but i want to redirect http to https..
The code is,
char[] password = "your5663".toCharArray();
KeyStore ks = KeyStore.getInstance("JKS");
ks.load(new FileInputStream("C:/OpenSSL-Win64/bin/keystore1.jks"),password);
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(ks, password);
TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmf.init(ks);
TrustManager[] tm = tmf.getTrustManagers();
SSLContext sslContext = SSLContext.getInstance("TLSv1.3");
sslContext .init( kmf.getKeyManagers(), tm, null);
SSLEngine sslengine = sslContext .createSSLEngine();
sslengine.setUseClientMode(false);
String[] DEFAULT_PROTOCOLS = { "TLSv1", "TLSv1.1", "TLSv1.2","TLSv1.3" };
String[] DEFAULT_CIPHERS = {"TLS_RSA_WITH_AES_128_CBC_SHA256", "TLS_RSA_WITH_AES_128_CBC_SHA"};
sslengine.setEnabledProtocols(DEFAULT_PROTOCOLS);
sslengine.setEnabledCipherSuites(DEFAULT_CIPHERS);
SslHandler sslHandler = new SslHandler(sslengine);
ch.pipeline().addLast("ssl", sslHandler);
ch.pipeline().addAfter("ssl","handshake",new StringEventHandler());
How do i make http to https redirect in this code.
Redirect works on the payload (http) level, not ssl transport level. You would need to listen on both protocol (http and https) and on the http channel you can respond with redirect status code. Long story short - there is no direct place on in your code you can do that.
Very commonly a proxy server is used for this task. I am not sure if you can do it in elasticsearch, you can try to configure a filter servlet to check the protocol respond with a redirect. This may be helpful https://github.com/elastic/elasticsearch-transport-wares
Another fact - if the redirect is for service clients (not browser based ui), the clients may/will consider a redirect response an an error response. Depending on your environment - maybe you can just expose the ssl endpoint (no redirects) and clients will have to comply
Netty has a built in handler for this, OptionalSslHandler.
You put it at the front of your pipeline and it detects if the message is encrypted or not. If it is, then the message will be sent onto the normal SSL pipeline, if not then you can specify somewhere else to send it, e.g. to a 301 redirect to https.
You could either use this Netty version or make your own handler that does something similar.
However, to use the Netty version you will need to refactor slightly to produce a Netty SslContext io.netty.handler.ssl.SslContext, instead of an SSLEngine.
Something like this:
char[] password = "your5663".toCharArray();
KeyStore ks = KeyStore.getInstance("JKS");
ks.load(new FileInputStream("C:/OpenSSL-Win64/bin/keystore1.jks"),password);
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(ks, password);
SslContext sslContext = SslContextBuilder.forServer(keyManagerFactory).build();
ch.pipeline().addLast("ssl", sslHandler);
// this is an imaginary handler you create that sends HTTP a 301 to HTTPS
// non-SSL can be detected easily because there is no SslHandler on this channel
ch.pipeline().addLast("redirectHandler", new RedirectHandler());
ch.pipeline().addLast("handshake",new StringEventHandler());

How do you use WebFlux to parse an event stream that does not conform to Server Sent Events?

I am trying to use WebClient to deal with the Docker /events endpoint. However, it does not conform to the text/eventstream contract in that each message is separated by 2 LFs. It just sends it as one JSON document followed by another.
It also sets the MIME type to application/json rather than text/eventstream.
What I am thinking of but not implemented yet is to create a node proxy that will add the required line feed and put that in between but I was hoping to avoid that kind of workaround.
Instead of trying to handle a ServerSentEvent, just receive it as a String. Then attempt to parse it as JSON (ignoring the ones that fail which I am presuming may happen but I haven't hit it myself)
#PostConstruct
public void setUpStreamer() {
final Map<String, List<String>> filters = new HashMap<>();
filters.put("type", Collections.singletonList("service"));
WebClient.create(daemonEndpoint)
.get()
.uri("/events?filters={filters}",
mapper.writeValueAsString(filters))
.retrieve()
.bodyToFlux(String.class)
.flatMap(Mono::justOrEmpty)
.map(s -> {
try {
return mapper.readValue(s, Map.class);
} catch (IOException e) {
log.warn("unable to parse {} as JSON", s);
return null;
}
})
.flatMap(Mono::justOrEmpty)
.subscribe(
event -> {
log.trace("event={}", event);
refreshRoutes();
},
throwable -> log.error("Error on event stream: {}", throwable.getMessage(), throwable),
() -> log.warn("event stream completed")
);
}

How to catch okhttp3 WebSocket network activity using okhttp3.Interceptor?

I have an okhttp3 (3.9.1) WebSocket instance and would like to view all it's network requests and responses. I tried to add some okhttp3.Interceptor instances to OkHttpClient instance before creating WebSocket on it but had no luck in viewing network activity. Here's sample code which demonstrates what I've tried to do:
package sample
import okhttp3.*
import java.io.IOException
import java.lang.Thread.sleep
fun main(args: Array<String>) {
val listener = object : WebSocketListener() {
override fun onMessage(webSocket: WebSocket?, text: String?) {
println("Got server message: $text")
}
}
val dummyInterceptor = Interceptor { chain ->
val request = chain.request()
val response = chain.proceed(request)
println("Dummy interceptor fired!\n\nRequest: ${request.headers()}\nResponse: ${response.headers()}")
return#Interceptor response
}
val dummyNetworkInterceptor = Interceptor { chain ->
val request = chain.request()
val response = chain.proceed(request)
println("Dummy network interceptor fired!\n\nRequest: ${request.headers()}\nResponse: ${response.headers()}")
return#Interceptor response
}
val okHttpClient = OkHttpClient.Builder()
.addInterceptor(dummyInterceptor)
.addNetworkInterceptor(dummyNetworkInterceptor)
.build()
val request = Request.Builder().url("ws://echo.websocket.org").build()
val webSocket = okHttpClient.newWebSocket(request, listener)
webSocket.send("Hello1!")
webSocket.send("Hello2!")
webSocket.send("Hello3!")
sleep(2000) //Just for this sample to ensure all WS requests done
println("\n\n\tSome network activity\n\n")
okHttpClient.newCall(Request.Builder().get().url("http://echo.websocket.org").build()).enqueue(object : Callback {
override fun onFailure(call: Call?, exc: IOException?) {
println("OnFailure: ${exc?.message}")
}
override fun onResponse(call: Call?, response: Response?) {
println("OnResponse: ${response?.headers()}")
}
})
}
I tried to dive into okhttp3 source code and didn't find any reason why any of my interceptors doesn't fire on WS requests but works perfectly for any OkHttpClient request.
Is it a bug in okhttp3 or am I doing something wrong or it's just not possible to monitor WS requests using okhttp3.Interceptor?
WebSocket calls made with OkHttp don't use the interceptor chains that HTTP calls do, therefore you can't monitor them through interceptors.
I've faced this issue before myself, and so I looked at the source code and found the following then:
The regular HTTP calls go through the getResponseWithInterceptorChain() method in the RealCall class, which quite clearly starts the chained call of interceptors for each request.
The okhttp3.internal.ws package that includes the implementation of the WebSocket handling contains no code related to interceptors.
And really, interceptors catching WebSocket requests wouldn't really make sense in the first place. The Request that you can obtain in an interceptor represents an HTTP request, which WebSocket messages are not.
It isn't possible at this point, there's a feature request open for OkHttp but it isn't getting much traction: https://github.com/square/okhttp/issues/4192

Resources