In my code i m trying to iterate n time building a client and sending an input stream to a remote endpoint:
for (int i=0; i<=100; i ++) {
Client client = Client.create();
WebResource webResource = client.resource("https://endpoint/EndOfDayRapport/eof_" + System.currentTimeMillis());
SPMResponse response5 = webResource
.type("text/xml").put(SPMResponse.class,myInputStream);
System.out.println(response5);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
The input stream contains an xml body of the message.
What i noticed is that on the
1st iteration... the request succeedes
2nd iteration the response is 400 bad request...
3rd 400 bad request....
so on...
if i change the inputstream to string... taking the real xml and sending it with the PUT method.
1st iteration... the request succeedes
2nd iteration... the request succeedes
3rd iteration... the request succeedes
so on...
What is the difference? why does it work with string and not with input stream?
(it seems somehow the inputstream changes after the first iteration)
Performance wise is it better using the inputstream or string?
Please check your inputstream. It seems to me that you have the following problem:
On the first request your inputstream is in a valid state with data available
On the following requests your stream is already at it's end. So there is nothing to read and you send empty data which the jersey handles as "bad request"
So try to use an inputstream that you can rewind and rewind it on every loop iteration.
But maybe that kind of defeats the whole point of using a stream anyway.
Related
I am developing a proxy service to a Minio server using WebClient that handles all Minio/S3 API endpoints. Most of them work fine, but I have encountered one case in which the PUT operation seems to get hung up when trying to set the body of the request to either an InputStream, a File, or a Resource pointing to it. (See epilogue at the bottom, as I'm left wondering where the problem really is.)
The only way I've found to make it work is to read the file contents to an in-memory byte array. The following baseline works, for example:
WebClient.UriSpec<WebClient.RequestBodySpec> uriSpec = client.method(request.getMethod());
WebClient.RequestBodySpec bodySpec = uriSpec.uri(uri);
WebClient.RequestHeadersSpec<?> headersSpec = bodySpec;
try {
// read file to byte array; works fine
byte[] bytes = Files.readAllBytes(Path.of(file.get().getFile().toURI()));
// set it to the request body
headersSpec = bodySpec.bodyValue(bytes);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
// manipulate some headers
headersSpec = headersSpec.headers(httpHeaders -> ...);
// execute the request; works fine in this scenario
return headersSpec.exchangeToMono(resp -> ...)
.doOnError(throwable -> log.error("Trouble proxying request: " + throwable.getMessage(), throwable));
However, every alternative that I try to stream this content instead, results in a request that seems to hang in the headersSpec.exchangeToMono invocation. I don't see any errors on the proxy service, and the client socket eventually gives up:
java.net.SocketTimeoutException: timeout
client-tester_1 | at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:143) ~[okio-jvm-2.8.0.jar:na]
client-tester_1 | Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Some examples of failure (or, paraphrasing Edison, I've successfully found at least a dozen ways that do not work):
// Use same byte array as above; Hangs
Resource resource = new ByteArrayResource(bytes);
headersSpec = bodySpec.bodyValue(resource);
// Read an input stream from the file (this one relies on a HttpMessageWriter<InputStream> that I configured on the client); Hangs
InputStream bodyStream = new BufferedInputStream(Files.newInputStream(Path.of(file.get().getFile().toURI())));
headersSpec = bodySpec.bodyValue(bodyStream);
// Resource for the file; Hangs
Resource resource = new FileSystemResource(Path.of(file.get().getFile().toURI()));
Flux<DataBuffer> flux = DataBufferUtils.read(resource, DefaultDataBufferFactory.sharedInstance, 4096);
headersSpec = bodySpec.body(flux, DataBuffer.class);
// Different resource; Hangs
Resource resource = new UrlResource(file.get().getFile().toURI());
headersSpec = bodySpec.bodyValue(resource);
// Try BodyInserters; Hangs
Flux<DataBuffer> flux = DataBufferUtils.read(Path.of(file.get().getFile().toURI()), DefaultDataBufferFactory.sharedInstance, 4096);
headersSpec = bodySpec.body(BodyInserters.fromDataBuffers(flux));
// Yet another attempt; Take a guess...
InputStream bodyStream = new BufferedInputStream(Files.newInputStream(Path.of(file.get().getFile().toURI())));
headersSpec = bodySpec.body(BodyInserters.fromResource(resource));
I'm using recent versions of the relevant libraries:
org.springframework.boot:spring-boot-starter-webflux -> 2.7.5
org.springframework.boot:spring-boot-starter-reactor-netty:2.7.5
org.springframework:spring-core:5.3.23
Epiloge I'm wondering if the problem is not necessarily with Spring/WebClient/Netty -- as many of these code samples were inspired by other examples I've found -- but rather by some nuance on the Minio server?
I am new to Reactive programming paradigm, but recently I have decided to base a simple Http client on Spring WebClient, since the old sync RestTemplate is already under maintenance and might be deprecated in upoming releases.
So first I had a look at Spring documentation and, after that, I've searched the web for examples.
I must say that (only for the time being) I have consciously decided not to go through the Reactor lib documentation, so beyond the Publisher-Subscriber pattern, my knowledge about Mono's and Flux's is scarce. I focused instead on having something working.
My scenario is a simple POST to send a callback to a Server from which the client is only interested in response status code. No body is returned. So I finally came up with this code snippet that works:
private void notifyJobSuccess(final InternalJobData jobData) {
SuccessResult result = new SuccessResult();
result.setJobId(jobData.getJobId());
result.setStatus(Status.SUCCESS);
result.setInstanceId(jobData.getInstanceId());
log.info("Result to send back:" + System.lineSeparator() + "{}", result.toString());
this.webClient.post()
.uri(jobData.getCallbackUrl())
.body(Mono.just(result), ReplaySuccessResult.class)
.retrieve()
.onStatus(s -> s.equals(HttpStatus.OK), resp -> {
log.info("Expected CCDM response received with HttpStatus = {}", HttpStatus.OK);
return Mono.empty();
})
.onStatus(HttpStatus::is4xxClientError, resp -> {
log.error("CCDM response received with unexpected Client Error HttpStatus {}. "
+ "The POST request sent by EDA2 stub did not match CCDM OpenApi spec", resp.statusCode());
return Mono.empty();
})
.onStatus(HttpStatus::is5xxServerError, resp -> {
log.error("CCDM response received with unexpected Server Error HttpStatus {}", resp.statusCode());
return Mono.empty();
}).bodyToMono(Void.class).subscribe(Eda2StubHttpClient::handleResponseFromCcdm);
}
My poor understanding of how the reactive WebClient works starts with the call to subscribe. None of the tens of examples that I checked before coding my client included such a call, but the fact is that before I included that call, the Server was sitting forever waiting for the request.
Then I bumped into the mantra "Nothing happens until you subscribe". Knowing the pattern Plublisher-Subscriber I knew that, but I (wrongly) assumed that the subscription was handled by WebClient API, in any of the exchage, or bodyToMono methods... block() definitely must subscribe, because when you block it, the request gets out at once.
So my first question is: is this call to subscribe() really needed?
Second question is why the method StubHttpClient::handleResponse is never called back. For this, the only explanation that I find is that as the Mono returned is a Mono<Void>, because there is nothing in the response besides the status code, as it is never instantiated, the method is totally dummy... I could even replace it by just .subscribe(). Is this a correct assumption.
Last, is it too much to ask for a complete example of a a method receiving a body in a Mono that is later consumed? All examples I find just focus on getting the request out, but how the Mono or Flux is later consumed is now beyond my understanding... I know that I have to end up checking the Reactor doc sooner better than later, but I would appreciate a bit of help because I am having issues with Exceptions and errors handlin.
Thanks!
Some time has passed since I asked for help here. Now I'd like not to edit but to add an answer to my previous question, so that the answer remains clear and separate from he original question and comments.
So here goes a complete example.
CONTEXT: An application, acting as a client, that requests an Access Token from an OAuth2 Authorization server. The Access Token is requested asynchronously to avoid blocking the appliction's thread while the token request is processed at the other end and the response arrives.
First, this is a class that serves Access Token to its clients (method getAccessToken): if the Access Token is already initialized and it's valid, it returns the value stored; otherwise fetches a new one calling the internal method fetchAccessTokenAsync:
public class Oauth2ClientBroker {
private static final String OAUHT2_SRVR_TOKEN_PATH= "/auth/realms/oam/protocol/openid-connect/token";
private static final String GRANT_TYPE = "client_credentials";
#Qualifier("oAuth2Client")
private final WebClient oAuth2Client;
private final ConfigurationHolder CfgHolder;
#GuardedBy("this")
private String token = null;
#GuardedBy("this")
private Instant tokenExpireTime;
#GuardedBy("this")
private String tokenUrlEndPoint;
public void getAccessToken(final CompletableFuture<String> completableFuture) {
if (!isTokenInitialized() || isTokenExpired()) {
log.trace("Access Token not initialized or has exired: go fetch a new one...");
synchronized (this) {
this.token = null;
}
fetchAccessTokenAsync(completableFuture);
} else {
log.trace("Reusing Access Token (not expired)");
final String token;
synchronized (this) {
token = this.token;
}
completableFuture.complete(token);
}
}
...
}
Next, we will see that fetchAccessTokenAsync does:
private void fetchAccessTokenAsync(final CompletableFuture<String> tokenReceivedInFuture) {
Mono<String> accessTokenResponse = postAccessTokenRequest();
accessTokenResponse.subscribe(tr -> processResponseBodyInFuture(tr, tokenReceivedInFuture));
}
Two things happen here:
The method postAccessTokenRequest() builds a POST request and declares how the reponse will be consumed (when WebFlux makes it available once it is received), by using exchangeToMono:
private Mono postAccessTokenRequest() {
log.trace("Request Access Token for OAuth2 client {}", cfgHolder.getClientId());
final URI uri = URI.create(cfgHolder.getsecServiceHostAndPort().concat(OAUHT2_SRVR_TOKEN_PATH));
} else {
uri = URI.create(tokenUrlEndPoint);
}
}
log.debug("Access Token endpoint OAuth2 Authorization server: {}", uri.toString());
return oAuth2Client.post().uri(uri)
.body(BodyInserters.fromFormData("client_id", cfgHolder.getEdaClientId())
.with("client_secret", cfgHolder.getClientSecret())
.with("scope", cfgHolder.getClientScopes()).with("grant_type", GRANT_TYPE))
.exchangeToMono(resp -> {
if (resp.statusCode().equals(HttpStatus.OK)) {
log.info("Access Token successfully obtained");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().equals(HttpStatus.BAD_REQUEST)) {
log.error("Bad request sent to Authorization Server!");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().equals(HttpStatus.UNAUTHORIZED)) {
log.error("OAuth2 Credentials exchange with Authorization Server failed!");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().is5xxServerError()) {
log.error("Authorization Server could not generate a token due to a server error");
return resp.bodyToMono(String.class);
} else {
log.error("Authorization Server returned an unexpected status code: {}",
resp.statusCode().toString());
return Mono.error(new Exception(
String.format("Authorization Server returned an unexpected status code: %s",
resp.statusCode().toString())));
}
}).onErrorResume(e -> {
log.error(
"Access Token could not be obtained. Process ends here");
return Mono.empty();
});
}
The exchangeToMono method does most of the magic here: tells WebFlux to return a Mono that will asynchronously receive a signal as soon as the response is received, wrapped in a ClientResponse, the parameter resp consumed in the lambda. But it is important to keep in mind that NO request has been sent out yet at this point; we are just passing in the Function that will take the ClientResponse when it arrives and will return a Mono<String> with the part of the body of our interest (the Access Token, as we will see).
Once the POST is built and the Mono returned, then the real thing starts when we subscribe to the Mono<String> returned before. As the Reacive mantra says: nothing happens until you subscribe or, in our case, the request is not actually sent until something attempts to read or wait for the response. There are other ways in WebClient fluent API to implicitly subscribe, but we have chosen here the explicit way of returing the Mono -which implements the reactor Publisher interface- and subscribe to it. Here we blocking the thread no more, releasing CPU for other stuff, probably more useful than just waiting for an answer.
So far, so good: we have sent out the request, released CPU, but where the processing will continue whenever the response comes? The subscribe() method takes as an argument a Consumer parameterized in our case with a String, being nothing less than the body of the response we are waiting for, wrapped in Mono. When the response comes, WebFlux will notify the event to our Mono, which will call the method processResponseBodyInFuture, where we finally receive the response body:
private void processResponseBodyInFuture(final String body, final CompletableFuture<String> tokenReceivedInFuture) {
DocumentContext jsonContext = JsonPath.parse(body);
try {
log.info("Access Token response received: {}", body);
final String aTkn = jsonContext.read("$.access_token");
log.trace("Access Token parsed: {}", aTkn);
final int expiresIn = jsonContext.read("$.expires_in");
synchronized (this) {
this.token = aTkn;
this.tokenExpireTime = Instant.now().plusSeconds(expiresIn);
}
log.trace("Signal Access Token request completion. Processing will continue calling client...");
tokenReceivedInFuture.complete(aTkn);
} catch (PathNotFoundException e) {
try {
log.error(e.getMessage());
log.info(String.format(
"Could not extract Access Token. The response returned corresponds to the error %s: %s",
jsonContext.read("$.error"), jsonContext.read("$.error_description")));
} catch (PathNotFoundException e2) {
log.error(e2.getMessage().concat(" - Unexpected json content received from OAuth2 Server"));
}
}
}
The invocation of this method happens as soon as the Mono is signalled about the reception of the response. So here we try to parse the json content with an Access Token and do something with it... In this case call complete() onto the CompletableFuture passed in by the caller of the initial method getAccessToken, that hopefully will know what to do with it. Our job is done here... Asynchronously!
Summary:
To summarize, these are the basic considerations to have your request sent out and the responses processed when you ise reactive WebClient:
Consider having a method in charge of preparing the request by means of the WebClient fluent API (to set http method, uri, headers and body). Remember: by doing this you are not sending any request yet.
Think on the strategy you will use to obtain the Publisher that will be receive the http client events (response or errors). retreive() is the most straight forward, but it has less power to manipulate the response than exchangeToMono.
Subscribe... or nothing will happen.
Many examples you will find around will cheat you: they claim to use WebClient for asyncrhony, but then they "forget" about subscribing to the Publisher and call block() instead. Well, while this makes things easier and they seem to work (you will see responses received and passed to your application), the thing is that this is not asynchronous anymore: your Mono (or Flux, whatever you use) will be blocking until the response arrives. No good.
Have a separate method (being the Consumer passed in the subscribe() method) where the response body is processed.
I am using the spring WebClient to make two calls in parallel.
One of the call results is passed back as a ResponseEntity, and the other result is inspected and then disregarded. Although the transactions are both successful, I see an IllegalReferenceCountException that occurs before any of the WebClient calls actually get executed.
What I see in my logging is that the container logs the exception, then my two HTTP requests get executed successfully, and one of these responses gets returned to the client.
If the shouldBackfill() function returns false, then I execute one HTTP request and return that response (and the IllegalReferenceCountException does not occur).
I was initially thinking that I should release the reference in the second response that I disregard.
If I attempt to call releaseBody() directly on the WebClient response. (See https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/reactive/function/client/ClientResponse.html), this does not help. I assume now that the container is detecting that the WebClient request that I disregarded is in an illegal state, hence the error detection. But what I don't understand is that the actual request occurs AFTER the IllegalReferenceCountException gets logged.
Any ideas here on how to get around this? I am wondering if the exception is actually NOT any kind of leak.
The code looks like this:
fun execute(routeHttpRequest: RouteHttpRequest): Mono<ResponseEntity<String>> =
propertyRepository.getProperty(routeHttpRequest.propertyId.orDefault())
.flatMap {
val status = it.getOrElse { unknownStatus(routeHttpRequest.propertyId.orDefault()) }
val response1 = execute(routeHttpRequest, routingRepository.webClientFor(routeHttpRequest))
if (shouldBackfill(routeHttpRequest, status.type())) {
val response2 =
execute(routeHttpRequest, routingRepository.shadowOrBackfillWebClientFor(routeHttpRequest))
zip(response1, response2).map { response ->
compare(routeHttpRequest, response.t1, response.t2, status.type())
response.t1 // response.t2 is NOT returned here..
}
} else response1
}
// This function returns a wrapper on a spring Webclient that makes an HTTP post.
//
private fun execute(routeHttpRequest: RouteHttpRequest, client: Mono<MyWebClient>) =
client
.flatMap { dataService.execute(routeHttpRequest, it) }
.subscribeOn(Schedulers.elastic()) // TODO: consider a dedicated executor here?
private fun shouldBackfill(routeHttpRequest: RouteHttpRequest, migrationStatus: MigrationStatusType): Boolean {
... this logic returns true when we should execute 2 requests in parallel
}
Here's the exception and partial trace:
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
Sorry for not posting the exact code. Fix- I was passing the incoming http request org.springframework.core.io.buffer.DataBuffer directly to the WebClient request body. This was intentional because my application is acting as a proxy service. The problem came up when I attempted to make two outbound WebClient calls in parallel - the container was trying to release the underlying buffer twice, and the IllegalReferenceCountException occurs. My fix was to just copy the DataBuffer byte array into a new buffer before sending the request along to it's destination.
I requested some data from server. So, i started asyncrontask. getGrup is get request of server.
getGrup="http://192.168.56.1:8084/Server/grup/getGrupId/tarihVeSaat/"
+grup.getGrupTarihVeSaat().getTime()+"/yoneticiId/"+grup.getYoneticiId()+";";
new GrupOlusturRequest().execute(getGrup);
sendUyeler="http://192.168.56.1:8084/Server/grupUyeleri/ekle";
while(true){
if(getGId){
hazırlaGrupUyeleri();
break;
}
But, Sometimes br.readline() returns null, i tested someone else rest client . it is always working. try-catch didn't throw exception. I can't understand that why it returns null variable.
url = new URL(param);
connection=(HttpURLConnection) url.openConnection();
connection.connect();
connection.setReadTimeout(100);
connection.setConnectTimeout(100);
is=connection.getInputStream();
br=new BufferedReader(new InputStreamReader(is,"UTF-8"));
String satir;
while((satir=br.readLine())!=null){
json2=new JSONObject(satir);
}
status=connection.getResponseCode();
Serverside:
Data sends as json from server. This method is resteasy get method. This method always works on postman rest client.
#GET
#Produces(MediaType.APPLICATION_JSON)
#Path("/getGrupId/tarihVeSaat/{tarihVeSaat}/yoneticiId/{yoneticiId}")
public Grup getGrupId(#PathParam ("tarihVeSaat") String tarihVeSaat,#PathParam ("yoneticiId") int yoneticiId){
Grup grup=null;
Timestamp tarih=new Timestamp(new java.util.Date(Long.parseLong(tarihVeSaat)).getTime());
try{
VeritabaniIslemleri vi=new VeritabaniIslemleri();
vi.baglan();
grup=vi.getGrupId(TarihVeSaat.tarihZamanGoster(tarih),yoneticiId);
vi.baglantiyiKes();
}catch(Exception e){
Logger.getLogger(KullaniciRestController.class.getName()).log(Level.SEVERE,null,e);
}
return grup;
}
According to the JavaDocs for BufferedReader.readLine() the only reason the method should return null is reaching the end of stream:
Returns:
A String containing the contents of the line, not including any line-termination characters, or null if the end of the stream has been reached
If a timeout or communication problem occurred you should get an exception thrown.
Is it possible the web application you're connecting to is terminating the connection without flushing its output buffer? It sounds like the issue is probably on the other end of the connection.
Hi I am using loadJSONObject function in Processing IDE to receive a JSON Object from a URL.
Following is my code:
JSONObject jsonUserLocations = loadJSONObject("http://smrt.utd.sg/eLocation/getLocs.php?userId="+usrID+"&sTime="+strtTime+"&eTime="+endTym);
This code works when the php returns some data (for some users).
The problem occurs when it doesn't return any data. (php doesn't return any data when there is no data, browser shows a blank page. this means user has no location data)
At this instance, the Processing IDE gives me an error saying;
a jsonobject text must begin with {
My question is how can I handle empty JSON Object in this type of situation? I need to skip if this is empty and request data for next user. Your help is much appreciated.
Thanks,
Hasala
Edit:
This is sample json object I receive when there is data.
{"locations":[{"latitude":"1.3809274","longitude":"103.7654596","startTime":"1421918587868","duration":"0","accuracy":"30"},{"latitude":"1.3805307","longitude":"103.7661015","startTime":"1421941711737","duration":"0","accuracy":"45"},{"latitude":"1.3805304","longitude":"103.7660959","startTime":"1421942011727","duration":"0","accuracy":"45"},{"latitude":"1.3799822","longitude":"103.7658037","startTime":"1421942311835","duration":"0","accuracy":"82.5"}],"success":1}
You have two options:
You could read the String form the URL first, and if it's blank, don't bother with the parsing.
Or you could just catch the exception that Processing throws:
try{
JSONObject jsonUserLocations = loadJSONObject("http://smrt.utd.sg/eLocation/getLocs.php?userId="+usrID+"&sTime="+strtTime+"&eTime="+endTym);
}
catch(JSONException e){
e.printStackTrace();
//json was blank, do something else
}