My builds have been failing due to some of the integration tests I've been running. I'm stuck on why it won't work. Here is an example of the output:
I'm using Maven to first build, then it calls the JUnit tests. I'm seeing this 401 Unauthorized message in every single test, and I believe that's what is causing the builds to fail. In my mind, this means there are some permissions / authentication parameters that need to be set. Where would I go about doing this in JUnit?
Edit
#Test
public void testXmlHorsesNonRunners() throws Exception {
String servletUrl = SERVER + "sd/date/2013-01-13/horses/nonrunners";
Document results = issueRequest(servletUrl, APPLICATION_XML, false);
assertNotNull(results);
// debugDocument(results, "NonRunners");
String count = getXPathStringValue(
"string(count(hrdg:data/hrdg:meeting/hrdg:event/hrdg:nonrunner/hrdg:selection))",
results);
assertEquals("non runners", "45", count);
}
If you can, try to ignore the detail. Effectively, this is making a request. This is a sample of a test that uses the issueRequest method. This method is what makes HTTP requests. (This is a big method, which is why I didn't post it originally. I'll try to make it as readable as possible.
logger.info("Sending request: " + servletUrl);
HttpGet httpGet = null;
// InputStream is = null;
DefaultHttpClient httpclient = null;
try {
httpclient = new DefaultHttpClient();
doFormLogin(httpclient, servletUrl, acceptMime, isIrishUser);
httpGet = new HttpGet(servletUrl);
httpGet.addHeader("accept", acceptMime);
// but more importantly now add the user agent header
setUserAgent(httpGet, acceptMime);
logger.info("executing request" + httpGet.getRequestLine());
// Execute the request
HttpResponse response = httpclient.execute(httpGet);
// Examine the response status
StatusLine statusLine = response.getStatusLine();
logger.info(statusLine);
switch (statusLine.getStatusCode()) {
case 401:
throw new HttpResponseException(statusLine.getStatusCode(),
"Unauthorized");
case 403:
throw new HttpResponseException(statusLine.getStatusCode(),
"Forbidden");
case 404:
throw new HttpResponseException(statusLine.getStatusCode(),
"Not Found");
default:
if (300 < statusLine.getStatusCode()) {
throw new HttpResponseException(statusLine.getStatusCode(),
"Unexpected Error");
}
}
// Get hold of the response entity
HttpEntity entity = response.getEntity();
Document doc = null;
if (entity != null) {
InputStream instream = entity.getContent();
try {
// debugContent(instream);
doc = documentBuilder.parse(instream);
} catch (IOException ex) {
// In case of an IOException the connection will be released
// back to the connection manager automatically
throw ex;
} catch (RuntimeException ex) {
// In case of an unexpected exception you may want to abort
// the HTTP request in order to shut down the underlying
// connection and release it back to the connection manager.
httpGet.abort();
throw ex;
} finally {
// Closing the input stream will trigger connection release
instream.close();
}
}
return doc;
} finally {
// Release the connection.
closeConnection(httpclient);
}
I notice that your test output shows HTTP/1.1 500 Internal Server Error a couple of lines before the 401 error. I wonder if the root cause could be hiding in there. If I were you I'd try looking for more details about what error happened on the server at that point in the test, to see if it could be responsible for the authentication problem (maybe the failure is in a login controller of some sort, or is causing a session to be cancelled?)
Alternately: it looks like you're using the Apache HttpClient library to do the request, inside the issueRequest method. If you need to include authentication credentials in the request, that would be the code you'd need to change. Here's an example of doing HTTP Basic authentication in HttpClient, if that helps. (And more examples, if that one doesn't.)
(I'd second the observation that this problem probably isn't specific to JUnit. If you need to do more research, I'd suggest learning more about HttpClient, and about what this app expects the browser to send. One possibility: use something like Chrome Dev Tools to peek at your communications with the server when you do this manually, and see if there's anything important that the test isn't doing, or is doing differently.
Once you've figured out how to login, it might make sense to do it in a #Before method in your JUnit test.)
HTTP permission denied has nothing to do with JUnit. You probably need to set your credentials while making the request in the code itself. Show us some code.
Also, unit testing is not really meant to access the internet. Its purpose is for testing small, concise parts of your code which shouldn't rely on any external factors. Integration tests should cover that.
If you can, try to mock your network requests using EasyMock or PowerMock and make them return a resource you would load from your local resources folder (e.g. test/resources).
Related
I am new to Reactive programming paradigm, but recently I have decided to base a simple Http client on Spring WebClient, since the old sync RestTemplate is already under maintenance and might be deprecated in upoming releases.
So first I had a look at Spring documentation and, after that, I've searched the web for examples.
I must say that (only for the time being) I have consciously decided not to go through the Reactor lib documentation, so beyond the Publisher-Subscriber pattern, my knowledge about Mono's and Flux's is scarce. I focused instead on having something working.
My scenario is a simple POST to send a callback to a Server from which the client is only interested in response status code. No body is returned. So I finally came up with this code snippet that works:
private void notifyJobSuccess(final InternalJobData jobData) {
SuccessResult result = new SuccessResult();
result.setJobId(jobData.getJobId());
result.setStatus(Status.SUCCESS);
result.setInstanceId(jobData.getInstanceId());
log.info("Result to send back:" + System.lineSeparator() + "{}", result.toString());
this.webClient.post()
.uri(jobData.getCallbackUrl())
.body(Mono.just(result), ReplaySuccessResult.class)
.retrieve()
.onStatus(s -> s.equals(HttpStatus.OK), resp -> {
log.info("Expected CCDM response received with HttpStatus = {}", HttpStatus.OK);
return Mono.empty();
})
.onStatus(HttpStatus::is4xxClientError, resp -> {
log.error("CCDM response received with unexpected Client Error HttpStatus {}. "
+ "The POST request sent by EDA2 stub did not match CCDM OpenApi spec", resp.statusCode());
return Mono.empty();
})
.onStatus(HttpStatus::is5xxServerError, resp -> {
log.error("CCDM response received with unexpected Server Error HttpStatus {}", resp.statusCode());
return Mono.empty();
}).bodyToMono(Void.class).subscribe(Eda2StubHttpClient::handleResponseFromCcdm);
}
My poor understanding of how the reactive WebClient works starts with the call to subscribe. None of the tens of examples that I checked before coding my client included such a call, but the fact is that before I included that call, the Server was sitting forever waiting for the request.
Then I bumped into the mantra "Nothing happens until you subscribe". Knowing the pattern Plublisher-Subscriber I knew that, but I (wrongly) assumed that the subscription was handled by WebClient API, in any of the exchage, or bodyToMono methods... block() definitely must subscribe, because when you block it, the request gets out at once.
So my first question is: is this call to subscribe() really needed?
Second question is why the method StubHttpClient::handleResponse is never called back. For this, the only explanation that I find is that as the Mono returned is a Mono<Void>, because there is nothing in the response besides the status code, as it is never instantiated, the method is totally dummy... I could even replace it by just .subscribe(). Is this a correct assumption.
Last, is it too much to ask for a complete example of a a method receiving a body in a Mono that is later consumed? All examples I find just focus on getting the request out, but how the Mono or Flux is later consumed is now beyond my understanding... I know that I have to end up checking the Reactor doc sooner better than later, but I would appreciate a bit of help because I am having issues with Exceptions and errors handlin.
Thanks!
Some time has passed since I asked for help here. Now I'd like not to edit but to add an answer to my previous question, so that the answer remains clear and separate from he original question and comments.
So here goes a complete example.
CONTEXT: An application, acting as a client, that requests an Access Token from an OAuth2 Authorization server. The Access Token is requested asynchronously to avoid blocking the appliction's thread while the token request is processed at the other end and the response arrives.
First, this is a class that serves Access Token to its clients (method getAccessToken): if the Access Token is already initialized and it's valid, it returns the value stored; otherwise fetches a new one calling the internal method fetchAccessTokenAsync:
public class Oauth2ClientBroker {
private static final String OAUHT2_SRVR_TOKEN_PATH= "/auth/realms/oam/protocol/openid-connect/token";
private static final String GRANT_TYPE = "client_credentials";
#Qualifier("oAuth2Client")
private final WebClient oAuth2Client;
private final ConfigurationHolder CfgHolder;
#GuardedBy("this")
private String token = null;
#GuardedBy("this")
private Instant tokenExpireTime;
#GuardedBy("this")
private String tokenUrlEndPoint;
public void getAccessToken(final CompletableFuture<String> completableFuture) {
if (!isTokenInitialized() || isTokenExpired()) {
log.trace("Access Token not initialized or has exired: go fetch a new one...");
synchronized (this) {
this.token = null;
}
fetchAccessTokenAsync(completableFuture);
} else {
log.trace("Reusing Access Token (not expired)");
final String token;
synchronized (this) {
token = this.token;
}
completableFuture.complete(token);
}
}
...
}
Next, we will see that fetchAccessTokenAsync does:
private void fetchAccessTokenAsync(final CompletableFuture<String> tokenReceivedInFuture) {
Mono<String> accessTokenResponse = postAccessTokenRequest();
accessTokenResponse.subscribe(tr -> processResponseBodyInFuture(tr, tokenReceivedInFuture));
}
Two things happen here:
The method postAccessTokenRequest() builds a POST request and declares how the reponse will be consumed (when WebFlux makes it available once it is received), by using exchangeToMono:
private Mono postAccessTokenRequest() {
log.trace("Request Access Token for OAuth2 client {}", cfgHolder.getClientId());
final URI uri = URI.create(cfgHolder.getsecServiceHostAndPort().concat(OAUHT2_SRVR_TOKEN_PATH));
} else {
uri = URI.create(tokenUrlEndPoint);
}
}
log.debug("Access Token endpoint OAuth2 Authorization server: {}", uri.toString());
return oAuth2Client.post().uri(uri)
.body(BodyInserters.fromFormData("client_id", cfgHolder.getEdaClientId())
.with("client_secret", cfgHolder.getClientSecret())
.with("scope", cfgHolder.getClientScopes()).with("grant_type", GRANT_TYPE))
.exchangeToMono(resp -> {
if (resp.statusCode().equals(HttpStatus.OK)) {
log.info("Access Token successfully obtained");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().equals(HttpStatus.BAD_REQUEST)) {
log.error("Bad request sent to Authorization Server!");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().equals(HttpStatus.UNAUTHORIZED)) {
log.error("OAuth2 Credentials exchange with Authorization Server failed!");
return resp.bodyToMono(String.class);
} else if (resp.statusCode().is5xxServerError()) {
log.error("Authorization Server could not generate a token due to a server error");
return resp.bodyToMono(String.class);
} else {
log.error("Authorization Server returned an unexpected status code: {}",
resp.statusCode().toString());
return Mono.error(new Exception(
String.format("Authorization Server returned an unexpected status code: %s",
resp.statusCode().toString())));
}
}).onErrorResume(e -> {
log.error(
"Access Token could not be obtained. Process ends here");
return Mono.empty();
});
}
The exchangeToMono method does most of the magic here: tells WebFlux to return a Mono that will asynchronously receive a signal as soon as the response is received, wrapped in a ClientResponse, the parameter resp consumed in the lambda. But it is important to keep in mind that NO request has been sent out yet at this point; we are just passing in the Function that will take the ClientResponse when it arrives and will return a Mono<String> with the part of the body of our interest (the Access Token, as we will see).
Once the POST is built and the Mono returned, then the real thing starts when we subscribe to the Mono<String> returned before. As the Reacive mantra says: nothing happens until you subscribe or, in our case, the request is not actually sent until something attempts to read or wait for the response. There are other ways in WebClient fluent API to implicitly subscribe, but we have chosen here the explicit way of returing the Mono -which implements the reactor Publisher interface- and subscribe to it. Here we blocking the thread no more, releasing CPU for other stuff, probably more useful than just waiting for an answer.
So far, so good: we have sent out the request, released CPU, but where the processing will continue whenever the response comes? The subscribe() method takes as an argument a Consumer parameterized in our case with a String, being nothing less than the body of the response we are waiting for, wrapped in Mono. When the response comes, WebFlux will notify the event to our Mono, which will call the method processResponseBodyInFuture, where we finally receive the response body:
private void processResponseBodyInFuture(final String body, final CompletableFuture<String> tokenReceivedInFuture) {
DocumentContext jsonContext = JsonPath.parse(body);
try {
log.info("Access Token response received: {}", body);
final String aTkn = jsonContext.read("$.access_token");
log.trace("Access Token parsed: {}", aTkn);
final int expiresIn = jsonContext.read("$.expires_in");
synchronized (this) {
this.token = aTkn;
this.tokenExpireTime = Instant.now().plusSeconds(expiresIn);
}
log.trace("Signal Access Token request completion. Processing will continue calling client...");
tokenReceivedInFuture.complete(aTkn);
} catch (PathNotFoundException e) {
try {
log.error(e.getMessage());
log.info(String.format(
"Could not extract Access Token. The response returned corresponds to the error %s: %s",
jsonContext.read("$.error"), jsonContext.read("$.error_description")));
} catch (PathNotFoundException e2) {
log.error(e2.getMessage().concat(" - Unexpected json content received from OAuth2 Server"));
}
}
}
The invocation of this method happens as soon as the Mono is signalled about the reception of the response. So here we try to parse the json content with an Access Token and do something with it... In this case call complete() onto the CompletableFuture passed in by the caller of the initial method getAccessToken, that hopefully will know what to do with it. Our job is done here... Asynchronously!
Summary:
To summarize, these are the basic considerations to have your request sent out and the responses processed when you ise reactive WebClient:
Consider having a method in charge of preparing the request by means of the WebClient fluent API (to set http method, uri, headers and body). Remember: by doing this you are not sending any request yet.
Think on the strategy you will use to obtain the Publisher that will be receive the http client events (response or errors). retreive() is the most straight forward, but it has less power to manipulate the response than exchangeToMono.
Subscribe... or nothing will happen.
Many examples you will find around will cheat you: they claim to use WebClient for asyncrhony, but then they "forget" about subscribing to the Publisher and call block() instead. Well, while this makes things easier and they seem to work (you will see responses received and passed to your application), the thing is that this is not asynchronous anymore: your Mono (or Flux, whatever you use) will be blocking until the response arrives. No good.
Have a separate method (being the Consumer passed in the subscribe() method) where the response body is processed.
I am using web client in a spring application
I am facing memory leak issues while doing the same
I am using below code to get the response body for non 2XX response from service:
return client.get()
.uri(uriString)
.headers(ServiceCommonUtil.getHttpHeaderConsumer(headersMap))
.exchange()
.flatMap(clientResponse -> {
try {
clientResponse.body((clientHttpResponse, context) ->
clientHttpResponse.getBody());
logResponseStatus(clientResponse.statusCode(), serviceName);
return clientResponse.bodyToMono(String.class);
} catch (Exception e) {
return null;
}
})
and later on subscriber uses subscribe/ error block to process this response.
responseMono.subscribe(response -> {
//process response string
},error->{
//process error response
});
My question is, if i use dispose method on responseMono, it takes way long time for processing while without it i face memory leak issues.
Am i doing anything wrong here?
Yes, actually you are not consumming response in case of Exception is thrown.
If you want to use exchange() your responsibillity is to consume response.
See: docs
Take a look on toBodilessEntity()/ releaseBody() in 'ClientResponse` api.
Seems you've gotten a little complicated. Why a try/catch block in the clientResponse lambda? If your logResponseStatus throws a checked exception then handle it there. I suggest starting simpler.
Ex 1:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").exchange().flatMap(clientResponse->clientResponse.bodyToMono(String.class));
stringMono.subscribe(System.out::println);
Ex 2:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").exchange().flatMap(clientResponse->clientResponse.body(BodyExtractors.toMono(String.class)));
stringMono.subscribe(System.out::println);
Ex 3:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").retrieve().bodyToMono(String.class);
stringMono.subscribe(System.out::println);
For logging it is better to use ExchangeFilterFunctions. See How to intercept a request when using SpringBoot WebClient
.
I am setting up a Client using Spring-boot in Java to access a soap endpoint (for testing purpose).What's the best approach to handle Exceptions? I want to handle SOAPFaultClientExceptions...
I have already tried this:
How to Parse SoapFaultClientException in spring-ws
but it didn't work properly, as I couldn't call the getValue() method on detail
try {
JAXBElement res = (JAXBElement) getWebServiceTemplate().marshalSendAndReceive(url, request);
return (GetBankResponseType) res.getValue();
}catch (SoapFaultClientException ex) {
SoapFaultDetail soapFaultDetail = ex.getSoapFault().getFaultDetail(); // <soapFaultDetail> node
// if there is no fault soapFaultDetail ...
if (soapFaultDetail == null) {
throw ex;
}
SoapFaultDetailElement detailElementChild = soapFaultDetail.getDetailEntries().next();
Source detailSource = detailElementChild.getSource();
Object detail = getWebServiceTemplate().getUnmarshaller().unmarshal(detailSource);
JAXBElement source = (JAXBElement) detail;
System.out.println("Text::"+source.getValue());
}//catch other Exceptions...Which ones?
return null;
}
Expected result is a handled Exception, (SOAPFaultClientException) or others... which get Thrown by the webservice when wrong parameters are passed. I don't find any suitable solutions.
Configure ClientInterceptor or FaultMessageResolver to your WebServiceTemplate and do your error handling there.
In our app (Silverlight 5 out-of-browser client hitting a WebApi server) we routinely use an HttpClient for posting/getting/deleting and so on all our entities between client and server. This all works fine most of the time, but recently we have run into an issue when uploading (posting) larger entities (> 30/35mb): we start the streaming process and BEFORE it is finished our Post method on the Web API is hit, receiving a null entity.
We can't understand what is going on, and suspect there must be some timing issue related since it all depends on the size of the upload.
To further explain, our client in summary is doing this:
HttpResponseMessage response = await _client.SendAsync(request);
string jsonResult = await response.Content.ReadAsStringAsync();
... where _client is our HttpClient and request our HttpRequestMessage. In case it is also relevant (I am trying not to flood the question with code :), the content in the request is created like this:
request.Content = new StringContent(JsonConvert.SerializeObject(content), Encoding.UTF8, "application/json");
Well, when we debug this the Post method on our server is hit before the await _client.SendAsync(request) finishes, which sort of "explains" why it is receiving a null entity in such cases (larger entities), where when it works that await call is finished and THEN the Post is hit.
In case if sheds more light into it, due to certain limitations on the HttpClient (regarding access to AllowWriteStreamBuffering), we have also tested an equivalent scenario but using directly an HttpWebRequest... unfortunately, the behavior is exactly the same. This is the relevant extract:
httpRequest.BeginGetRequestStream(RequestStreamCallback, httpRequest);
(where httpRequest is our HttpWebRequest with AllowWriteStreamBuffering = false), and the callback to handle the request stream is as follows:
private void RequestStreamCallback(IAsyncResult ar)
{
var request = ar.AsyncState as System.Net.HttpWebRequest;
if (request != null)
{
var requestStream = request.EndGetRequestStream(ar);
var streamWriter = new StreamWriter(requestStream) {AutoFlush = true};
streamWriter.Write(_jsonContent);
streamWriter.Close();
requestStream.Close(); // Belt and suspenders... shouldn't be needed
// Make async call for response
request.BeginGetResponse(ResponseCallback, request);
}
}
Again, for larger entities when we debug the Post method on the Web API is hit (with a null parameter) BEFORE the streamWriter.Write finalizes and the streamWriter.Close is hit.
We've been reading all over the place and fighting with this for days on now. Any help will be greatly appreciated!
In case somebody runs into this, I finally figured out what was going on.
In essence, the model binding mechanism in the Web API Post method was throwing an exception when de-serializing the JSON, but the exception was somewhat "hidden"... at least if you did not know that much about the inner workings of the Web API, as was my case.
My Post method originally lacked this validation check:
var errors = "";
if (!ModelState.IsValid)
{
foreach (var prop in ModelState.Values)
{
foreach (var modelError in prop.Errors.Where(modelError => modelError != null))
{
if (modelError.Exception != null)
{
errors += "Exception message: " + modelError.Exception.Message + Environment.NewLine;
errors += "Exception strack trace: " + modelError.Exception.StackTrace + Environment.NewLine;
}
else
errors += modelError.ErrorMessage + Environment.NewLine;
errors += " --------------------- " + Environment.NewLine + Environment.NewLine;
}
}
return Request.CreateErrorResponse(HttpStatusCode.NoContent, errors);
}
This is a "sample" check, the main idea being verifying the validity of the ModelState... in our breaking scenarios is wasn't valid because the Web API hadn't been able to bind the entity, and the reason could be found within the Errors properties of the ModelState.Values. The Post was being hit ok, but with a null entity, as mentioned.
By the way, the problem was mainly caused by the fact that we weren't really streaming the content, but using a StringContent which was attempted to be de-serialized in full... but that is another story, we were mainly concerned here with not understanding what was breaking and where.
Hope this helps.
Pretty basic test:
[TestClass]
public class ApiClientTest
{
private RestClient _client;
[TestInitialize()]
public virtual void TestInitialize()
{
_client = new RestClient("http://localhost:24144");
_client.CookieContainer = new System.Net.CookieContainer();
}
[TestMethod]
public void ApiClientTestCRUD()
{
// 1. Log out twice. Verify Unauthorized.
var response = LogOut();
response = LogOut();
Assert.AreEqual(response.StatusCode, HttpStatusCode.Unauthorized);
// Error here:
Result Message: Assert.AreEqual failed. Expected:<0>.
Actual:< Unauthorized >.
I get <0>, which isn't even something that my WebAPI returns.
I think the issue is with my use of RestSharp, because if I debug one time it passes, and then subsequent runs pass. Any clue what's going on?
To be clear - this occurs when I open up my solution and attempt to run the test for the first time. I can fix it by debugging once, watching it pass, and then running without debugging as much as I want. I can reproduce this by closing VS and opening up the solution again - and running the test without debugging first.
Here's the LogOut method in my WebAPI:
[Authorize]
public HttpResponseMessage LogOut()
{
try
{
if (User.Identity.IsAuthenticated)
{
WebSecurity.Logout();
return Request.CreateResponse(HttpStatusCode.OK, "logged out successfully.");
}
return Request.CreateResponse(HttpStatusCode.Conflict, "already done.");
}
catch (Exception e)
{
return Request.CreateResponse(HttpStatusCode.InternalServerError, e);
}
}
UPDATE:
I ended up running the tests with Trace.WriteLine:
// 1. Log out twice. Verify Unauthorized.
Trace.WriteLine("ENTERING FIRST LOGOUT");
var response = LogOut();
Trace.WriteLine("Content: " + response.Content);
Trace.WriteLine("ErrorMessage: " + response.ErrorMessage);
Trace.WriteLine("ResponseStatus: " + response.ResponseStatus);
Trace.WriteLine("StatusCode: " + response.StatusCode);
Trace.WriteLine("StatusDescription: " + response.StatusDescription);
response = LogOut();
Trace.WriteLine("COMPLETED LOGOUTS");
Assert.AreEqual(response.StatusCode, HttpStatusCode.Unauthorized);
And I found the following:
ENTERING FIRST LOGOUT
Content:
ErrorMessage: Unable to connect to the remote server
ResponseStatus: Error
StatusCode: 0
StatusDescription:
COMPLETED LOGOUTS
My solution has a test project with this RestSharp test, and a WebAPI project that's supposed to be accepting these requests. If I debug, the RestClient connects. If not, it times out. Any tips?
When debugging is not possible to solve the problem go to the old fashion way.
Add Trace.WriteLine (or even append text to a C:\temp.txt file).
Write some string before every return in the LogOut method, then try writing some more information (if it's the last return then write the Exception message, if it's the second return write the Identity information.
Hope this helps.
How are you hosting the server? I see this that you're using port 24144. Maybe in debug mode you're running the express IIS Web Server and that's the port, but in non-debug mode it's not?