I'm using Elasticsearch's bulk requests in Java and I'm trying to handle the situations where some error happens:
BulkResponse bulkResponse = bulkRequest.get();
if (bulkResponse.hasFailures()) {
for (BulkItemResponse response : bulkResponse) {
if (response.isFailed()
|| response.getResponse().getShardInfo().getFailed() > 0) {
//Find the corresponding request and resend it
}
}
}
After finding the response with error, I want to re-send its request since in my case errors could be momentary and in most of the cases a retry could resolve the problem.
So my question is, how to get from BulkItemResponse to the original Request that led it? Is there any way better than relying on the order of requests and responses?
No you don't have that AFAIK. You need somehow to keep all elements you added to a bulk and in case of error use the id coming from response.getId() to associate that with the original data.
So something similar to:
HashMap<String, Object> myData = new HashMap<>();
BulkRequestBuilder brb = BulkAction.INSTANCE.newRequestBuilder(client);
myData.put(myObject.getId(), myObject);
brb.add(new IndexRequest("index", "type", myObject.getId()).source(myObject.toJson()));
// Other actions
BulkResponse response = client.bulk(brb.request()).get();
response.forEach(bulkItemResponse -> {
if (bulkItemResponse.isFailed()) {
Object objectToSendAgain = myData.get(bulkItemResponse.getId());
// Do what is needed with this object
}
});
myData = null;
I hope this helps.
Related
I am from the old world that think webapi should return a strong typed object and let json serialization return data.
However, recently we got this requirement:
We have a sql table which has more than 500 columns.
The customer always want to return all the columns.
Our c# code does nothing other than reading the SqlDatareader, convert the reader to a c# object and return result.
In this case, wouldn't better to do this (example copied from another stackoverflow post). Basically just return a stream? Does returning a stream still considered to be anti-pattern?
public HttpResponseMessage SomeMethod(List<string> someIds)
{
HttpResponseMessage resp = new HttpResponseMessage();
resp.Content = new PushStreamContent(async (responseStream, content, context) =>
{
await CopyBinaryValueToResponseStream(responseStream, someIds);
});
return resp;
}
private static async Task CopyBinaryValueToResponseStream(Stream responseStream, int imageId)
{
// PushStreamContent requires the responseStream to be closed
// for signaling it that you have finished writing the response.
using (responseStream)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
await connection.OpenAsync();
using (SqlCommand command = new SqlCommand("SELECT 500 columns FROM [StupidWideTable] WHERE ....", connection))
{
.....
using (SqlDataReader reader = await command.ExecuteReaderAsync(CommandBehavior.SequentialAccess))
{
if (await reader.ReadAsync())
{
if (!(await reader.IsDBNullAsync(0)))
{
using (Stream data = reader.GetStream(0))
{
// Asynchronously copy the stream from the server to the response stream
await data.CopyToAsync(responseStream);
}
}
}
}
}
}
}// close response stream
}
Does returning a stream still considered to be anti-pattern?
Well, that depends on what you want to do. For example, if you want to return a 500 if the SQL server fails partway through, then you shouldn't return a stream.
Streaming results works fine on ASP.NET, but it's important to note that all headers (including the response status code) are sent before the stream begins. So you'll send an immediate 200 when you start streaming the result, and if there's an error later on there's no way to go back in time and change that to a 500. Or add some kind of Continue header.
In other words, yes it's supported; but you lose all the benefits of model binding, content negotiation, exception handlers, etc., because you're bypassing that whole pipeline.
I am facing the MissingRequiredPropertyException: Missing required property 'BulkRequest.operations' exception. I know what it is. But I don't know how to resolve it.
if (Some Condition) {
// Keep adding the optimised objects to bulk requests so can write it back later in one call.
bulkRequestBuilder.operations(op -> op.index(idx -> idx
.id(esKey)
.document(JSON OBJECT)));
} else {
log.info("I am not building any bulk request.", esKey);
}
Later on, I want to write this bulk request to Elastic search. While writing, I need to check if the bulk request has any operations inside it. So I am doing the below thing.
BulkRequest bulkRequest = bulkRequestBuilder.build();
if (!bulkRequest.operations().isEmpty()) {
BulkResponse bulkResponse = repo.saveToIndexByBulkRequest(bulkRequest);
}
In the above code, .build() is throwing me the MissingRequiredPropertyException. When the bulk Request doesn't have any operations in it, it will throw that exception.
Before building, how can I check if it has any operations inside it?
Let me know if you need more info.
For this particular scenario, you can use exception handling and resolve this like:
try {
BulkRequest bulkRequest = bulkRequestBuilder.build();
if (!bulkRequest.operations().isEmpty()) {
BulkResponse bulkResponse = repo.save(bulkRequest);
if (bulkResponse.errors()) {
throw new CustomException(CustomMessage);
}
}
} catch (MissingRequiredPropertyException exception) {
// either throw exception or continue based on requirement.
log.error("Bulk RequestBuilder has no operations in it.");
}
I'm trying to do repeat payments with Form Integration in Sagepay (now Opayo).
From an earlier problem posted on here, I get that the securitykey is needed but is not returned in the Form call, so an additional call needs to be made to the getTransactionDetails command.
I have the securitykey and can now make a call to https://test.sagepay.com/gateway/service/repeat.vsp to initiate the repeat payment. However, the documentation does not say where the response to that call goes. I assume therefore, that it would go to the NotificationURL that is set up with a payment when using the Server or Direct integrations. Since I'm using Form, this is not set.
The question is, is there any way of capturing the response to the https://test.sagepay.com/gateway/service/repeat.vsp call if the initial payment was created using Form integration?
I suppose the second question is, has anybody successfully made repeat payments work with Sagepay Form integration?
Not sure if this helps you and we didn't do repeat payments; but we are looking at releasing deferred payments and I think it is a similar approach.
How do you make the call to 'https://test.sagepay.com/gateway/service/repeat.vsp'?
Could you use a 'HttpWebRequest' to make the call then capture the direct response in 'HttpWebResponse'?
EG:
private static void DeferredSharedApiCall(Dictionary<string, string> data, string type, string url)
{
string postData = string.Join("&", data.Select(x => $"{x.Key}={HttpUtility.UrlEncode(x.Value)}"));
HttpWebRequest request = WebRequest.CreateHttp(url);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
using (TextWriter tw = new StreamWriter(request.GetRequestStream()))
{
tw.Write(postData);
}
HttpWebResponse response = null;
try
{
response = request.GetResponse() as HttpWebResponse;
}
catch (WebException ex)
{
//log.Error($"{type} Error, data: {postData}", ex);
}
catch (Exception ex)
{
//log.Error($"{type} Error, data: {postData}", ex);
}
if (response != null)
{
using (TextReader tr = new StreamReader(response.GetResponseStream()))
{
string result = tr.ReadToEnd();
//log.Info($"{type} Response: {Environment.NewLine}{result}");
}
}
}
This question is related to the excellent answer by Youssef. I love OnSendingHeaders callback. I can now add the response headers without worrying about switching streams. Anyways, here is my question. Is it possible to read the response body inside the callback, like so.
public override async Task Invoke(OwinRequest request, OwinResponse response)
{
request.OnSendingHeaders(state =>
{
var resp = (OwinResponse)state;
// Here, I want to convert resp, which is OwinResponse
// to HttpResponseMessage so that when Content.ReadAsStringAsync
// is called off this HttpResponseMessage object, I want the
// response body as string.
var responseMessage = new HttpResponseMessage();
responseMessage.Content = new StreamContent(resp.Body);
// Here I would like to call
// responseMessage.Content.ReadAsStringAsync()
}, response);
await Next.Invoke(request, response);
}
The methods I want to call from the callback are part of classes that depend on HttpResponseMessage and do not want to change them.
If I set the response body to memory stream before the pipeline processing starts (as was initially suggested by Youssef in the linked answer), I'm able to get this working. Is there a better way to do this here in the callback instead of that?
EDIT:
Is this okay?
public override async Task Invoke(OwinRequest request, OwinResponse response)
{
// Do something with request
Stream originalStream = response.Body;
var buffer = new MemoryStream();
response.Body = buffer;
await Next.Invoke(request, response);
var responseMessage = new HttpResponseMessage();
response.Body.Seek(0, SeekOrigin.Begin);
responseMessage.Content = new StreamContent(response.Body);
// Pass responseMessage to other classes for the
// response body to be read like this
// responseMessage.Content.ReadAsStringAsyn()
// Add more response headers
if (buffer != null && buffer.Length > 0)
{
buffer.Seek(0, SeekOrigin.Begin);
await buffer.CopyToAsync(originalStream);
}
}
What do you want to do with the response body?
This callback is invoked on first write, so it's too late to replace the stream. You also can't read from the response stream as there is nothing stored in it normally. This is normally a write-only stream that goes out to the network.
Replacing the response stream earlier is the correct approach here.
I created some web apis and when an error happens the api returns HttpResponseMessage that is created with CreateErrorResponse message. Something like this:
return Request.CreateErrorResponse(
HttpStatusCode.NotFound, "Failed to find customer.");
My problem is that I cannot figure out how to retrieve the message (in this case "Failed to find customer.") in consumer application.
Here's a sample of the consumer:
private static void GetCustomer()
{
var client = new HttpClient();
client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
string data =
"{\"LastName\": \"Test\", \"FirstName\": \"Test\"";
var content = new StringContent(data, Encoding.UTF8, "application/json");
var httpResponseMessage =
client.PostAsync(
new Uri("http://localhost:55202/api/Customer/Find"),
content).Result;
if (httpResponseMessage.IsSuccessStatusCode)
{
var cust = httpResponseMessage.Content.
ReadAsAsync<IEnumerable<CustomerMobil>>().Result;
}
}
Any help is greatly appreciated.
Make sure you set the accept and or content type appropriately (possible source of 500 errors on parsing the request content):
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
content.Headers.ContentType = new MediaTypeWithQualityHeaderValue("application/json");
Then you could just do:
var errorMessage = response.Content.ReadAsStringAsync().Result;
That's all on the client of course. WebApi should handle the formatting of the content appropriately based on the accept and/or content type. Curious, you might also be able to throw new HttpResponseException("Failed to find customer.", HttpStatusCode.NotFound);
One way to get the message is to do:
((ObjectContent)httpResponseMessage.Content).Value
This will give you a dictionary that contains also the Message.
UPDATE
Refer to the official page:
http://msdn.microsoft.com/en-us/library/jj127065(v=vs.108).aspx
You have to vary the way you're reading the successful response and the error response as one is obviously in your case StreamContent, and the other should be ObjectContent.
UPDATE 2
Have you tried doing it this way ?
if (httpResponseMessage.IsSuccessStatusCode)
{
var cust = httpResponseMessage.Content.
ReadAsAsync<IEnumerable<CustomerMobil>>().Result;
}
else
{
var content = httpResponseMessage.Content as ObjectContent;
if (content != null)
{
// do something with the content
var error = content.Value;
}
else
{
Console.WriteLine("content was of type ", (httpResponseMessage.Content).GetType());
}
}
FINAL UPDATE (hopefully...)
OK, now I understand it - just try doing this instead:
httpResponseMessage.Content.ReadAsAsync<HttpError>().Result;
This is an option to get the message from the error response that avoids making an ...Async().Result() type of call.
((HttpError)((ObjectContent<HttpError>)response.Content).Value).Message
You should make sure that response.Content is of type ObjectContent<HttpError> first though.
It should be in HttpResponseMessage.ReasonPhrase. If that sounds like a bit of a strange name, it's just because that is the way it is named in the HTTP specification http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html
OK this is hilarious, but using QuickWatch I came up with this elegant solution:
(new System.Collections.Generic.Mscorlib_DictionaryDebugView(((System.Web.Http.HttpError)(((System.Net.Http.ObjectContent)(httpResponseMessage.Content)).Value)))).Items[0].Value
That is super readable!