MissingRequiredPropertyException: Missing required property 'BulkRequest.operations' Elasticsearch - elasticsearch

I am facing the MissingRequiredPropertyException: Missing required property 'BulkRequest.operations' exception. I know what it is. But I don't know how to resolve it.
if (Some Condition) {
// Keep adding the optimised objects to bulk requests so can write it back later in one call.
bulkRequestBuilder.operations(op -> op.index(idx -> idx
.id(esKey)
.document(JSON OBJECT)));
} else {
log.info("I am not building any bulk request.", esKey);
}
Later on, I want to write this bulk request to Elastic search. While writing, I need to check if the bulk request has any operations inside it. So I am doing the below thing.
BulkRequest bulkRequest = bulkRequestBuilder.build();
if (!bulkRequest.operations().isEmpty()) {
BulkResponse bulkResponse = repo.saveToIndexByBulkRequest(bulkRequest);
}
In the above code, .build() is throwing me the MissingRequiredPropertyException. When the bulk Request doesn't have any operations in it, it will throw that exception.
Before building, how can I check if it has any operations inside it?
Let me know if you need more info.

For this particular scenario, you can use exception handling and resolve this like:
try {
BulkRequest bulkRequest = bulkRequestBuilder.build();
if (!bulkRequest.operations().isEmpty()) {
BulkResponse bulkResponse = repo.save(bulkRequest);
if (bulkResponse.errors()) {
throw new CustomException(CustomMessage);
}
}
} catch (MissingRequiredPropertyException exception) {
// either throw exception or continue based on requirement.
log.error("Bulk RequestBuilder has no operations in it.");
}

Related

Simultaneous http post request spring boot

Hi,
I have a list with size of 500k and I have to make a request to a server with hash parameters.
The server accepts JSON Array of 200 objects. so I could send 200 items each time.
But still I need to split the list each time and send that part to server.
I have a method for this which makes the http post request. and I want to use spring boot options (if available) to call the method with different threads and get the response back and merge them into one.
I did it using java CompletableFuture class without any springboot tags. but you Could use #async for your method too. sample code :
var futures = new ArrayList<CompletableFuture<List<Composite>>>();
var executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
for (List<CompositeRecord> records : Lists.partition(recordsList, 200)) {
var future = CompletableFuture.supplyAsync(() -> /* call your method here */, executor);
futures.add(future);
Thread.sleep(2000);
}
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).exceptionally(ex -> null).join(); // avoid throwing an exception in the join() call
var futureMap = futures.stream().collect(Collectors.partitioningBy(CompletableFuture::isCompletedExceptionally));
var compositeWithErrorList = new ArrayList<Composite>();
futureMap.get(false).forEach(l -> {
try {
compositeWithErrorList.addAll(l.get());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
});
after the code is executed you will have a map of done and undone futures.

How to consume spring web client response

I am using web client in a spring application
I am facing memory leak issues while doing the same
I am using below code to get the response body for non 2XX response from service:
return client.get()
.uri(uriString)
.headers(ServiceCommonUtil.getHttpHeaderConsumer(headersMap))
.exchange()
.flatMap(clientResponse -> {
try {
clientResponse.body((clientHttpResponse, context) ->
clientHttpResponse.getBody());
logResponseStatus(clientResponse.statusCode(), serviceName);
return clientResponse.bodyToMono(String.class);
} catch (Exception e) {
return null;
}
})
and later on subscriber uses subscribe/ error block to process this response.
responseMono.subscribe(response -> {
//process response string
},error->{
//process error response
});
My question is, if i use dispose method on responseMono, it takes way long time for processing while without it i face memory leak issues.
Am i doing anything wrong here?
Yes, actually you are not consumming response in case of Exception is thrown.
If you want to use exchange() your responsibillity is to consume response.
See: docs
Take a look on toBodilessEntity()/ releaseBody() in 'ClientResponse` api.
Seems you've gotten a little complicated. Why a try/catch block in the clientResponse lambda? If your logResponseStatus throws a checked exception then handle it there. I suggest starting simpler.
Ex 1:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").exchange().flatMap(clientResponse->clientResponse.bodyToMono(String.class));
stringMono.subscribe(System.out::println);
Ex 2:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").exchange().flatMap(clientResponse->clientResponse.body(BodyExtractors.toMono(String.class)));
stringMono.subscribe(System.out::println);
Ex 3:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").retrieve().bodyToMono(String.class);
stringMono.subscribe(System.out::println);
For logging it is better to use ExchangeFilterFunctions. See How to intercept a request when using SpringBoot WebClient
.

Using BulkInsert with OracleDB

I have some kind of promblem with BulkInsert on my OracleDB. I need to insert couple of thousand objects so I decided to use EF.BulkInsert.Oracle added by Nuget which is extension of EF6.BulkInsert for Oracle.
private IOracleDbContext _context;//Class property
//method body:
EF6.BulkInsert.ProviderFactory.Register<EF6.BulkInsert.Providers.OracleBulkInsertProvider>("BulkInsertProvider");
using (var context = (OracleDbContext)_context)
{
using (var dbContextTransaction = context.Database.BeginTransaction())
{
try
{
//Preparing list of objects
var opt = new EF6.BulkInsert.BulkInsertOptions();
opt.Connection = context.Database.Connection;
await context.BulkInsertAsync<ObjectType>(ObjectList,opt);
await context.SaveChangesAsync();
dbContextTransaction.Commit();
stopwatch.Stop();
}
catch (Exception ex)
{
dbContextTransaction.Rollback();
throw ex;
}
}
}
Without adding opt (BulkInsertOptions object) as parameter of BulkInsert it is trying to connect with SQLServer (which don't exist so I get connection failure). After add this BulkOptions with connection I get exception that connection is already part of transaction :/
Traditional way (_context.TableName.Add() ) of course works but It takes unacceptable amount of time.
Any idea what I did wrong here?
I found better way (BulkInsert still do not cooperate). I used Array Binding
mentioned here
It reduced insert time from ~6 minutes to ~1-1.5 seconds :D (7770 records)

How to get from a BulkItemResponse to corresponding Request

I'm using Elasticsearch's bulk requests in Java and I'm trying to handle the situations where some error happens:
BulkResponse bulkResponse = bulkRequest.get();
if (bulkResponse.hasFailures()) {
for (BulkItemResponse response : bulkResponse) {
if (response.isFailed()
|| response.getResponse().getShardInfo().getFailed() > 0) {
//Find the corresponding request and resend it
}
}
}
After finding the response with error, I want to re-send its request since in my case errors could be momentary and in most of the cases a retry could resolve the problem.
So my question is, how to get from BulkItemResponse to the original Request that led it? Is there any way better than relying on the order of requests and responses?
No you don't have that AFAIK. You need somehow to keep all elements you added to a bulk and in case of error use the id coming from response.getId() to associate that with the original data.
So something similar to:
HashMap<String, Object> myData = new HashMap<>();
BulkRequestBuilder brb = BulkAction.INSTANCE.newRequestBuilder(client);
myData.put(myObject.getId(), myObject);
brb.add(new IndexRequest("index", "type", myObject.getId()).source(myObject.toJson()));
// Other actions
BulkResponse response = client.bulk(brb.request()).get();
response.forEach(bulkItemResponse -> {
if (bulkItemResponse.isFailed()) {
Object objectToSendAgain = myData.get(bulkItemResponse.getId());
// Do what is needed with this object
}
});
myData = null;
I hope this helps.

JUnit needs special permissions?

My builds have been failing due to some of the integration tests I've been running. I'm stuck on why it won't work. Here is an example of the output:
I'm using Maven to first build, then it calls the JUnit tests. I'm seeing this 401 Unauthorized message in every single test, and I believe that's what is causing the builds to fail. In my mind, this means there are some permissions / authentication parameters that need to be set. Where would I go about doing this in JUnit?
Edit
#Test
public void testXmlHorsesNonRunners() throws Exception {
String servletUrl = SERVER + "sd/date/2013-01-13/horses/nonrunners";
Document results = issueRequest(servletUrl, APPLICATION_XML, false);
assertNotNull(results);
// debugDocument(results, "NonRunners");
String count = getXPathStringValue(
"string(count(hrdg:data/hrdg:meeting/hrdg:event/hrdg:nonrunner/hrdg:selection))",
results);
assertEquals("non runners", "45", count);
}
If you can, try to ignore the detail. Effectively, this is making a request. This is a sample of a test that uses the issueRequest method. This method is what makes HTTP requests. (This is a big method, which is why I didn't post it originally. I'll try to make it as readable as possible.
logger.info("Sending request: " + servletUrl);
HttpGet httpGet = null;
// InputStream is = null;
DefaultHttpClient httpclient = null;
try {
httpclient = new DefaultHttpClient();
doFormLogin(httpclient, servletUrl, acceptMime, isIrishUser);
httpGet = new HttpGet(servletUrl);
httpGet.addHeader("accept", acceptMime);
// but more importantly now add the user agent header
setUserAgent(httpGet, acceptMime);
logger.info("executing request" + httpGet.getRequestLine());
// Execute the request
HttpResponse response = httpclient.execute(httpGet);
// Examine the response status
StatusLine statusLine = response.getStatusLine();
logger.info(statusLine);
switch (statusLine.getStatusCode()) {
case 401:
throw new HttpResponseException(statusLine.getStatusCode(),
"Unauthorized");
case 403:
throw new HttpResponseException(statusLine.getStatusCode(),
"Forbidden");
case 404:
throw new HttpResponseException(statusLine.getStatusCode(),
"Not Found");
default:
if (300 < statusLine.getStatusCode()) {
throw new HttpResponseException(statusLine.getStatusCode(),
"Unexpected Error");
}
}
// Get hold of the response entity
HttpEntity entity = response.getEntity();
Document doc = null;
if (entity != null) {
InputStream instream = entity.getContent();
try {
// debugContent(instream);
doc = documentBuilder.parse(instream);
} catch (IOException ex) {
// In case of an IOException the connection will be released
// back to the connection manager automatically
throw ex;
} catch (RuntimeException ex) {
// In case of an unexpected exception you may want to abort
// the HTTP request in order to shut down the underlying
// connection and release it back to the connection manager.
httpGet.abort();
throw ex;
} finally {
// Closing the input stream will trigger connection release
instream.close();
}
}
return doc;
} finally {
// Release the connection.
closeConnection(httpclient);
}
I notice that your test output shows HTTP/1.1 500 Internal Server Error a couple of lines before the 401 error. I wonder if the root cause could be hiding in there. If I were you I'd try looking for more details about what error happened on the server at that point in the test, to see if it could be responsible for the authentication problem (maybe the failure is in a login controller of some sort, or is causing a session to be cancelled?)
Alternately: it looks like you're using the Apache HttpClient library to do the request, inside the issueRequest method. If you need to include authentication credentials in the request, that would be the code you'd need to change. Here's an example of doing HTTP Basic authentication in HttpClient, if that helps. (And more examples, if that one doesn't.)
(I'd second the observation that this problem probably isn't specific to JUnit. If you need to do more research, I'd suggest learning more about HttpClient, and about what this app expects the browser to send. One possibility: use something like Chrome Dev Tools to peek at your communications with the server when you do this manually, and see if there's anything important that the test isn't doing, or is doing differently.
Once you've figured out how to login, it might make sense to do it in a #Before method in your JUnit test.)
HTTP permission denied has nothing to do with JUnit. You probably need to set your credentials while making the request in the code itself. Show us some code.
Also, unit testing is not really meant to access the internet. Its purpose is for testing small, concise parts of your code which shouldn't rely on any external factors. Integration tests should cover that.
If you can, try to mock your network requests using EasyMock or PowerMock and make them return a resource you would load from your local resources folder (e.g. test/resources).

Resources