Following function run range queries on the elastic search, when I use this function repeatedly in a multithread environment, The memory increases drastically until the application crashes.
Does anyone have a solution to this problem?
/**
RestHighLevelClient restHighLevelClient = new RestHighLevelClient( //
RestClient.builder(new HttpHost("localhost",9200,"http")));
**/
public List<Map<String, Object>> getAllDocumentsInRange(long startTime, long endTime,
RestHighLevelClient restHighLevelClient) {
try {
QueryBuilder queryBuilder = QueryBuilders//
.rangeQuery("date")//
.gte(startTime)//
.lte(endTime);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(queryBuilder);
searchSourceBuilder.sort(new FieldSortBuilder("date").order(SortOrder.ASC));
searchSourceBuilder.size(10000);
searchSourceBuilder.timeout(new TimeValue(20, TimeUnit.SECONDS));
List<Map<String, Object>> docsMap = new ArrayList<>();
SearchRequest searchRequest = new SearchRequest(new String[] { "mainIndex" }, searchSourceBuilder);
Scroll scroll = new Scroll(TimeValue.timeValueSeconds(30));
searchRequest.scroll(scroll);
RestHighLevelClient restHighLevelClient = new RestHighLevelClient( //
RestClient.builder(new HttpHost("localhost",9200,"http")));
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
String scrollId = searchResponse.getScrollId();
SearchHit[] searchHits = searchResponse.getHits().getHits();
while (searchHits != null && searchHits.length > 0) {
for (SearchHit hit : searchHits) {
Map<String, Object> elasticDocVersion = hit.getSourceAsMap();
docsMap.add(elasticDocVersion);
}
SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
scrollRequest.scroll(scroll);
searchResponse = restHighLevelClient.scroll(scrollRequest, RequestOptions.DEFAULT);
scrollId = searchResponse.getScrollId();
searchHits = searchResponse.getHits().getHits();
}
ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
clearScrollRequest.addScrollId(scrollId);
ClearScrollResponse clearScrollResponse = //
restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
boolean succeeded = clearScrollResponse.isSucceeded();
logger.trace("search scroll clearation:{}", succeeded);
} catch (Exception e) {
logger.error("error in creaing QueryBuilder class: {}", e.getMessage());
}
return new ArrayList<>();
}
this is my memory usage image after running the application
I have tried different solutions, like synchronize the above function but none of them solved the problem!
Not sure what elasticsearch client version you are using, but you may be interested in this elasticsearch rest client memory leak.
Also, I'm not sure which one is intended the restHighLevelClient that is passed in or the one that is created in the method. Only one is valid, but in any case, I would suggest calling restHighLevelClient.close() or using it in a try-with-resources when done with the client.
try (RestHighLevelClient restHighLevelClient =
new RestHighLevelClient(RestClient.builder(new HttpHost("localhost",9200,"http"))) {
...
}
Related
I have SearchRequest object with all the Elasticsearch(ES) query data set. I cannot use RestHighLevel client for my usecase because it requires endpoint need to be passed at the time of instantiation. I gets ES endpoint dynamically based on some condition. One way is to always create new RestHighLevel client which will be inefficient approach. Other way is to create static CloseableHttpClient on service start and make HttpPost request with dynamic endpoint. I wanted to take later approach but don't know how to convert SearchRequest object into json query string.
Any code reference/snippet would be very helpful
private final CloseableHttpClient client;
public GenericElasticSearchResponse search(#Nonnull final SearchRequest searchRequest,
#Nonnull final RoutingConfig route) {
final URIBuilder builder = new URIBuilder()
.setScheme(route.getScheme())
.setHost(route.getESEndpoint())
.setPort(Optional.ofNullable(route.getPort())
.orElse(80))
.setPath("/sessions*/_search");
final URI uri = builder.build();
final ContentType contentType = ContentType.create("application/json", "UTF-8");
final HttpPost httpPost = new HttpPost(uri);
httpPost.setEntity(entity);
final CloseableHttpResponse response = client.execute(httpPost);
final String responseEntity;
try (final Reader reader = new InputStreamReader(response.getEntity().getContent(), Charsets.UTF_8)) {
responseEntity = CharStreams.toString(reader);
}
final SearchResponse searchResponse = objectMapper.readValue(responseEntity, SearchResponse.class);
return new ElasticSearchResponse(searchResponse);
}
I found searchRequest.source().toString() was actually returning json form of SearchRequest. Following is complete code snippet for hitting ES via Apache client
final EndpointConfig endpoint = route.getEndpoint();
final URIBuilder builder = new URIBuilder()
.setScheme(endpoint.getScheme())
.setHost(endpoint.getHost())
.setPort(Optional.ofNullable(endpoint.getPort())
.orElse(HTTPS_PORT))
.setPath(Optional.ofNullable(endpoint.getQueryPath())
.orElse(StringUtils.EMPTY));
final URI uri = builder.build();
final ContentType contentType = ContentType.create("application/json", "UTF-8");
final String queryString = searchRequest.source().toString();
final StringEntity entity = new StringEntity(queryString, contentType);
final HttpPost httpPost = new HttpPost(uri);
httpPost.setEntity(entity);
final CloseableHttpResponse response = sendRequest(httpPost);
final String responseEntity;
try (final Reader reader = new InputStreamReader(response.getEntity().getContent(), Charsets.UTF_8)) {
responseEntity = CharStreams.toString(reader);
}
log.info("ElasticSearchClient response: Code: {}, Entity {}", response.getCode(), responseEntity);
SearchResponse searchResponse = null;
if (Objects.nonNull(responseEntity)) {
searchResponse = parseResponse(responseEntity, searchRequest, response.getCode());
log.info("ElasticSearchClient searchResponse- {} ", searchResponse);
}
return new ElasticSearchResponse(searchResponse);
} catch (final URISyntaxException e) {
throw new IllegalStateException(
String.format("Invalid URI. host: %s", route.getEndpoint()), e);
} catch (final IOException e) {
throw new IllegalStateException("ElasticSearch Request failed.", e);
}
private SearchResponse parseResponse(#Nonnull final String responseEntity,
#Nonnull final SearchRequest searchRequest,
final int responseCode) {
if (responseCode >= 400 || responseCode < 200) {
log.info("ES error response - {} ", responseEntity);
final ESErrorResponse response = GSON.fromJson(responseEntity, ESErrorResponse.class);
throw new IllegalStateException();
}
SearchResponse searchResponse = null;
final NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
final XContentParser parser;
try {
parser = JsonXContent.jsonXContent.createParser(registry,
DeprecationHandler.THROW_UNSUPPORTED_OPERATION, responseEntity);
searchResponse = SearchResponse.fromXContent(parser);
} catch (IOException e) {
throw new IllegalStateException("Error while parsing response ", e);
}
return searchResponse;
}
public static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
final Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
map.put(TopHitsAggregationBuilder.NAME, (p, c) -> ParsedTopHits.fromXContent(p, (String) c));
map.put(StringTerms.NAME, (p, c) -> ParsedStringTerms.fromXContent(p, (String) c));
return map.entrySet().stream()
.map(entry -> new NamedXContentRegistry.Entry(Aggregation.class, new ParseField(entry.getKey()), entry.getValue()))
.collect(Collectors.toList());
}
private CloseableHttpResponse sendRequest(final HttpPost httpPost) throws IOException {
return client.execute(httpPost);
}
I am wondering about an alternative method to TermFilter() in ES 7.5, it's deprecated since ES 2.x.
My current method:
#Override
public boolean deleteAnnotation(String userName, String id) throws Exception {
final IndexAndType annotationIndex = indexConfiguration.getIndex(Annotation);
final SearchResponse searchResponse = esClient.prepareSearch(annotationIndex.getIndex()).setTypes(annotationIndex.getType()).setQuery(
filteredQuery(QueryBuilders.termQuery(AnnotationField.id.field(), id), termFilter(AnnotationField.user.field(), userName))
).execute().actionGet();
if (0 < searchResponse.getTotalShards()) {
final SearchHit hit = searchResponse.getHits().getAt(0);
final DeleteResponse deleteResponse = esClient.prepareDelete(annotationIndex.getIndex(), annotationIndex.getType(), hit.getId()).execute().actionGet();
return deleteResponse.isFound();
}
return false;
}
Also isFound() is deprecated, is there any alternative?
I've updated elasticsearch from 6.4.2 to 6.5.0 and the BulkRequest using the rest high level client api is not working anymore:
final BulkRequest request = new BulkRequest();
myarray().forEach(elem -> {
try {
final XContentBuilder builder = XContentFactory.jsonBuilder();
builder.startObject();
{
builder.timeField("#timestamp", elem.getTimestamp());
builder.field("metric", elem.getMetric().getName());
builder.field("resource", elem.getResource().getName());
builder.field("value", elem.getValue());
builder.field("windowId", elem.getWindowId());
}
builder.endObject();
System.out.println(builder.prettyPrint());
final IndexRequest indexRequest = new IndexRequest("myindex-2018.11.20", "mytype");
indexRequest.source(builder);
request.add(indexRequest);
} catch (IOException e) {
logger.log(Level.SEVERE, "Error forward", e);
}
});
BulkResponse response = client.bulk(request, RequestOptions.DEFAULT);
If I perform a single IndexRequest it works fine:
final XContentBuilder builder = XContentFactory.jsonBuilder();
builder.startObject();
{
builder.timeField("#timestamp", System.currentTimeMillis());
builder.field("metric", "test");
builder.field("resource", "test");
builder.field("value", "-1");
builder.field("windowId", "test");
}
builder.endObject();
final IndexRequest indexRequest = new IndexRequest("myindex-2018.11.20", "mytype");
indexRequest.source(builder);
final IndexResponse response = client.index(indexRequest, RequestOptions.DEFAULT);
Both queries where running well with the version 6.4.2, now with the version 6.5.0 only the second one is working.
I don't receive any errors
I am using Spring Data Elasticsearch and I can successfully run a match query that returns some items. I would like to display the score as well, but it always returns zero.
Here is the code:
String query = "test";
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(matchQuery("_all", query))
.build();
String scrollId = elasticsearchTemplate.scan(searchQuery, 1000, false);
List<GlobalSearchDTO> sampleEntities = new ArrayList<GlobalSearchDTO>();
boolean hasRecords = true;
while (hasRecords){
Page<GlobalSearchDTO> page = elasticsearchTemplate.scroll(scrollId, 5000L , new SearchResultMapper(){
#Override
public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
List<GlobalSearchDTO> out = new ArrayList<>();
GlobalSearchDTO tmp;
ObjectMapper objectMapper = new ObjectMapper();
log.debug("MAX SCORE {}", response.getHits().getMaxScore()); // <-- ZERO
for (SearchHit hit : response.getHits()) {
tmp = new GlobalSearchDTO();
tmp.setId(Long.valueOf(hit.getId()));
tmp.setType(hit.getType());
log.debug("SCORE: {}", hit.getScore()); // <-- ZERO
try {
switch(hit.getIndex()) {
case "document": // map to DTO
tmp.setObj(objectMapper.readValue(hit.getSourceAsString(), Document.class));
break;
case "product": //map to DTO
tmp.setObj(objectMapper.readValue(hit.getSourceAsString(), Product.class));
break;
}
} catch (IOException e) {
e.printStackTrace();
}
out.add(tmp);
}
if (out.size() > 0) {
return new AggregatedPageImpl<T>((List<T>) out);
}
return null;
}
});
I'd like to fetch sonar.timemachine.period1 via wsclient.
Seeing that it doesn't have one, I decided to bake one for myself
private Map<String, String> retrievePeriodProperties(final WsClient wsClient, int requestedPeriod) {
if (requestedPeriod > 0) {
final WsRequest propertiesWsRequestPeriod =
new GetRequest("api/properties/sonar.timemachine.period" + requestedPeriod);
final WsResponse propertiesWsResponsePeriod =
wsClient.wsConnector().call(propertiesWsRequestPeriod);
if (propertiesWsResponsePeriod.isSuccessful()) {
String resp = propertiesWsResponsePeriod.content();
Map<String, String> map = new HashMap<>();
map.put(Integer.toString(requestedPeriod), resp);
return map;
}
}
return new HashMap<>();
}
but it always return an empty Map<>
Any lead where I can go from this direction?
You can use org.sonar.api.config.Settings to fetch properties defined in SonarQube.