Aggregations in Java client through JSON query - without AggregationBuilder - elasticsearch

I am able to implement aggregation functionality via JSON query in HTTP based JEST client but not in TCP based Java client.
Through JEST client (HTTP REST based) it is possible to implement aggregation through query String.
JEST sample code:
JestClientFactory factory = new JestClientFactory();
HttpClientConfig httpClientConfig = new HttpClientConfig
.Builder("http://localhost:9201")
.build();
factory.setHttpClientConfig(httpClientConfig);
JestClient client = factory.getObject();
String queryString ="{\"query\":{\"match_all\": {}},\"aggs\":{\"avg1\":{\"avg\":{\"field\":\"age\"} } }}";
Search.Builder searchBuilder = new Search.Builder(queryString)
.addIndex("st1index")
.addType("st1type");
SearchResult response = client.execute(searchBuilder.build());
System.out.println(response.getJsonString());
client.shutdownClient();
Printing response of JEST client shows aggregation results.
Using TCP client in elasticsearch, aggregation is possible through AggregationBuilder.
When I tried to implement JSON query in TCP, it did not return aggregation results.
Is there any reason why TCP do not support aggregation through query string but supports with adding aggregation options?
TCP Java client sample code:
Edited
Removed WrapperQueryBuilder surrounding the queryString.
Settings settings = ImmutableSettings.settingsBuilder()
.put("cluster.name", "javaEscluster")
.put("node.name", "arivu").build();
Client client = new TransportClient(settings)
.addTransportAddress(new InetSocketTransportAddress("localhost", 9303));
String queryString ="{\"match_all\": {},\"aggs\":{\"avg1\":{\"avg\":{\"field\":\"age\"} } }}";
SearchResponse response = client.prepareSearch("st1index").setTypes("st1type").setQuery(queryString).execute().actionGet();
System.out.println("Getresponse-->" +"Index-->"+ response.toString());
//closing node
client.close();
System.out.println("completed");
This code retrieves only search results and empty aggregation result data.
Edited:
Any reference material which explains the reason would be great.

In the main documentation of the WrapperQueryBuilder class, it is stated:
A Query builder which allows building a query given JSON string or binary data provided as input. This is useful when you want to use the Java Builder API but still have JSON query strings at hand that you want to combine with other query builders.
The keyword in here is the word query, i.e. the part named query in the request you send to the ES _search endpoint, i.e.:
{
"sort": {
... <--- whatever sorting definition you have goes here
},
"_source": {
... <--- whatever source definition you have goes here
},
"query": {
... <--- this is the content you can use with WrapperQueryBuilder
},
"aggs": {
... <--- whatever aggs definition you have goes here
}
}
WrapperQueryBuilder will only ever consider whatever you can fit inside that query section, so as you can see that doesn't include aggregations, which are in another top-level section of the request.
So, in the JSON query string you give, only the match_all will be considered, because that's the only valid token that is allowed to appear in the query section, the aggs:{...} part is not.
"{\"match_all\": {},\"aggs\":{\"avg1\":{\"avg\":{\"field\":\"age\"} } }}"
^ ^
| |
this is valid this is NOT valid

Related

Is it possible to set several routing values using Elasticsearch NEST?

I need to query data from several shards. Elasticsearch REST API provides a possibility to send a request with several routing keys:
//https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html#_searching_with_custom_routing
GET my-index-000001/_search?routing=user1,user2
{
"query": {
"match": {
"title": "document"
}
}
}
Is it possible to do the same by NEST client?
Yes, you can pass a comma-separated string to the Routing() method of a search request.

Elasticsearch Jest client add condition to json query

I am using Elasticsearch 6.3 with Jest client 6.3 (Java API)
Search search = new Search.Builder(jsonQueryString)
.addIndex("SOME_INDEX")
.build();
SearchResult result = jestClient.execute(search);
And this is my sample JSON query
{
"query": {
"bool" : {
"filter": {
"match" :{
"someField" : "some value"
}
}
}
}
}
The JSON query string is accepted as a POST request body and then passed to the Jest client. Before I can execute the json query on the Jest client, I need to add conditions to the query for e.g.
{
"query": {
"bool" : {
"filter": {
"match" :{
"someField" : "some value"
}
}
},
"must": {
"match" :{
"systemField" : "pre-defined value"
}
}
}
}
}
Is there an API that allows to parse the JSON query and add conditions to it before it can be executed on Jest client? The JSON query can be any query supported by Query DSL and not necessarily contain bool condition. I need to add a pre-defined condition to the query. I appreciate any help on this. Thanks very much.
There is no out of the box Elasticsearch or Jest API to achieve the above, the workaround I implemented is using Jackson ObjectMapper
// convert the search request body into object node
ObjectNode searchRequestNode = objectMapper.readValue(queryString, ObjectNode.class);
// extract the query
String query = searchRequestNode.get("query").toString();
// wrap the original query and add conditions
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must(QueryBuilders.wrapperQuery(query));
boolQueryBuilder.filter(QueryBuilders.termsQuery("fieldA", listOfValues));
boolQueryBuilder.filter(QueryBuilders.termQuery("fieldB", value));
// convert querybuilder to json query string
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(queryBuilder);
String queryWithFilters = searchSourceBuilder.toString();
// convert json string to object node
ObjectNode queryNode = objectMapper.readValue(queryWithFilters, ObjectNode.class);
// replace original query with the new query containing added conditions
searchRequestNode.set("query", queryNode.get("query"));
String finalSearchRequestWithOwnFilters = searchRequestNode.toString();

ElasticSearch NEST Executing raw Query DSL

I'm trying to create the simplest proxy possible in an API to execute searches on ElasticSearch nodes. The only reason for the proxy to be there is to "hide" the credentials and abstract ES from the API endpoint.
Using Nest.ElasticClient, is there a way to execute a raw string query?
Example query that is valid in vanilla ES:
{
"query": {
"fuzzy": { "title": "potato" }
}
}
In my API, I tried Deserializing the raw string into a SearchRequest, but it fails. I'm assuming it cannot deserialize the field:
var req = m_ElasticClient.Serializer.Deserialize<SearchRequest>(p_RequestBody);
var res = m_ElasticClient.Search<T>(req);
return m_ElasticClient.Serializer.SerializeToString(res);
System.InvalidCastException: Invalid cast from 'System.String' to 'Newtonsoft.Json.Linq.JObject'.
Is there a way to just forward the raw string query to ES and return the string response? I tried using the LowLevel.Search method without luck.
NEST does not support deserializing the short form of "field_name" : "your_value" of the Elasticsearch Query DSL, but it does support the long form "field_name" : { "value" : "your_value" }, so the following works
var client = new ElasticClient();
var json = #"{
""query"": {
""fuzzy"": {
""title"": {
""value"": ""potato""
}
}
}
}";
SearchRequest searchRequest;
using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(json)))
{
searchRequest = client.Serializer.Deserialize<SearchRequest>(stream);
}
As Rob has answered, NEST also supports supplying a raw json string as a query
Yes, you can do this with NEST, check out the following
var searchResponse = client.Search<object>(s => s
.Type("type").Query(q => q.Raw(#"{""match_all"":{}}")));
Hope that helps.

Post Filter Query in Elasticsearch 2.3.3 using Java

I have built a web app on top of elasticsearch (v2.3.3). To filter the query, I am using post filter of elasticsearch. But I came to know that, if I use post filter then the performance benefit of filtering will be lost since I am not using any aggregation or differential filtering. (Reference: https://www.elastic.co/guide/en/elasticsearch/guide/current/_post_filter.html)
This is how my elasticsearch client looks like:
Client client = TransportClient.builder().build().addTransportAddress(
new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"),
9300));
SearchResponse response = client.prepareSearch("index_name")
.setTypes("index_type")
.setQuery(QueryBuilders.simpleQueryStringQuery(query)
.field("newContent").field("T"))
.setPostFilter(QueryBuilders.termQuery(Collection, true))
.setFetchSource(new String[] { "U", "UE", "UD", "T" }, null)
.setVersion(true).addHighlightedField("newContent").setFrom(0)
.setSize(10).execute().actionGet();
I have also read that filtered query is depreciated in elasticsearch 2.x versions. Is there any other way which will help me to apply a filter before the query is executed? I might be missing something obvious. I would appreciate your help.
You simply need to bring the filter present in post filter inside a bool/filter query. Try to do hits instead:
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery()
.must(QueryBuilders.simpleQueryStringQuery(query)
.field("newContent").field("T"))
.filter(QueryBuilders.termQuery(Collection, true));
SearchResponse response = client.prepareSearch("index_name")
.setTypes("index_type")
.setQuery(boolQuery)
.setFetchSource(new String[] { "U", "UE", "UD", "T" }, null)
.setVersion(true).addHighlightedField("newContent").setFrom(0)
.setSize(10).execute().actionGet();

Obtaining string query (JSON) from SearchQuery object

For debugging purposes, I need to know what query spring-data-elasticsearch is sending to the ElasticSearch cluster. I have tried to call the toString method on the SearchQuery object, and doesn't return what I need.
What I am doing in Java (using spring-data-elasticsearch) is:
private FilterBuilder getFilterBuilder(String id) {
return orFilter(
termFilter("yaddayaddayadda.id", id),
termFilter("blahblahblah.id", id)
);
}
SearchQuery sq = NativeSearchQueryBuilder()
.withQuery(new MatchAllQuery())
.withFilter(fb)
.build();
And I expect to return something like this plain query executed in ES cluster REST API is returning:
{
"query": {
"filtered": {
"filter": {
"or": [
{
"term": {
"yaddayaddayadda.id": "9"
}
},
{
"term": {
"blahblahblah.id": "9"
}
}
]
}
}
}
}
Thanks in advance!
One way to achieve this is to log the queries on the ES/server-side into the slowlog file. Open your elasticsearch.yml config file and towards the bottom uncomment/edit the two lines below:
...
index.search.slowlog.threshold.query.info: 1ms
...
index.search.slowlog.threshold.fetch.info: 1ms
...
The advantage of this solution is that whatever client technology you're using to query your ES server (Spring Data, Ruby, Browser, Javascript, etc), you'll be able to dump and debug your queries in a single location.
SearchQuery Interface has a method getQuery() and getFilter() to get the information you need.
System.out.println(searchQuery.getQuery());
System.out.println(searchQuery.getFilter());
Hope this helps.
When using SearchRequest or SearchSourceBuilder, calling .toString() method on their instance will get you actual JSON query:
SearchRequest searchRequest = new SearchRequest("index");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
// building the query
// ...
searchSourceBuilder.query(query);
searchRequest.source(searchSourceBuilder);
System.out.println(searchSourceBuilder.toString()); // prints json query
System.out.println(searchRequest.toString()); // prints json query + other information

Resources