While exploring aggregation in elasticsearch I found out that aggregation functionality can be implemented via JSON query in HTTP based JEST client but not in TCP based Java client.
I am using Jest Client and implemented aggregation through query String that works fine. But I feel it gets quiet cumbersome as the filters increase.
I want to know if there is a way to implement aggregations other than using JSON query in JEST client (Something like aggregation builders in TCP client) and how do we implement it?
This was my solution to implement aggregations, don't forget to add the query first.
org.elasticsearch.search.builder.SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
TermsAggregationBuilder termsAggregationBuilder =
AggregationBuilders.terms(fieldName).field("keyword");
termsAggregationBuilder.minDocCount(minDocCount);
termsAggregationBuilder.size(size);
searchRequestBuilder.aggregation(termsAggregationBuilder);
Search search = new io.searchbox.core.Search.Builder(searchSourceBuilder.toString())
.addIndex(indexNameBuilder.getIndexName(searchContext.getCompanyId()))
.build();
SearchResult result = jestConnectionManager.getClient().execute(search);
Here's what I landed on:
SearchSourceBuilder searchBuilder = SearchSourceBuilder
.searchSource()
.size(0)
.query(QueryBuilders.termQuery("field", "my value"))
.aggregation(
AggregationBuilders
.sum("number_field_sum")
.field("number_field")
);
Search search = new Search.Builder(searchBuilder.toString())
.addIndex("my-index")
.build();
SearchResult result = jestClient.execute(search);
This is largely the same as what you came up with, but I put the query into the SearchSourceBuilder and I simplified the example a bit.
Related
Steps to write elastic search count api using spring boot.?
how to use countRequest to get count of documents...??
need code snippets.
Using new Java API Client:
var countRequestResponse = client.count(CountRequest.of(c -> c.index("idx_name")));
System.out.println(countRequestResponse.count());
Using Java High Client Library
var countRequest = new CountRequest();
countRequest.indices("idx_name");
var response = getClient().count(countRequest, RequestOptions.DEFAULT);
System.out.println(response.getCount());
I have elastic query built with ruflin/Elastica, with global aggregation. Is it possible to somehow add some filters to it, separate from my main query.
It looks like so:
$query = new Query($boolQuery);
$categoryAggregation = new Terms('category_ids');
$categoryAggregation->setField('category_ids');
$categoryAggregation->setSize(0);
$manufacturerAggregation = new Terms('manufacturer_ids');
$manufacturerAggregation->setField('manufacturer_id');
$manufacturerAggregation->setSize(0);
$globalAggregation = new GlobalAggregation('global');
$globalAggregation->addAggregation($categoryAggregation);
$globalAggregation->addAggregation($manufacturerAggregation);
$query->addAggregation($globalAggregation);
I would like to add some custom filters to manufacturer_ids and category_ids aggregations. At the moment they are aggregated from all documents. Is there any way to do it via Elastica API, so that it applies some filtering to it?
I found it myself through trial and error, it goes as following:
$categoryAggregation = new Terms('category_ids');
$categoryAggregation->setField('category_ids');
$categoryAggregation->setSize(0);
$filter = new Filter('category_ids', $merchantIdQuery);
$filter->addAggregation($categoryAggregation);
$globalAggregation = new GlobalAggregation('global');
$globalAggregation->addAggregation($filter);
I am using Elasticsearch SDK 2.3 and also Jest 2.0.0 to create a client. I am trying to get average aggregation (and other aggregations) implemented which will retrieve results from a certain time period.
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//Queries builder
searchSourceBuilder.query(QueryBuilders.matchAllQuery());
GregorianCalendar gC = new GregorianCalendar();
gC.set(2016, Calendar.OCTOBER, 18, 0, 0, 0);
long from = gC.getTimeInMillis();
gC.add(Calendar.MINUTE, 15);
long to = gC.getTimeInMillis();
searchSourceBuilder.postFilter(QueryBuilders.rangeQuery("timestamp").from(from).to(to));
JestClient client = getJestClient();
AvgBuilder aggregation2 = AggregationBuilders
.avg(AvgAggregation.TYPE)
.field("backend_processing_time");
AggregationBuilder ag = AggregationBuilders.dateRange(aggregation2.getName())
.addRange("timestamp", from, to).
subAggregation(aggregation2);
searchSourceBuilder.aggregation(ag);
String query = searchSourceBuilder.toString();
Search search = new Search.Builder(query)
.addIndex(INDEX)
.addType(TYPE)
// .addSort(new Sort("code"))
.setParameter(Parameters.SIZE, 5)
// .setParameter(Parameters.SCROLL, "5m")
.build();
SearchResult result = client.execute(search);
System.out.println("ES Response with aggregation:\n" + result.getJsonString());
And the error that I am getting is the following:
{"error":{"root_cause":[{"type":"aggregation_execution_exception","reason":"could not find the appropriate value context to perform aggregation [avg]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"elbaccesslogs_2016_10","node":"5ttEmYcTTsie-z23OpKY0A","reason":{"type":"aggregation_execution_exception","reason":"could not find the appropriate value context to perform aggregation [avg]"}}]},"status":500}
Looks like the reason is 'could not find the appropriate value context to perform aggregation [avg]' ... which I don't know really what is going on.
Can anyone suggest anything please? Or if need more info on this before responding, please let me know.
Thanks,
Khurram
I have found the solution myself so I am explaining below and closing the issue which is as follows:
Basically the code
searchSourceBuilder.postFilter(QueryBuilders.rangeQuery("timestamp").from(from).to(to));
doesn't work with aggregations. We need to remove 'postFilter' code and the following code:
AggregationBuilder ag = AggregationBuilders.dateRange(aggregation2.getName())
.addRange("timestamp", from, to).
subAggregation(aggregation2);
And change the following code too:
searchSourceBuilder.query(QueryBuilders.matchAllQuery());
to
searchSourceBuilder.query(
QueryBuilders.boolQuery().
must(QueryBuilders.matchAllQuery()).
filter(QueryBuilders.rangeQuery("timestamp").from(from).to(to))
);
So here is the entire code again:
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
GregorianCalendar gC = new GregorianCalendar();
gC.set(2016, Calendar.OCTOBER, 18, 0, 0, 0);
long from = gC.getTimeInMillis();
gC.add(Calendar.MINUTE, 15);
long to = gC.getTimeInMillis();
searchSourceBuilder.query(
QueryBuilders.boolQuery().
must(QueryBuilders.matchAllQuery()).
filter(QueryBuilders.rangeQuery("timestamp").from(from).to(to))
);
JestClient client = getJestClient(); // just a private method creating a client the way Jest client is created
AvgBuilder avg_agg = AggregationBuilders
.avg(AvgAggregation.TYPE)
.field("backend_processing_time");
searchSourceBuilder.aggregation(avg_agg);
String query = searchSourceBuilder.toString();
Search search = new Search.Builder(query)
.addIndex(INDEX)
.addType(TYPE)
.setParameter(Parameters.SIZE, 10)
.build();
SearchResult result = client.execute(search);
System.out.println("ES Response with aggregation:\n" + result.getJsonString());
is there a way I can create ISearchResponse to simulate return by elastic search from JSON string? I need to write unittests for my API. The API building the query has histogram, date filters etc and hence response will be as per that and I want to simulate that.
Thanks
You can deserialize json into an instance of ISearchResponse<T> with
ISearchResponse<T> searchResponse = null;
using (var stream = File.OpenRead("path-to-json.json"))
{
searchResponse = client.Serializer.Deserialize<SearchResponse<T>>(stream);
}
If this is stub data, I'd be more inclined to have a stub implementation of ISearchResponse<T> in code as opposed to deserializing json to create an instance; maybe a little easier to maintain.
I'm doing a small research about a client for elastic search in .net and I found that NEST is one of the most supported solutions for this matter.
I was looking at Nest's docummentation and I couldn´t find a way to output a raw json from a query and avoid the serialization into an object, because I'm using angularJs in the front end I don´t want to overload the process of sending the information to the client with some unnecessary steps.
......and also I'd like to know how can I overrdide the serialization process?
I found that NEST uses Json.NET which I would like to change for the servicestack json serielizer.
thanks!
Hi Pedro you can do this with NEST
var searchDescriptor = new SearchDescriptor<ElasticSearchProject>()
.Query(q=>q.MatchAll());
var request = this._client.Serializer.Serialize(searchDescriptor);
ConnectionStatus result = this._client.Raw.SearchPost(request);
Assert.NotNull(result);
Assert.True(result.Success);
Assert.IsNotEmpty(result.Result);
This allows you to strongly type your queries, but return the string .Result which is the raw response from elasticsearch as string to your
request can be an object or the string so if you are OK with the internal json serialize just pass searchDescriptor directly
Use RequestResponseSerializer instead of Serializer.
var searchDescriptor = ...;
...
byte[] b = new byte[60000];
using (MemoryStream ms = new MemoryStream(b))
{
this._client.RequestResponseSerializer.Serialize(searchDescriptor , ms);
}
var rawJson = System.Text.Encoding.Default.GetString(b);