i am trying to get statistics on analyzed string field.
i am trying to get AVG length of string field (in this example its title, and title sometimes can be empty/none).
tried:
GET book/_search
{
"facets" : {
"stat1" : {
"statistical" : {
"script" : "_source.title?.length()"
}
}
}
}
and i get an error:
Query Failed [Failed to execute main query]]; nested: NullPointerException; }]",
"status": 500
}
how can i accomplish that?
Any reason why you are using facets and not aggregations? Unless you use an Elasticsearch version that only supports facets, I recommend switching to aggregations. Facets are deprecated in 1.x and removed completely in 2.x.
And an aggregation like this one should work just fine:
GET /book/_search
{
"aggs": {
"stat1": {
"stats": {
"script": "_source.title?.length() ?: 0"
}
}
}
}
Related
I want to retrieve all the JSON objects in Elasticsearch that have a null value for awsKafkaTimestamp. This is the query I have set up:
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "tracer.awsKafkaTimestamp"
}
}
}
}
}
When I curl to my elasticsearch endpoint with the DSL I only get a few values back. I am expecting all (10000+) of them because I know for sure all the awsKafkaTimestamp values are null
This is the response I get when I use Postman. As you can see, there are only 10 JSON objects returned to me:
It's correct behaviour of the elasticsearch. By default, it only returns 10 records and provides information in hits.total field about the total number of documents matching search criteria. To retrieve more data than 10 you should specify size field in your query as shown below (you can read more about it here: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html):
{
"from" : 0, "size" : 10,
"query" : {
"term" : { "user" : "kimchy" }
}
}
By default elasticsearch will give you 10 results, even if it matches to 10212. You can set the size parameter but that is limited to 10000, so your only option is to use the scroll API to get,
Example from elasticsearch site Scroll API
curl -XGET 'localhost:9200/twitter/tweet/_search?scroll=1m' -d '
{
"query": {
"match" : {
"title" : "elasticsearch"
}
}
}
'
I want to access a document in my index directly using its _id field and I want to highlight a word in messageTextfield, for this, I created below query but highlight attribute does not appear in result response.
{
"query":{
"term":{
"_id": "1006382869737"
}
},
"highlight" : {
"tags_schema" : "styled",
"fields" : {
"messageText" : {
"highlight_query":{
"term": {
"messageText":"car"
}
}
}
}
}
}
I'm sure that car is occurred in messageText field for document by Id 1006382869737. so I'm sure highlights must exist in response, but it's not.
if it is important I'm using 2.3.4 version of elasticsearch. and query has been created according to this documentation. I'm not sure what rescore_query is in this documentation, if it is important please tell me how to edit my query, else give me another suggestion.
tnx :)
Additional Information
I also try this below query:
{
"query":{
"bool": {
"must":[
{
"term":{
"_id": "1006382869737"
}
},
{
"term": {
"messageText":"car"
}
}
]
}
},
"highlight" : {
"fields" : {
"messageText" : {}
}
}
}
but it causes no document hits.
I found solution:
My data comes from a server that I haven't access to its code. After many debugging I figure out that in some cases server send the text data in another field named caption.
Unfortunately The server has not a good documentation, that causes this problem.
Finally I find problem and add a more highlight_query for caption field and it's work fine now.
I have field hcc_member_id as of Integer type. I want to perform range query on this field. I tried queries given in the ES documentation, but it does not seem to work. No matter what the query is it always returns same response.
I think I am doing things in a wrong way but not able to identify the problem. Any help is good.
You should use POST instead of GET. Otherwise your Json will be ignored.
Furtermore you should add a "query" field to our json:
(without query you will get something like No parser for element [range]])
{
"query": {
"range": {
"hc_member_id": {
"gte": 1000
}
}
}
}
this is a working (for me) query
//EDIT // IT WORK ONLY IN POST NOT GET
{
"query" : {
"range" : {
"hcc_member_id" : {
"gte" : 1000
}
}
}
}
I have a field like this in my indexed documents
"screen_name : "9GAG"
And this is my query:
{
"query": {
"term": {
"screen_name": "9gag"
}
}
}
Im getting zero hits. But when I replace "9gag" with "9GAG" it works fine. Why is this happening and how can this be fixed?
I'm using Elasticsearch and Nest to create a query for documents within a specific time range as well as doing some filter facets. The query looks like this:
{
"facets": {
"notfound": {
"query": {
"term": {
"statusCode": {
"value": 404
}
}
}
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"time": {
"from": "2014-04-05T05:25:37",
"to": "2014-04-07T05:25:37"
}
}
}
]
}
}
}
In the specific case, the total hits of the search is 21 documents, which fits the documents within that time range in Elasticsearch. But the "notfound" facet returns 38, which fits the total number of ErrorDocuments with a StatusCode value of 404.
As I understand the documentation, facets collects data from withing the search. In this case, the "notfound" facet should never be able to return a count higher that 21.
What am I doing wrong here?
There's a distinct difference between filter/query/filtered_query/facet filter which is good to know.
Top level filter
{
filter: {}
}
This acts as a post-filter, meaning it will filter the results after the query phase has ended. Since facets are part of the query phase filters do not influence the documents that are facetted over. Filters do not alter score and are therefor very cacheable.
Top level query
{
query: {}
}
Queries influence the score of a document and are therefor less cacheable than filters. Queries run in the query phase and thus also influence the documents that are facetted over.
Filtered query
{
query: {
filtered: {
filter: {}
query: {}
}
}
}
This allows you to run filters in the query phase taking advantage of their better cacheability and have them influence the documents that are facetted over.
Facet filter
"facets" : {
"<FACET NAME>" : {
"<FACET TYPE>" : {
...
},
"facet_filter" : {
"term" : { "user" : "kimchy"}
}
}
}
this allows you to apply a filter to the documents that the facet is run over. Remember that the it'll be a combination of the queryphase/facetfilter unless you also specify global:true on the facet as well.
Query Facet/Filter Facet
{
"facets" : {
"wow_facet" : {
"query" : {
"term" : { "tag" : "wow" }
}
}
}
}
Which is the one that #thomasardal is using in this case which is perfectly fine, it's a facet type which returns a single value: the query hit count.
The fact that your Query Facet returns 38 and not 21 is because you use a filter for your time range.
You can fix this by either doing the filter in a filtered_query in the query phase or apply a facet filter(not a filter_facet) to your query_facet although because filters are cached better you better use facet filter inside you filter facet.
Confusingly Filter Facets are specified using .FacetFilter() on the search object. I will change this in 1.0 to avoid future confusion.
Sadly: .FacetFilter() and .FacetQuery() in NEST do not allow you to specify a facet filter like you can with other facets:
var results = typedClient.Search<object>(s => s
.FacetTerm(ft=>ft
.OnField("myfield")
.FacetFilter(f=>f.Term("filter_facet_on_this_field", "value"))
)
);
You issue here is that you are performing a Filter Facet and not a normal facet on your query (which will follow the restrictions applied via the query filter). In the JSON, the issue is because of the "query" between the facet name "notfound" and the "terms" entry. This is telling Elasticsearch to run this as a separate query and facet on the results of this separate query and not your main query with the date range filter. So your JSON should look like the following:
{
"facets": {
"notfound": {
"term": {
"statusCode": {
"value": 404
}
}
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"time": {
"from": "2014-04-05T05:25:37",
"to": "2014-04-07T05:25:37"
}
}
}
]
}
}
}
Since I see you have this tagged with NEST as well, in your call using NEST, you are probably using FacetFilter on your search request, switch this to just Facet to get the desired result.