I am new on Elastic Search. I really need the result about calculating the difference of two set.
Here is the mapping of a index:
{
"mappings": {
"properties": {
"Date": { "type": "date", "format": "yyyyMMdd"},
"areaID": { "type": "keyword" },
"deviceID": { "type": "keyword" }
}
}
}
The date range is from October to November.
I want to get a response for counting November's all new distinct 'deviceID' which grouped by 'areaID'.
I have no idea about how to implement it in ES syntax. Any ES master could give me some hints?
THANKS SO MUCH!
You can using aggs of elasticseach to group by areaID.
This is example with kibana
GET your_index/_search
{
"size": 1000000,
"query": {
"range": {
"Date": {
"gte": "2020-10-01",
"lte": "2020-11-31
}
}
}
},
"aggs": {
"area_id": {
"terms": {
"field": "areaID.keyword"
},
"aggs": {
"Date": {
"date_range": {
"field": "Date",
"ranges": [
{
"from": "2020-11-01",
"to": "2020-11-31"
}
]
},
"aggs": {
"device_id": {
"terms": {
"field": "deviceID.keyword",
}
}
}
}
}
}
}
}
Related
I'm trying to query an exact date in elasticsearch and when I submit, I received date that is not part of the query. I have this code
{
"query": {
"bool": {
"must": [
{
"match": {
"device.id": 1374
}
},
{
"match": {
"readingType.id": 1048
}
},
{
"match": {
"created": "2022-11-16"
}
}
]
}
},
"size": 1000,
"from": 0,
"sort": [
{
"created": {
"order": "desc"
}
}
]
}
so I try search date 2022-11-16 and I have this data
May I know why the elasticsearch add 2022-11-17 to the list? I already checked the documentation but I failed to see the issue of why I have this list. I all I want are the data for 2022-11-16
I have this setup for my created column.
{
"list": {
"aliases": {},
"mappings": {
"reading": {
"_meta": {
"model": "App\\Entity\\List"
},
"dynamic_date_formats": [],
"properties": {
"created": {
"type": "date"
},
"device": {
"properties": {
"id": {
"type": "long"
}
}
},
To query an exact date you need to do it like this:
{
"range": {
"created": {
"gte": "2022-11-16",
"lte": "2022-11-16"
}
}
}
I am trying to fetch all the documents within a radius of a particular location (lat,long).
Here's the mapping with location as the geo_point:
{
"mappings": {
"_doc": {
"properties": {
"color": {
"type": "long"
},
"createdTime": {
"type": "date"
},
"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
}
}
}
}
}
And here's my query
{
"aggregations": {
"weather_agg": {
"geo_distance": {
"field": "location",
"origin": "41.12,-100.77",
"unit": "km",
"distance_type": "plane",
"ranges": [
{
"from": 0,
"to": 100
}
]
},
"aggregations": {
"timerange": {
"filter": {
"range": {
"createdTime": {
"gte": "now-40h",
"lte": "now"
}
}
},
"aggregations": {
"weather_stats": {
"stats": {
"field": "color"
}
}
}
}
}
}
}
}
I am getting 0 hits for this. My question is whether there's something wrong with the mapping or the query ? We recently migrated to a newer cloud version and there's a possibility that something broke because of that.
Instead of mapping lat and long as float you should geo-point mapping
I have documents in the following style in my index:
{
"docType": {
"valuesOverTime": [
{
"begin": 1488442858570,
"end": 1488442860570,
"values": [
{
"name": "level",
"segCount": 4
}
]
},
{
"begin": 1488442860571,
"end": 1488442890592,
"mcdn": [
{
"name": "level",
"segCount": 10
}
]
},
{
"begin": 1488442890593,
"end": 1488442890600,
"mcdn": [
{
"name": "level",
"segCount": 7
}
]
}
]
}
}
and want to query the sum of the docType.valuesOverTime.values.segCount in a certain time range, like the following range
{
"range": {
"docType.valuesOverTime.begin": {
"gte": 1488442858570,
"lte": 1488442860571
}
}
},
{
"range": {
"docType.valuesOverTime.end": {
"gte": 1488442860570,
"lte": 1488442890592
}
}
}
should get me the sum of the first two entries: 14.
However I am absolutely stuck getting the query right! I always get the sum of all the entries in docType.valueOverTime.values.segCount being 21 in this case.
I tried the following queries and some variations on them which where of course all wrong:
{
"size": 0,
"aggs": {
"myfilter": {
"filter": {
"bool": {
"must": [
{
"range": {
"docType.valuesOverTime.begin": {
"gte": 1488442858570,
"lte": 1488442860571
}
}
},
{
"range": {
"docType.valuesOverTime.end": {
"gte": 1488442860570,
"lte": 1488442890592
}
}
}
]
}
},
"aggs": {
"summe": {
"sum": {
"field": "docType.valuesOverTime.values.segCount"
}
}
}
}
}
}
and
{
"_source": "docType.valuesOverTime.values",
"query": {
"constant_score" : {
"filter" : {
"bool": {
"must": [
{
"range": {
"docType.valuesOverTime.begin": {
"gte": 1488442858570,
"lte": 1488442860571
}
}
},
{
"range": {
"docType.valuesOverTime.end": {
"gte": 1488442860570,
"lte": 1488442890592
}
}
}
]
}
}
}
},
"aggs": {
"summe": {
"sum": {
"field": "docType.valuesOverTime.values.segCount"
}
}
}
}
Can someone please tell me, what I got wrong? And how to do it right!
Working on es 5.2.2 with the following mapping
"valuesOverTime": {
"properties": {
"begin": {
"type": "long"
},
"end": {
"type": "long"
},
"values": {
"properties": {
"name": {
"type": "keyword"
},
"segCount": {
"type": "long"
}
}
}
}
}
I solved it myself:
first, everything in valuesOverTime needs to be nested, so that the mapping looks like
"valuesOverTime": {
"type": nested"
"properties": {
"begin": {
"type": "long"
},
"end": {
"type": "long"
},
"values": {
"properties": {
"name": {
"type": "keyword"
},
"segCount": {
"type": "long"
}
}
}
}
}
Then I can query as follows:
{
"size": 0,
"aggs": {
"nestedAcq": {
"nested": {"path": "docType.valuesOverTime"},
"aggs": {
"rangeAcq": {
"range": {
"field": "ocType.valuesOverTime.begin",
"ranges": [
{
"from": 1488442858570,
"to": 1488442860572
}
]
},
"aggs": {
"theSum": {
"sum": {
"field": "docType.valuesOverTime.values.segCount"
}
}
}
}
}
}
}
}
and get
"theSum": {
"value": 14
}
I've worked on several facet and filter based searches in the past with Solr but I'm struggling to achieve parity with Elasticsearch.
I understand that aggregations are calculated against the results of a query or globally if no query is specified. This is fine, however I would like the counts of those aggregations to be based on the results of a filter.
In Solr this is straightforward - just specify a query and filter - but with Elasticsearch a filter has no effect on aggregates and the documentation is very confusing.
My desired output for the following query is for the suggestions bucket to be scoped to the query but the resulting counts therein to be scoped to the specified filter:
{
"size": 0,
"query": {
"range": {
"published": {
"gte": "now-1y",
"lt": "now"
}
}
},
"filter": {
{
"term": {
"tag.id": "123"
}
},
{
"term": {
"tag.id": "456"
}
},
},
"aggs": {
"tags": {
"nested": {
"path": "tag"
},
"aggs": {
"suggestions": {
"terms": {
"field": "name",
"size": 10,
"min_doc_count": 1
},
"aggs": {
"id": {
"terms": {
"field": "id",
"size": 1
}
}
}
}
}
}
}
}
And given the example mapping:
{
"mappings":{
"content":{
"properties":{
"id":{
"type":"string",
"index":"not_analyzed"
},
"title":{
"type":"string"
},
"byline":{
"type":"string",
"index":"not_analyzed"
},
"body":{
"type":"string"
},
"publishedDate":{
"type":"date",
"format":"dateOptionalTime"
},
"tag":{
"type":"nested",
"include_in_parent":true,
"properties":{
"id":{
"type":"integer"
},
"name":{
"type":"string"
}
}
}
}
}
}
}
Any help is appreciated.
You can get the expected results by :
keeping the query where it is,
but moving the filter part (what you have is equivalent to a post_filter actually and is only applied on the results after aggregations have run) into a filter aggregation
Basically this should work:
{
"size": 0,
"query": {
"range": {
"published": {
"gte": "now-1y",
"lt": "now"
}
}
},
"aggs": {
"tags": {
"nested": {
"path": "tag"
},
"aggs": {
"suggestions": {
"terms": {
"field": "tag.name",
"size": 10,
"min_doc_count": 1
},
"aggs": {
"filtered": {
"filter": {
"terms": {
"tag.id": [
"123",
"456"
]
}
},
"aggs": {
"id": {
"terms": {
"field": "tag.id",
"size": 1
}
}
}
}
}
}
}
}
}
}
When using a Term Filter, I'm not able to use now elasticsearch 1.7.1 anymore. It worked fine in previous versions, but now it returns:
nested: IllegalArgumentException[Invalid format: \"now/y\"]
A query example is:
GET _search
{
"size": 0,
"aggs": {
"price": {
"nested": {
"path": "prices"
},
"aggs": {
"valid": {
"filter": {
"term": {
"prices.referred_year": "now/y"
}
},
"aggs": {
"ranged": {
"range": {
"field": "prices.price",
"ranges": [
{
"to": 10
},
{
"from": 10
}
]
}
}
}
}
}
}
}
}
Schema:
curl -XPUT 'http://localhost:9200/test/' -d '{
"mappings": {
"product": {
"properties": {
"prices": {
"type": "nested",
"include_in_parent": true,
"properties": {
"price": {
"type": "float"
},
"referred_year": {
"type": "date",
"format": "year"
}
}
}
}
}
}
}'
Document example:
curl -XPUT 'http://localhost:9200/test/product/1' -d '{
"prices": [
{
"referred_year": "2015",
"price": "10.00"
},
{
"referred_year": "2016",
"price": "11.00"
}
]
}'
Expected result for the aggregation (gotten by substituting now/y with 2015):
"aggregations": {
"price": {
"doc_count": 2,
"valid": {
"doc_count": 1,
"ranged": {
"buckets": [
{
"key": "*-10.0",
"to": 10,
"to_as_string": "10.0",
"doc_count": 0
},
{
"key": "10.0-*",
"from": 10,
"from_as_string": "10.0",
"doc_count": 1
}
]
}
}
}
}
now/y etc still works fine in the Range Filter and in queries.
I appreciate any help on this. Thanks!
------- UPDATE -------
So, it seems now doesn't work in Term Filters at all, no matter the rounding.
So, although I haven't found any documentation saying so, it seems using the now operator is not allowed in Term Filters. Which actually makes sense.
The correct query would be:
GET test/_search
{
"size": 0,
"aggs": {
"price": {
"nested": {
"path": "prices"
},
"aggs": {
"valid": {
"filter": {
"range": {
"prices.referred_year": {
"gte": "now/y",
"lte": "now/y"
}
}
},
"aggs": {
"ranged": {
"range": {
"field": "prices.price",
"ranges": [
{
"to": 10
},
{
"from": 10
}
]
}
}
}
}
}
}
}
}