query on a date range in elasticsearch - elasticsearch

Question

I want to get documents from last 30 days in elastic search but it returns empty.
it is my mapping:
PUT /books
{
"mappings": {
"impressions": {
"properties": {
"booksCreated" : {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis",
"index": true
}
}
}
}
}
and it is my query:
POST /books/_search?size=0
{
"aggs": {
"range": {
"date_range": {
"field": "booksCreated",
"format": "yyyy-MM-dd",
"ranges": [
{ "to": "now" },
{ "from": "now-1M/M" }
]
}
}
}
}
I've tried all possible ways but it returns empty.
but i can query on #timestamp field
the problem is that logstash changes the field type from date to string. my json is :
{
"index":"books",
"type":"book",
"body":{
"impressions":{
"_source":{
"enabled":true
},
"properties":{
"BookCreated":"2017-09-18 12:18:39"
}
}
}
}
and my logstash config:
input {
file {
path => "E:\data2\log\logstash.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => json
}
}
filter {
mutate {
strip => ["message"]
}
}
output {
elasticsearch {
hosts => "localhost"
index => "books"
document_type => "book"
}
}
i will log the json to a log file and logstash send them to elasticsearch
after adding json the mapping chasnges to this:
{
"Books": {
"mappings": {
"Books": {
"properties": {
"#timestamp": {
"type": "date"
},
"#version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"BookCreated": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"body": {
"properties": {
"Books": {
"properties": {
"_source": {
"properties": {
"enabled": {
"type": "boolean"
}
}
},
"properties": {
"properties": {
"BookCreated": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"index": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
it has two BookCreated one isdate and the other is text

Answer 1

You need to put from and to in the same range, like this:
POST /books/_search?size=0
{
"aggs": {
"range": {
"date_range": {
"field": "BookCreated",
"format": "yyyy-MM-dd",
"ranges": [
{
"from": "now-1M/M",
"to": "now"
}
]
}
}
}
}

Answer 2

I'm pretty sure there is an issue with your mapping. First of all, make sure the bookCreated field is named consistently, both in regards to naming as well as capitalization!
Secondly, I believe the reason you have two bookCreated is because your mapping contains a bookCreated property. Your JSON however contains a nested structure: body => properties => bookCreated. Either flatten/transform the book in logstash to the required index structure, or model your index according to your json, which could look something like this?
"mappings": {
"properties": {
"body": {
"type": "object",
"properties": {
"properties": {
"type": "object",
"properties": {
"bookCreated": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis",
"index": true
}
}
}
}
}
}
}
Either way, I recommend you to set "dynamic": "strict" so you will actually see when you make a mistake in the mapping rather than just new fields being created