I have some logs like this:
2022-12-07 17:22:53,838 [INFO]: {"status_code": 304, "method": "GET", "url": "/backend/some/url", "remote_ip": "rem.ote.ip.add", "response_time": 101.61018371582031} - tornado.access
I use an aggregator to parse those logs into the fields timestamp, loglevel, status_code etc. and send them to my opensearch instance. The problem is that the timestamp field is of type string in opensearch, so it cannot be sorted by in the Discover page. Oddly enough, it also has a field date of type float.
I then tried creating a new index with a changed timestamp format as per this answer (second option).
The problem here is that opensearch either uses the date it received the logs as the timestamp, or it just doesn't recognize any timestamp field:
If you follow this solution, the timestamp field type should be the date.
Can you share the mapping of the index?
GET index_name
Can you check the index_pattern and make sure the log-9 index_pattern date is set to timestamp
Related
I am new to apache NiFi.
I am trying to put data into elasticsearch using nifi.
I want to specify an index name by combining a specific string and the value converted from a timestamp field into date format.
I was able to create the desired shape with the expression below, but failed to create the index name with the value of the timestamp field of the content.
${now():format('yyyy-MM-dd')}
example json data
{
"timestamp" :1625579799000,
"data1": "abcd",
"date2": 12345
}
I would like to get the following result:
index : "myindex-2021.07.06"
What should I do? please tell me how
I know that if you use the PutElasticSearch Processor, you can provide it with a specific index name to use. And as long as the index name meets the proper ElasticSearch format for naming a new index, if the enable auto index creation in ElasticSearch is turned on, then when sent, Elastic will create the new index with the desired name. This has worked for me. Double check the Elastic Naming Rules that can be found here or https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-indexing.html
I have an index with a field I am storing date information in. The field is in conflict at the moment. As far as I can tell there are three kinds of values:
some documents don't have the field
some documents have the field like this in the JSON:
"timestamp": "2019-03-01T23:32:28Z"
other documents have the field like this in the JSON:
"timestamp": "1551206688760"
I want to fix the conflict.
I've tried doing a wildcard search and I get the following error:
failed to parse date field [*] with format [strict_date_optional_time||epoch_millis]
I have two questions, ultimately.
1) Is the core problem causing the conflict that when I tried to represent the timestamp in epoch_millis that I used a string rather than a number? IOW, "timestamp": 1551206688760 would have been good?
2) What is the proper way to fix this without simply tossing all the data out?
Unfortunately you will need to reindex.
Create new index with date mapping to provide multiple formats
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
Reindex your data
Use aliases
I have an Elastic-Search index which stores products with 'createdTime' field which signifies when a document/product is added, currently i am saving 'createdTime' as 'Long' instead of 'Date' and in that field i am storing milliseconds from epoch, My question is will storing 'createdTime' as 'Long' instead of 'Date' makes sorting or any such operations slow.
ES version: 2.4.1
Yes, Long is faster but let Elastic handle that.
Internally, fields with Date datatype are stored as long.
Following is the excerpt from Elastic Docs:
Internally, dates are converted to UTC (if the time-zone is specified) and stored as a long number representing milliseconds-since-the-epoch
Storing the date as long datatype requires you to do extra processing on your side.
In addition, you are loosing the ability to view the date field as like date. For ease of use, looking at a long date isn't right.
If you use Date datatype, elastic will store the field in its inverted index as a long value. However, it will convert it to viewable date format before retrieving it back to you.
So, just use Date datatype.
I have an issue similar to this Difference between #timestamp and timestamp field in elasticsearch. But I will need solution for it.
We use Kibana which by default use #timestamp as time filter. Yes, I can change it to whatever field manually EVERYTIME people create the time filter, but it is impossible for EVERYBODY in our big team to know it. So we need #timestamp.
#timestamp won't show up even I use the mapping here,
"_timestamp" : {
"enabled" : true,
"store" : true
}
So I workaround by adding a field name called #timestamp. I can use curl to add documents to it and The time filer start working.
However, when I move to use NEST api which cannot create #timestamp field. Even I define the field name as #timestamp, NEST api automatically change it to timestamp.
So Kibana time filter broken again.
Any suggestion?
Just figured it out. Nest API does have a way to explicitly set the field name.
[ElasticProperty(Name = "#timestamp", Type = FieldType.Date, DateFormat = "yyyy-MM-dd'T'HH:mm:ss", Store = true)]
So this is resolved.
Steps:
1. Define a date field in a mapping.
2. Insert a epoch millisecond (long) value into that field.
Can elastic search returns a string value (yyyy-MM-ddTHH:mm:SS) of that field for a search?
From what I understand of the date-format documentation of ElasticSearch, it will always accept a milliseconds-since-epoch input next to input in the format given by the format, and it will produce a String output using the (first) format given. If you don't provide a format, then the "date_optional_time" format will be used (yyyy-MM-dd’T'HH:mm:ss.SSSZZ).
If the time zone in there is a problem for you, you'd need to give ElasticSearch your intended format.
I don't have the code to hand, but in my testing I believe I managed to do the following:
I used the date formatter on the field and the query fields definition to do this:
curl -XGET 'http://localhost:9200/twitter/tweet/1?fields=title,date_field.date_time'
using the date formats specified here: http://www.elasticsearch.org/guide/reference/mapping/date-format/
If you want a full document returned, this may be onerous. In which case is it possible to use an alias 'view' mapping to get the result to return differently from your primary mapping? Possibly this has become a half-answer.