I have an index with a field I am storing date information in. The field is in conflict at the moment. As far as I can tell there are three kinds of values:
some documents don't have the field
some documents have the field like this in the JSON:
"timestamp": "2019-03-01T23:32:28Z"
other documents have the field like this in the JSON:
"timestamp": "1551206688760"
I want to fix the conflict.
I've tried doing a wildcard search and I get the following error:
failed to parse date field [*] with format [strict_date_optional_time||epoch_millis]
I have two questions, ultimately.
1) Is the core problem causing the conflict that when I tried to represent the timestamp in epoch_millis that I used a string rather than a number? IOW, "timestamp": 1551206688760 would have been good?
2) What is the proper way to fix this without simply tossing all the data out?
Unfortunately you will need to reindex.
Create new index with date mapping to provide multiple formats
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
Reindex your data
Use aliases
Related
I've a field number on Elasticsearch via Kibana.
This is an example field:
"insert.datePeticio:1,546,185,770,733"
I need to convert this field to date, for visualize purpose.
How can I parse it to date mode?
As far as I understand, you have an integer field on Elasticsearch that stores date/time in milliseconds since the epoch. You would like to convert it to an appropriate date type to visualise on Kibana properly. In this case, I would recommend two solutions:
1) If you are able to define a mapping, use the the following for insert.datePeticio field:
"mappings": {
"_doc": {
"properties": {
"insert.datePeticio": {
"format": "epoch_millis",
"type": "date"
}
}
}
}
This allows Kibana to define and represent insert.datePeticio field as date even though the actual value is store in miliseconds as integer on Elasticsearch.
2) If not, so cannot make any changes on the original mapping, create a scripted field on Kibana as follows:
Go to Management > Index Patterns
Select the Index Pattern you want to modify
Select the index pattern’s Scripted Fields tab
Click Add Scripted Field
Fill the form referring to the image below, and Save it.
If you go to Kibana > Discover, you can see two fields together in different representations and types: insert.datePeticio: 1,546,185,770,733 and insert.datePeticio_UTC:December 30th 2018, 16:02:50.733. Since being a date type, the scripted field insert.datePeticio_UTC can be easily used to create visualisations based on date aggregations.
Note: Scripted fields compute data on the fly from the data in your Elasticsearch indices. Keep in mind that computing data on the fly with scripted fields can be very resource intensive and can have a direct impact on Kibana’s performance.
I am using packetbeat to monitor mysql port on 3306 and it is working very well.
I can easily search for any word on discovery tab. For e.g.
method:SET
This works as expected. But If I change it to
query:SET
then it does not return the documents with the word "SET" in query field. Is the query field indexed differently?
How do I make "query" field searchable?
Update:
Is this because of parameter "ignore_above" that is used for all string fields? I checked the mapping using this API...
GET /packetbeat-2018.02.01/_mapping/mysql/
How do I remove this restriction and make all future beats to index query field?
Update 2:
If I mention the entire string in the search based on "query" field, it works as expected...
query:"SELECT name, type, comment FROM mysql.proc WHERE name like 'residentDetails_get' and db <=> 'portal' ORDER BY name, type"
This returns all 688 records in the last 15 minutes. When I search the following, I expect to get more...
query:"SELECT"
But I do not get a single record. I guess this is because the way document is indexed. I will prefer to get back equivalent of SQL : query like '%SELECT%'
That's the correct behavior, given the query and the mapping of the field. And it's not about the 1024 limit. You can either omit query: part so that Elasticsearch will use the _all field (which will be removed in the near future) but here it depends on the version of the Stack you use.
Or, better and more correct approach, is to configure the query field differently in the packetbeat template (so that next indices will use the new mapping) to be like this:
"query": {
"type": "text",
"fields": {
"raw": {
"type": "keyword",
"ignore_above": 1024
}
}
}
The main idea is that ES is not splitting the values in the query field (since it's keyword) and you need a way to do this. You could use wildcards, but ES doesn't like them (especially the leading wildcards) and you could have performance issue when running such a query. The "correct" approach from ES point of view is the one I already mentioned: make the field analyzed, keep a raw version of it (for sorting and aggregations) and the simple version of it for searches.
The query field of packetbeat is declared as "keyword". Therefore you can search the entire query only. For e.g.
query: "select * from mytable"
But what if we need to search for query: "mytable" ?
You need to make the query field searchable by modifying fields.yml file. Add the type:text parameter to query field of MySQL section of fields.yml file found in /etc/packetbeat
The relevant section of the file will look like this...
- name: query
type: text
description: >
The query in a human readable format. For HTTP, it will typically be
something like `GET /users/_search?name=test`. For MySQL, it is
something like `SELECT id from users where name=test`.
I'm using Elastic Search 5.6.2 with Kibana and I'm currently facing a problem
My documents are indexed on the field timestamp which is normally an integer, however recently somebody has logged a document with a timestamp that is not an integer, and Kibana complains of conflicting type.
The discover panels display nothing and the following errors pop:
Saved "field" parameter is now invalid. Please select a new field.
Discover: "field" is a required parameter
How can I look for the document(s) causing these conflicts so that to find the service creating bad logs ?
The field type (either integer or text/keyword) is not defined on per document basis but rather on per index basis (in the mappings). I guess you are manipulating timeseries data, and you probably have un index per day (or per month or ...).
In Kibana Dev Tools:
List the created indices with GET _cat/indices
For each index (logstash-2017.09.28 in my example) do a GET logstash-2017.09.28/_mapping and check the type of the field in #timestamp
The field type is probably different between indices.
You won't be able to change the field type on created indices. Deleting document won't solve you're problem. The only solution is to drop the index or reindex the whole index with a new field type (in a specific mapping).
To avoid this problem on future indices, the solution is to create an index template with a mapping telling that the field #timestamp is of type date or whatever.
So I have following problem which I'm trying to solve last two days. I have python script which parses logs and inserts data in elastic search, dynamically creating indices via bulk function.
Problem is my mapping has one "type": "nested" property, something like "users" field. And particularly when I'm only adding "type": "nested" in this property I can't query objects from Kibana nor creating any vizualization (because nested objects are separate documents If I'm not making mistakes). First think I tried: adding aditional "include_in_parent": true parameter to users field, but as result I got "wrong" queries (i.e. running something like +users.name: 'test' +users.age: 30) would result in ANY document which has those two fields, not exactly referring to ONE user object. Also vizualization was obviously wrong too.
Second solution I found was adding parent-child relationship. But this could be potentially be waste of time as I don't know will it result in correct queries. So I'm asking, if it will be normal solution to my problem?
Found out that Kibana doesn't support nested objects.
But ppadovani made this fork which supports this feature.
https://github.com/homeaway/kibana/tree/nestedSupport-4.5.4
While trying to update the mapping I get the following error:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"mapper [timestamp] of different type, current_type [string], merged_type [date]"}],"type":"illegal_argument_exception","reason":"
mapper [timestamp] of different type, current_type [string], merged_type [date]"},"status":400}
I m trying to run the following command on windows
curl -XPUT localhost:9200/logstash-*/_mapping/log?update_all_types -d "{
"properties":
{
"timestamp":
{
"type": "date",
"format": "MM-dd-yyyy HH:mm:ss",
"fielddata":{"loading" : "lazy"} }
}
}";
How I can change the datatype of date field from string to date type with a particular format.
I tried to change the mapping of a string datatype to change it to eager loading and not_analyzed from analyzed, but it gives the following error:
{"root_cause":[{"type":"illegal_argument_exception","reason":"Mapper for [AppName] conflicts with existing mapping in other types:\n[mapper [AppName] has different [index] values, mapper [App
different [doc_values] values, cannot change from disabled to enabled, mapper [AppName] has different [analyzer]]"}],"type":"illegal_argument_exception","reason":"Mapper for [AppName] conflict with
existing mapping in other types:\n[mapper [AppName] has different [index] values, mapper [AppName] has different [doc_values] values, cannot change from disabled to enabled, mapper [AppName]
rent [analyzer]]"},"status":400}
Here is my query for the same:
curl -XPUT localhost:9200/logstash-*/_mapping/log?update_all_types -d "{
"properties":
{"AppName":
{
"type": "string",
"index" : "not_analyzed",
"fielddata":{"loading" : "eager"}
}
}
}"
However, if I change it from not_analyzed to analyzed it gives a acknowledged=true message. How can I change the analyzer.
You cannot change existing data types mapping. As Elastic docs say:
Although you can add to an existing mapping, you can’t change existing field mappings. If a mapping already exists for a field, data from that field has probably been indexed. If you were to change the field mapping, the indexed data would be wrong and would not be properly searchable.
We can update a mapping to add a new field, but we can’t change an
existing field from analyzed to not_analyzed.
Your only option is to create a new index with the new mapping and reindex the data from the old index to the new one.
No, you cannot change a single field definition.
If you want to change the field definition for a single field in a single type, you have little option but to reindex all of the documents in your index.
Why can't you change mappings? This article Changing Mapping with Zero Downtime explains,
In order to make
your data searchable, your database needs to know what type of data
each field contains and how it should be indexed.
If you switch a
field type from e.g. a string to a date, all of the data for that
field that you already have indexed becomes useless. One way or
another, you need to reindex that field.
This applies not just to Elasticsearch, but to any database that uses
indices for searching. And if it isn't using indices then it is
sacrificing speed for flexibility.
What happens when you index a document with an incorrect field type?
A conversion will be attempted. If no valid conversion exists, an exception is thrown.
Elasticsearch: The Definitive Guide has a note regarding an example, a string is entered but long is expected. A conversion will be attempted. But an exception will still be thrown if no valid conversion exists.
[...] if the field is already
mapped as type long, then ES will try to convert the string
into a long, and throw an exception if it can’t.
Can I have the document indexed anyway, ignoring the malformed fields?
Yes. ES5 provides a ignore_malformed mapping parameter. Elasticsearch Reference explains that,
Trying to index the wrong datatype into a field throws an exception by
default, and rejects the whole document. The ignore_malformed
parameter, if set to true, allows the exception to be ignored. The
malformed field is not indexed, but other fields in the document are
processed normally.