I am trying to set all mapped fields to string ie if a json message comes with following:
{
"logDate": "2012-04-23T18:25:43.511Z",
"logId": 123131,
"message": {
"username": "pera",
"password": "pera123"
}
}
I need to log every value as string ie. logId should be logged as "logId": "123131".
Is there a way to tell fluent bit what index mapping to use of maybe there is another setting that changes dynamic type to string?
Maybe can try adding an index template.
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html
Related
I am trying to resolve this error:
Fielddata is disabled on text fields by default. Set fielddata=true on
and saw one post which suggested me to do this; but I didn't get what is your_type endpoint in the given snippet:
PUT your_index/_mapping/your_type
I don't know what version of ElasticSearch you have but as of 7.x the mapping type has been removed.
In your case it could run like this (version > 7.x)
PUT my-index-000001/_mapping
{
"properties": {
"name-field": {
"type": "text",
"fielddata": true
}
}
}
A little about the mapping type:
Since the first release of Elasticsearch, each document has been
stored in a single index and assigned a single mapping type. A mapping
type was used to represent the type of document or entity being
indexed, for instance a twitter index might have a user type and a
tweet type.
Each mapping type could have its own fields, so the user type might
have a full_name field, a user_name field, and an email field, while
the tweet type could have a content field, a tweeted_at field and,
like the user type, a user_name field.
More information here:
https://www.elastic.co/guide/en/elasticsearch/reference/6.5/removal-of-types.html#_why_are_mapping_types_being_removed
I am saving logs to Elasticsearch for analysis but I found there are mixed types of a particular field which causing error when indexing the document.
For example, I may save below log to the index where uuid is an object.
POST /index-000001/_doc
{
"uuid": {"S": "001"}
}
but from another event, the log would be:
POST /index-000001/_doc
{
"uuid": "001"
}
the second POST will fail because the type of uuid is not an object. so I get this error: object mapping for [uuid] tried to parse field [uuid] as object, but found a concrete value
I wonder what the best solution for that? I can't change the log because they are from different application. The first log is from the data of dynamodb while the second one is the data from application. How can I save both types of logs into ES?
If I disable dynamic mapping, I will have to specify all fields in the index mapping. For any new fields, I am not able to search them. so I do need dynamic mapping.
There will be many cases like that. so I am looking for a solution which can cover all conflict fields.
It's perfectly possible using ingest pipelines which are run before the indexing process.
The following would be a solution for your particular use case, albeit somewhat onerous:
create a pipeline
PUT _ingest/pipeline/uuid_normalize
{
"description" : "Makes sure uuid is a hash map",
"processors" : [
{
"script": {
"source": """
if (ctx.uuid != null && !(ctx.uuid instanceof java.util.HashMap)) {
ctx.uuid = ['S': ctx.uuid]; // hash map init
}
"""
}
}
]
}
run the pipeline when ingesting a new doc
POST /index-000001/_doc
{
"uuid": {"S": "001"}
}
POST /index-000001/_doc?pipeline=uuid_normalize <------
{
"uuid": "001"
}
You could now extend this to be as generic as you like but it is assumed that you know what you expect as input in each and every doc. In other words, unlike dynamic templates, you need to know what you want to safeguard against.
You can read more about painless script operators here.
You just cannot.
You should either normalize all your field in a way or another.
Or use 2 separate field.
I can suggest to use a field like this :
"uuid": {"key": "S", "value": "001"}
and skip the key when not necessary.
But you will have to preprocess your value before ingestion.
The thing is that I already defined a field "myvalue" as INTEGER. Now I think was a mistake and I want to store in the same field an string, so I want to change it, without loosing data, to STRING. is there any way of making it?, or I need to re-create the index and re-index the whole data?
I already tried running:
{
"mappings": {
"myvalue": {
"type":"string"
}
}
}
But if I get the mapping again from the server still appear as Integer
There is not any way to change the mapping on a core field type for existing data. You will need to re-create the index with the myvalue field defined as a string and re-index your data.
From this question I can see that it is possible Use existing field as id in elasticsearch
My question is, if can do similar thing but concatenating fields.
{
"RecordID": "a06b0000004SWbdAAG",
"SystemModstamp": "01/31/2013T07:46:02.000Z",
"body": "Test Body"
}
And then do something like
{
"your_mapping" : {
"_id" : {
"path" : "RecordID" + "body"
}
}
}
So the id is automatically formed from concatenating those fields.
No you can't, you can only make the _id point to a field that's within the document, using the dot notation as well if needed (e.g. level1,level2.id).
I'd suggest to have a field that contains the whole id in your documents, or even better to take the id out and provide it in the url, as configuring a path causes the document to be parsed when not needed.
Just started using elasticSearch today. I was wondering if it would be possible to set in some kind of global parameter to use a certain field within a document as the ID always?
My JSON documents will always have it's own unique ID
{
"Record ID": "a06b0000004SWbdAAG",
"System Modstamp": "01/31/2013T07:46:02.000Z",
"body": "Test Body"
}
Here I would like to use Record ID as the ID field.
Regards
You want to use the path setting, see the docs here:
http://www.elasticsearch.org/guide/reference/mapping/id-field/
specifically something like this should work in your mapping:
{
"your_mapping" : {
"_id" : {
"path" : "Record ID"
}
}
}
I've never tried having variable names split up though. You might want to camelcase or underscore them if you run into wierdness.