elasticsearch - add custom field to a specific index - elasticsearch

I hava the JSON for my index that looks like this:
{
"_index": "myindes",
"_type": "external",
"_id": "1",
"_source": {
"id": "1",
"name": "myName",
"description": "myDescription",
"source": "mySource",
}
}
And i want to add a string field in _source named topic
How can i do


You can update the index mapping as
curl -XPUT 'http://localhost:9200/myindex/_mapping/external' -d '
{
"external" : {
"properties" : {
"id": {"type":"string"},
"name": {"type":"string"},
"description": {"type":"string"},
"source": {"type":"string"},
"topic":{"type":"string"} // <---new field
}
}
}'
Although the above step was not necessary but always good to control what you are indexing.
Now, you can index your documents with the new field and it will reflect in new updates. However, old indexed documents will still not contain this new field. You will have to reindex them.

Related

Elasticsearch merge multiple indexes based on common field

I'm using ELK to generate views out of the data from two different DB. One is mysql other one is PostgreSQL. There is no way of writing join query between those two DB instance. But I have a common field call "nic". Following are the documents from each index.
MySQL
index: user_detail
"_id": "871123365V",
"_source": {
"type": "db-poc-user",
"fname": "Iraj",
"#version": "1",
"field_lname": "Sanjeewa",
"nic": "871456365V",
"#timestamp": "2020-07-22T04:12:00.376Z",
"id": 2,
"lname": "Santhosh"
}
PostgreSQL
Index: track_details
"_id": "871456365V",
"_source": {
"#version": "1",
"nic": "871456365V",
"#timestamp": "2020-07-22T04:12:00.213Z",
"track": "ELK",
"type": "db-poc-ceg"
},
I want to merge both index in to single index using common field "nic". And create new index. So I can create visualization on Kibana. How can this be achieved?
Please note that each document in new index should have
"nic,fname,lname,track" as fields. Not the aggregation.

I would leverage the enrich processor to achieve this.
First, you need to create an enrich policy (use the smallest index, let's say it's user_detail):
PUT /_enrich/policy/user-policy
{
"match": {
"indices": "user_detail",
"match_field": "nic",
"enrich_fields": ["fname", "lname"]
}
}
Then you can execute that policy in order to create an enrichment index
POST /_enrich/policy/user-policy/_execute
The next step requires you to create an ingest pipeline that uses the above enrich policy/index:
PUT /_ingest/pipeline/user_lookup
{
"description" : "Enriching user details with tracks",
"processors" : [
{
"enrich" : {
"policy_name": "user-policy",
"field" : "nic",
"target_field": "tmp",
"max_matches": "1"
}
},
{
"script": {
"if": "ctx.tmp != null",
"source": "ctx.putAll(ctx.tmp); ctx.remove('tmp');"
}
},
{
"remove": {
"field": ["#version", "#timestamp", "type"]
}
}
]
}
Finally, you're now ready to create your target index with the joined data. Simply leverage the _reindex API combined with the ingest pipeline we've just created:
POST _reindex
{
"source": {
"index": "track_details"
},
"dest": {
"index": "user_tracks",
"pipeline": "user_lookup"
}
}
After running this, the user_tracks index will contain exactly what you need, for instance:
{
"_index" : "user_tracks",
"_type" : "_doc",
"_id" : "0uA8dXMBU9tMsBeoajlw",
"_score" : 1.0,
"_source" : {
"fname" : "Iraj",
"nic" : "871456365V",
"lname" : "Santhosh",
"track" : "ELK"
}
}
If your source indexes ever change (new users, changed names, etc), you'll need to re-run the above steps, but before doing it, you need to delete the ingest pipeline and the ingest policy (in that order):
DELETE /_ingest/pipeline/user_lookup
DELETE /_enrich/policy/user-policy
After that you can freely re-run the above steps.
PS: Just note that I cheated a bit since the record in user_detail doesn't have the same nic in your example, but I guess it was a copy/paste issue.

Replace document in Elasticsearch index with field condition

If I have indexed a document in Elasticsearch that contains a datetime parameter, or some kind of sequence number, can I update/replace the entire document with a new version if, and only if, the value in my new document is greater than that in the currently indexed document?
Searching has shown me so far how I can affect the values of specific fields through scripting, but I'm not sure if I can use a script or operation as an update criterion, and replace the whole document if it's met.
To be more specific, we have a document object that contains a timestamp of when it was placed on the queue for processing, and since we may have multiple processors pulling things off the queue we would like to ensure that we only index documents newer than the one we already have in the index, discarding any old changes.

Try to use the _update_by_query Api.
Update By Query
Example:
Mappings
PUT my_index
{
"mappings": {
"properties": {
"user": {
"type": "keyword"
},
"timestamp": {
"type": "keyword"
}
}
}
}
Indexing documents
POST my_index/_doc/1
{
"user":"user1",
"timestamp":1234
}
POST my_index/_doc/2
{
"user":"user2",
"timestamp":1235
}
Update By Query
Let's update only documents with timestamp greater than 1234.
POST /my_index/_update_by_query
{
"script": {
"source": "ctx._source.user='new user';", ----> updating field user
"lang": "painless"
},
"query": {
"range": {
"timestamp": {
"gt": 1234
}
}
}
}
You can update other fields or insert new ones, just play with "source": "ctx._source.user='new user';ctx._source.timestamp=456";ctx._source.new_field=value"
Results
{
"_index": "my_index",
"_type": "_doc",
"_id": "2",
"_score": 1,
"_source": {
"user": "new user",
"timestamp": 1235
}
}
Hope this helps

How to concatenate a string on elastic search

How to concatenate a string on elastic search.
for eg: here dasboradList.views has appended to new fields.
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "35",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
},
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "30",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
}
Above code specifies Elastic search index.
we want to append new field in a dasboradList.dasboradList has string type.
Needed json structure is..
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "35",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",`\"id\":\"name+"_"+userId\",\"createdDate\":\"01-01-2016\",\"expirydays\":\"10\"`,\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
},
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "30",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",`\"id\":\"name+"_"+userId\",\"createdDate\":\"01-01-2016\",\"expirydays\":\"10\"`,\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
}

In addition to what #jhilden said , we can indeed update an specific field in a ES document. But you need to enable scripting first.
Directly from the documentation :
#Index a document
curl -XPUT localhost:9200/test/type1/1 -d '{
"counter" : 1,
"tags" : ["red"]
}'
#Increase the count using inline scripting
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : {
"inline": "ctx._source.counter += count",
"params" : {
"count" : 4
}
}
}'
#Add a new field
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : "ctx._source.name_of_new_field = \"value_of_new_field\""
}'
You can also update by query in case that you don't know the id of the document or if you want to do a bulk update.
POST /twitter/_update_by_query
{
"script": {
"inline": "ctx._source.likes++"
},
"query": {
"term": {
"user": "kimchy"
}
}
}
More details of both concepts:
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-update.html
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-update-by-query.html
More information about inline scripting :
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-scripting.html

If I understand your problem correctly you want to UPDATE a record in ElasticSearch. There is no way in ES to do a partial update. What I mean is, there is no equivalent to this:
UPDATE tbl1
SET col1 = 'I am updating only 1 column'
WHERE id = 123
In ElasticSaerch we update a record by:
GET the record you are looking for
update the record
POST the FULL, updated, record back to ElasticSearch specifying the existing _id field.
This will overwrite the old record, something you can verify by looking at the _version property.

Kibana 4 index patterns time-field

Is there a way to make Kibana-4 show a timestamp field which is a epoch time as the time-field when creating an index pattern.
I know how to make this with the _timestamp field by editing the metaFields in the settings, but I would like this to be a custom field.
Eg: Let's say this is the document I am storing in ES:
{
"_id": "AVCbqgiV7A6BIPyJuJRS",
"_index": "scm-get-config-stg",
"_score": 1.0,
"_source": {
"serverDetails": {
"cloudDC": "xxx",
"cloudName": "yyyy",
"hostName": "hostname",
"ipAddress": "10.247.194.49",
"runOnEnv": "stg",
"serverTimestamp": 1445720623246
}
},
"_type": "telemetry"
}
Now I would like to create an index pattern where the Time-field name should be serverTimestamp.

Elastic Search - Querying on values

I have an elasticsearch index with the following values
{
"_index": "article",
"_type": "articleId",
"_id": "10970",
"_score": 1,
"_source": {
"url": "http%3A%2F%2Fwww.tomshardware.com%2Fnews%2FAir-Traffic-Software-DoS-Attacks%2C16471.html%23xtor%3DRSS-181",
"title": "Air%20Traffic%20Software%20Vulnerable%20to%20DoS%20Attacks",
"publicationId": "888",
"text": "%20%3Cp%3E%3Cstrong%3EA%20security%20researcher%20revealed%20a%20flaw%20in%20commonly%20used%20air%20traffic%20control%20software%20that%20would%20allow%20an%20attacker%20to%20create%20an%20unlimited%20number%20of%20phantom%20flights.%3C%2Fstrong%3E%3C%2Fp%3E%20%3Cp%3E%3Ca%20target%3D%22_blank%22%3E%3C%2Fa%3E%3C%2Fp%3E%20%3Cp%3EAccording%20to%20Andrei%20Costin%2C%20%242%2C000%20in%20equipment%20and%20%22modest%20tech%20skills%22%20are%20enough%20to%20throw%20an%20air%20traffic%20control%20system%20of%20virtually%20any%20airport%20into%20complete%20disarray.%20The%20ADS-B%20system%20that%20is%20used%20across%20the%20world%20is%20vulnerable%20as%20it%20does%20not%20verify%20that%20incoming%20traffic%20signals%20as%20genuine.%20%3C%2Fp%3E%20%3Cp%3ECostin%20says%20that%20a%20hacker%20could%20inject%20flights%20that%20do%20not%20exist%20and%20could%20confuse%20an%20air%20controller%20station.%20Air%20controllers%20could%20cross-check%20flights%20with%20flight%20schedules%2C%20but%20if%20the%20number%20of%20phantom%20flights%20is%20high%20enough%2C%20there%20is%20no%20way%20that%20cross-checks%20would%20work.%20Consider%20it%20like%20an%20DoS%20attack%20on%20an%20air%20traffic%20control%20system.%3C%2Fp%3E%20%3Cp%3ECostin%20noted%20that%20rogue%20signals%20from%20the%20ground%20can%20be%20generally%20identified%20and%20ruled%20out%20as%20malicious%20signals%2C%20but%20there%20is%20no%20way%20to%20do%20the%20same%20for%20robotic%20aircraft%2C%20for%20example.%20He%20also%20noted%20that%20data%20sent%20from%20airplanes%20to%20air%20traffic%20controllers%20is%20unencrypted%20and%20can%20be%20captured%20by%20unidentified%20sources.%20Since%20this%20applies%20to%20any%20aircraft%2C%20it%20is%20in%20theory%20possible%20to%20deploy%20airplane%20tracking%20devices%20to%20track%20specific%20aircraft.%3C%2Fp%3E%20%3C%2Fp%3E%3Cp%3E%20%3Cp%3E%3Ca%20target%3D%22_blank%22%20href%3D%22mailto%3Anews-us%40bestofmedia.com%3Fsubject%3DNews%2520Article%2520Feedback%22%3E%3Cem%3E%3Csub%3EContact%20Us%20for%20News%20Tips%2C%20Corrections%20and%20Feedback%3C%2Fsub%3E%3C%2Fem%3E%3C%2Fa%3E%3C%2Fp%3E",
"keywords": {
"air": "3.4965034965034962",
"traffic": "3.4965034965034962",
"flights": "2.797202797202797",
"": "2.797202797202797",
"Costin": "2.097902097902098",
"aircraft": "2.097902097902098",
"signals": "2.097902097902098",
"control": "2.097902097902098",
"system": "2.097902097902098",
"there": "1.3986013986013985"
}
}
}
I am trying to write a query to search does this index have the keyword flights (which it does) but I am having difficulty
Its straightforward running a match query on one of the other fields like text but encountering problems when trying to do the same or similar for keywords
Is there a way of performing this search with the current setup or should I add the keywords in differently?

If I understood you correctly, you would like to find all records that have the field keyword.flights and the value of this field is not important. You can do it using string query:
curl "http://localhost:9200/_search?q=keywords.flights:*"
Or using the exist filter:
curl "http://localhost:9200/_search" -d '{
"query": {
"constant_score" : {
"filter" : {
"exists" : { "field" : "keywords.flights" }
}
}
}
}'

Resources