How to concatenate a string on elastic search - elasticsearch

How to concatenate a string on elastic search.
for eg: here dasboradList.views has appended to new fields.
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "35",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
},
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "30",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
}
Above code specifies Elastic search index.
we want to append new field in a dasboradList.dasboradList has string type.
Needed json structure is..
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "35",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",`\"id\":\"name+"_"+userId\",\"createdDate\":\"01-01-2016\",\"expirydays\":\"10\"`,\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
},
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "30",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",`\"id\":\"name+"_"+userId\",\"createdDate\":\"01-01-2016\",\"expirydays\":\"10\"`,\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
}

In addition to what #jhilden said , we can indeed update an specific field in a ES document. But you need to enable scripting first.
Directly from the documentation :
#Index a document
curl -XPUT localhost:9200/test/type1/1 -d '{
"counter" : 1,
"tags" : ["red"]
}'
#Increase the count using inline scripting
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : {
"inline": "ctx._source.counter += count",
"params" : {
"count" : 4
}
}
}'
#Add a new field
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : "ctx._source.name_of_new_field = \"value_of_new_field\""
}'
You can also update by query in case that you don't know the id of the document or if you want to do a bulk update.
POST /twitter/_update_by_query
{
"script": {
"inline": "ctx._source.likes++"
},
"query": {
"term": {
"user": "kimchy"
}
}
}
More details of both concepts:
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-update.html
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-update-by-query.html
More information about inline scripting :
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-scripting.html

If I understand your problem correctly you want to UPDATE a record in ElasticSearch. There is no way in ES to do a partial update. What I mean is, there is no equivalent to this:
UPDATE tbl1
SET col1 = 'I am updating only 1 column'
WHERE id = 123
In ElasticSaerch we update a record by:
GET the record you are looking for
update the record
POST the FULL, updated, record back to ElasticSearch specifying the existing _id field.
This will overwrite the old record, something you can verify by looking at the _version property.

Related

ElasticSearch: POST update without Index, Type and Id

Here is one of my documents in elasticsearch:
{
"_index": "2017-10-21",
"_type": "cat",
"_id": "14",
"_score": 2.2335923,
"_source": {
"name": "Biscuit",
"breed": "Persian",
"age": "3"
}
}
I know that it's possible to do a POST update to add a new field to an existing document this way:
POST [index] / [type] / [id] / _update
So for example, if I want to add a new field "hairy" to my document:
POST 2017-10-21 / cat / 14 / _update
{
"script" : "ctx._source.hairy = 'yes'"
}
I will have this result:
{
"_index": "2017-10-21",
"_type": "cat",
"_id": "14",
"_source": {
"name": "Biscuit",
"breed": "Persian",
"age": "3",
"hairy": "yes"
}
}
However, I would like to add a new field to ALL my documents, no matter their index, type or id. Unfortunately, even after hours of research I haven't found a way to do a POST update without using index, type or id.
So, my questions are: Is it even possible? If it's not, is there another way to do what I want to do?
Thank you in advance for any help you can provide!
I have finally found a solution!
I needed to use POST update_by_query instead of POST update
POST * / _update_by_query
{
"script" : {
"inline": "ctx._source.hairy = 'yes'"
}
}
The star means that you want to select all existing indexes/indices

Elasticsearch match phrase prefix not matching all terms

I am having an issue where when I use the match_phrase_prefix query in Elasticsearch, it is not returning all the results I would expect it to, particularly when the query is one word followed by one letter.
Take this index mapping (this is a contrived example to protect sensitive data):
http://localhost:9200/test/drinks/_mapping
returns:
{
"test": {
"mappings": {
"drinks": {
"properties": {
"name": {
"type": "text"
}
}
}
}
}
}
And amongst millions of other records are these:
{
"_index": "test",
"_type": "drinks",
"_id": "2",
"_score": 1,
"_source": {
"name": "Johnnie Walker Black Label"
}
},
{
"_index": "test",
"_type": "drinks",
"_id": "1",
"_score": 1,
"_source": {
"name": "Johnnie Walker Blue Label"
}
}
The following query, which is one word followed by two letters:
POST http://localhost:9200/test/drinks/_search
{
"query": {
"match_phrase_prefix" : {
"name" : "Walker Bl"
}
}
}
returns this:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.5753642,
"hits": [
{
"_index": "test",
"_type": "drinks",
"_id": "2",
"_score": 0.5753642,
"_source": {
"name": "Johnnie Walker Black Label"
}
},
{
"_index": "test",
"_type": "drinks",
"_id": "1",
"_score": 0.5753642,
"_source": {
"name": "Johnnie Walker Blue Label"
}
}
]
}
}
Whereas this query with one word and one letter:
POST http://localhost:9200/test/drinks/_search
{
"query": {
"match_phrase_prefix" : {
"name" : "Walker B"
}
}
}
returns no results. What could be happening here?
I will assume that you are working with Elasticsearch 5.0 and above.
I think it might have to be because of the max_expansions default value.
As seen in the documentation here, the max_expansions parameters is used to control how many prefixes the last term will be expanded with. The default value is 50 and it might explain why you find "black" and "blue" with the two first letters B and L, but not with the B only.
The documentation is pretty clear about it:
The match_phrase_prefix query is a poor-man’s autocomplete. It is very easy to use, which let’s you get started quickly with search-as-you-type but it’s results, which usually are good enough, can sometimes be confusing.
Consider the query string quick brown f. This query works by creating a phrase query out of quick and brown (i.e. the term quick must exist and must be followed by the term brown). Then it looks at the sorted term dictionary to find the first 50 terms that begin with f, and adds these terms to the phrase query.
The problem is that the first 50 terms may not include the term fox so the phase quick brown fox will not be found. This usually isn’t a problem as the user will continue to type more letters until the word they are looking for appears
I wouldn't be able to tell you if it's ok to increase this parameter above 50 if you are looking for good performances since I never tried myself.

elasticsearch - add custom field to a specific index

I hava the JSON for my index that looks like this:
{
"_index": "myindes",
"_type": "external",
"_id": "1",
"_source": {
"id": "1",
"name": "myName",
"description": "myDescription",
"source": "mySource",
}
}
And i want to add a string field in _source named topic
How can i do
You can update the index mapping as
curl -XPUT 'http://localhost:9200/myindex/_mapping/external' -d '
{
"external" : {
"properties" : {
"id": {"type":"string"},
"name": {"type":"string"},
"description": {"type":"string"},
"source": {"type":"string"},
"topic":{"type":"string"} // <---new field
}
}
}'
Although the above step was not necessary but always good to control what you are indexing.
Now, you can index your documents with the new field and it will reflect in new updates. However, old indexed documents will still not contain this new field. You will have to reindex them.

Get elasticsearch result based on two keys

I want to get all docs who's "PayerAccountId" should equal to "123" and "UsageStartDate" should be in range [2015-05-01 TO 2015-05-10]
I am expecting something to run like this,
curl -X GET http://192.168.1.3:9200/_all/_search -d '{"query" : {"match" : { "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-01-10]" }}}'
Obviously it's not working any suggestions
Here is my one doc
{
"_index": "logstash-2015.04.01",
"_type": "sam_billing_hourly",
"_id": "ef06aeb5fbab7191a43335740779fc73b667ff0b",
"_score": 1.0,
"_source": {
"#version": "1",
"#timestamp": "2015-04-01T02:00:00.000Z",
"Operation": "A",
"PayerAccountId": "156023466485",
"PricingPlanId": 482457,
"RateId": 3035133,
"RecordType": "LineItem",
"UnBlendedCost": 4.0e-07,
"UnBlendedRate": 4.0e-07,
"UsageEndDate": "2015-04-01T03:00:00Z",
"UsageQuantity": 1,
"UsageStartDate": "2015-04-01T02:00:00Z",
"UsageType": "DNS-Queries",
"fingerprint": "ef06aeb5fbab7191a43335740779fc73b667ff0b"
}
You need a query_string not a match:
{
"query": {
"query_string": {
"query": "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-10-01]"
}
}
}

Elastic Search - Querying on values

I have an elasticsearch index with the following values
{
"_index": "article",
"_type": "articleId",
"_id": "10970",
"_score": 1,
"_source": {
"url": "http%3A%2F%2Fwww.tomshardware.com%2Fnews%2FAir-Traffic-Software-DoS-Attacks%2C16471.html%23xtor%3DRSS-181",
"title": "Air%20Traffic%20Software%20Vulnerable%20to%20DoS%20Attacks",
"publicationId": "888",
"text": "%20%3Cp%3E%3Cstrong%3EA%20security%20researcher%20revealed%20a%20flaw%20in%20commonly%20used%20air%20traffic%20control%20software%20that%20would%20allow%20an%20attacker%20to%20create%20an%20unlimited%20number%20of%20phantom%20flights.%3C%2Fstrong%3E%3C%2Fp%3E%20%3Cp%3E%3Ca%20target%3D%22_blank%22%3E%3C%2Fa%3E%3C%2Fp%3E%20%3Cp%3EAccording%20to%20Andrei%20Costin%2C%20%242%2C000%20in%20equipment%20and%20%22modest%20tech%20skills%22%20are%20enough%20to%20throw%20an%20air%20traffic%20control%20system%20of%20virtually%20any%20airport%20into%20complete%20disarray.%20The%20ADS-B%20system%20that%20is%20used%20across%20the%20world%20is%20vulnerable%20as%20it%20does%20not%20verify%20that%20incoming%20traffic%20signals%20as%20genuine.%20%3C%2Fp%3E%20%3Cp%3ECostin%20says%20that%20a%20hacker%20could%20inject%20flights%20that%20do%20not%20exist%20and%20could%20confuse%20an%20air%20controller%20station.%20Air%20controllers%20could%20cross-check%20flights%20with%20flight%20schedules%2C%20but%20if%20the%20number%20of%20phantom%20flights%20is%20high%20enough%2C%20there%20is%20no%20way%20that%20cross-checks%20would%20work.%20Consider%20it%20like%20an%20DoS%20attack%20on%20an%20air%20traffic%20control%20system.%3C%2Fp%3E%20%3Cp%3ECostin%20noted%20that%20rogue%20signals%20from%20the%20ground%20can%20be%20generally%20identified%20and%20ruled%20out%20as%20malicious%20signals%2C%20but%20there%20is%20no%20way%20to%20do%20the%20same%20for%20robotic%20aircraft%2C%20for%20example.%20He%20also%20noted%20that%20data%20sent%20from%20airplanes%20to%20air%20traffic%20controllers%20is%20unencrypted%20and%20can%20be%20captured%20by%20unidentified%20sources.%20Since%20this%20applies%20to%20any%20aircraft%2C%20it%20is%20in%20theory%20possible%20to%20deploy%20airplane%20tracking%20devices%20to%20track%20specific%20aircraft.%3C%2Fp%3E%20%3C%2Fp%3E%3Cp%3E%20%3Cp%3E%3Ca%20target%3D%22_blank%22%20href%3D%22mailto%3Anews-us%40bestofmedia.com%3Fsubject%3DNews%2520Article%2520Feedback%22%3E%3Cem%3E%3Csub%3EContact%20Us%20for%20News%20Tips%2C%20Corrections%20and%20Feedback%3C%2Fsub%3E%3C%2Fem%3E%3C%2Fa%3E%3C%2Fp%3E",
"keywords": {
"air": "3.4965034965034962",
"traffic": "3.4965034965034962",
"flights": "2.797202797202797",
"": "2.797202797202797",
"Costin": "2.097902097902098",
"aircraft": "2.097902097902098",
"signals": "2.097902097902098",
"control": "2.097902097902098",
"system": "2.097902097902098",
"there": "1.3986013986013985"
}
}
}
I am trying to write a query to search does this index have the keyword flights (which it does) but I am having difficulty
Its straightforward running a match query on one of the other fields like text but encountering problems when trying to do the same or similar for keywords
Is there a way of performing this search with the current setup or should I add the keywords in differently?
If I understood you correctly, you would like to find all records that have the field keyword.flights and the value of this field is not important. You can do it using string query:
curl "http://localhost:9200/_search?q=keywords.flights:*"
Or using the exist filter:
curl "http://localhost:9200/_search" -d '{
"query": {
"constant_score" : {
"filter" : {
"exists" : { "field" : "keywords.flights" }
}
}
}
}'

Resources