Elastic Search - Querying on values - elasticsearch

I have an elasticsearch index with the following values
{
"_index": "article",
"_type": "articleId",
"_id": "10970",
"_score": 1,
"_source": {
"url": "http%3A%2F%2Fwww.tomshardware.com%2Fnews%2FAir-Traffic-Software-DoS-Attacks%2C16471.html%23xtor%3DRSS-181",
"title": "Air%20Traffic%20Software%20Vulnerable%20to%20DoS%20Attacks",
"publicationId": "888",
"text": "%20%3Cp%3E%3Cstrong%3EA%20security%20researcher%20revealed%20a%20flaw%20in%20commonly%20used%20air%20traffic%20control%20software%20that%20would%20allow%20an%20attacker%20to%20create%20an%20unlimited%20number%20of%20phantom%20flights.%3C%2Fstrong%3E%3C%2Fp%3E%20%3Cp%3E%3Ca%20target%3D%22_blank%22%3E%3C%2Fa%3E%3C%2Fp%3E%20%3Cp%3EAccording%20to%20Andrei%20Costin%2C%20%242%2C000%20in%20equipment%20and%20%22modest%20tech%20skills%22%20are%20enough%20to%20throw%20an%20air%20traffic%20control%20system%20of%20virtually%20any%20airport%20into%20complete%20disarray.%20The%20ADS-B%20system%20that%20is%20used%20across%20the%20world%20is%20vulnerable%20as%20it%20does%20not%20verify%20that%20incoming%20traffic%20signals%20as%20genuine.%20%3C%2Fp%3E%20%3Cp%3ECostin%20says%20that%20a%20hacker%20could%20inject%20flights%20that%20do%20not%20exist%20and%20could%20confuse%20an%20air%20controller%20station.%20Air%20controllers%20could%20cross-check%20flights%20with%20flight%20schedules%2C%20but%20if%20the%20number%20of%20phantom%20flights%20is%20high%20enough%2C%20there%20is%20no%20way%20that%20cross-checks%20would%20work.%20Consider%20it%20like%20an%20DoS%20attack%20on%20an%20air%20traffic%20control%20system.%3C%2Fp%3E%20%3Cp%3ECostin%20noted%20that%20rogue%20signals%20from%20the%20ground%20can%20be%20generally%20identified%20and%20ruled%20out%20as%20malicious%20signals%2C%20but%20there%20is%20no%20way%20to%20do%20the%20same%20for%20robotic%20aircraft%2C%20for%20example.%20He%20also%20noted%20that%20data%20sent%20from%20airplanes%20to%20air%20traffic%20controllers%20is%20unencrypted%20and%20can%20be%20captured%20by%20unidentified%20sources.%20Since%20this%20applies%20to%20any%20aircraft%2C%20it%20is%20in%20theory%20possible%20to%20deploy%20airplane%20tracking%20devices%20to%20track%20specific%20aircraft.%3C%2Fp%3E%20%3C%2Fp%3E%3Cp%3E%20%3Cp%3E%3Ca%20target%3D%22_blank%22%20href%3D%22mailto%3Anews-us%40bestofmedia.com%3Fsubject%3DNews%2520Article%2520Feedback%22%3E%3Cem%3E%3Csub%3EContact%20Us%20for%20News%20Tips%2C%20Corrections%20and%20Feedback%3C%2Fsub%3E%3C%2Fem%3E%3C%2Fa%3E%3C%2Fp%3E",
"keywords": {
"air": "3.4965034965034962",
"traffic": "3.4965034965034962",
"flights": "2.797202797202797",
"": "2.797202797202797",
"Costin": "2.097902097902098",
"aircraft": "2.097902097902098",
"signals": "2.097902097902098",
"control": "2.097902097902098",
"system": "2.097902097902098",
"there": "1.3986013986013985"
}
}
}
I am trying to write a query to search does this index have the keyword flights (which it does) but I am having difficulty
Its straightforward running a match query on one of the other fields like text but encountering problems when trying to do the same or similar for keywords
Is there a way of performing this search with the current setup or should I add the keywords in differently?


If I understood you correctly, you would like to find all records that have the field keyword.flights and the value of this field is not important. You can do it using string query:
curl "http://localhost:9200/_search?q=keywords.flights:*"
Or using the exist filter:
curl "http://localhost:9200/_search" -d '{
"query": {
"constant_score" : {
"filter" : {
"exists" : { "field" : "keywords.flights" }
}
}
}
}'

Related

Elasticsearch query nested object

I have this record in elastic:
{
"FirstName": "Winona",
"LastName": "Ryder",
"Notes": "<p>she is an actress</p>",
"Age": "40-50",
"Race": "Caucasian",
"Gender": "Female",
"HeightApproximation": "No",
"Armed": false,
"AgeCategory": "Adult",
"ContactInfo": [
{
"ContactPoint": "stranger#gmail.com",
"ContactType": "Email",
"Details": "Details of tv show",
}
]
}
I want to query inside the contact info object and I used the query below but I dont get any result back:
{
"query": {
"nested" : {
"path" : "ContactInfo",
"query" : {
"match" : {"ContactInfo.Details" : "Details of tv show"}
}
}
}
}
I also tried:
{
"query": {
"term" : { "ContactInfo.ContactType" : "email" }
}
}
here is the mapping for contact info:
"ContactInfo":{
"type": "object"
}
I think I know the issue which is the field is not set as nested in mapping, is there a way to still query nested without changing the mapping, I just want to avoid re-indexing data if its possible.
I'm pretty new to elastic search so need your help.
Thanks in advance.

Elasticsearch has no concept of inner objects.
Some important points from Elasticsearch official documentation on Nested field type
The nested type is a specialized version of the object data type that allows arrays of objects to be indexed in a way that they can be queried independently of each other.
If you need to index arrays of objects and to maintain the independence of each object in the array, use the nested datatype instead of the object data type.
Internally, nested objects index each object in the array as a separate hidden document, such that that each nested object can be queried independently of the others with the nested query.
Refer to this SO answer, to get more details on this
Adding a working example with index mapping, search query, and search result
You have to reindex your data, after applying nested data type
Index Mapping:
{
"mappings": {
"properties": {
"ContactInfo": {
"type": "nested"
}
}
}
}
Search Query:
{
"query": {
"nested" : {
"path" : "ContactInfo",
"query" : {
"match" : {"ContactInfo.Details" : "Details of tv show"}
}
}
}
}
Search Result:
"hits": [
{
"_index": "stof_64269180",
"_type": "_doc",
"_id": "1",
"_score": 1.1507283,
"_source": {
"FirstName": "Winona",
"LastName": "Ryder",
"Notes": "<p>she is an actress</p>",
"Age": "40-50",
"Race": "Caucasian",
"Gender": "Female",
"HeightApproximation": "No",
"Armed": false,
"AgeCategory": "Adult",
"ContactInfo": [
{
"ContactPoint": "stranger#gmail.com",
"ContactType": "Email",
"Details": "Details of tv show"
}
]
}
}
]

SuggestionBuilder with BoolQueryBuilder in Elasticsearch

I am currently using BoolQueryBuilder to build a text search. I am having an issue with wrong spellings. When someone searches for a "chiar" instead of "chair" I have to show them some suggestions.
I have gone through the documentation and observed that the SuggestionBuilder is useful to get the suggestions.
Can I send all the requests in a single query, so that I can show the suggestions if the result is zero?

No need to send different search terms ie chair, chiar to get suggestions, it's not efficient and performant and you don't know all the combinations which user might misspell.
Instead, Use the fuzzy query or fuzziness param in the match query itself, which can be used in the bool query.
Let me show you an example, using the match query with the fuzziness parameter.
index def
{
"mappings": {
"properties": {
"product": {
"type": "text"
}
}
}
}
Index sample doc
{
"product" : "chair"
}
Search query with wrong term chiar
{
"query": {
"match" : {
"product" : {
"query" : "chiar",
"fuzziness" : "4" --> control it according to your application
}
}
}
}
Search result
"hits": [
{
"_index": "so_fuzzy",
"_type": "_doc",
"_id": "1",
"_score": 0.23014566,
"_source": {
"product": "chair"
}
}

how to properly use wildcard on a elastic search find?

I just jumped into ES and I dont have a lot of experience on this, so might be something I am missing on this.
I found this documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html that basically explains how to do a wildcard search.
I am trying to look for all messages inside my document that have certain patter.
So, using Kibana Sense (Elastic search query UI)I did this:
GET _search
{
"query": {
"wildcard" : {
"model.message": "my*"
}
}
}
with this I am trying to obtain all the messages that start with "my"
But I get no results...
Here is a copy of my document structure (or at least the first lines...)
"_index": "my_index",
"_type": "my_type",
"_id": "123456",
"_source": {
"model": {
"id": "123456",
"message": "my message",
Any idea what could be wrong?

Your sample document actually contains the model.content.message field but not the model.message field, so the following query should work:
GET _search
{
"query": {
"wildcard" : {
"model.content.message": "my*"
}
}
}

Can you share your mapping? It looks like you need to use a nested query:
GET /_search
{
"query": {
"nested" : {
"path" : "model",
"score_mode" : "avg",
"query" : {
"wildcard" : {
"model.message": "my*"
}
}
}
}
}
You can read more about nested queries here.

How to concatenate a string on elastic search

How to concatenate a string on elastic search.
for eg: here dasboradList.views has appended to new fields.
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "35",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
},
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "30",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
}
Above code specifies Elastic search index.
we want to append new field in a dasboradList.dasboradList has string type.
Needed json structure is..
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "35",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",`\"id\":\"name+"_"+userId\",\"createdDate\":\"01-01-2016\",\"expirydays\":\"10\"`,\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
},
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "30",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",`\"id\":\"name+"_"+userId\",\"createdDate\":\"01-01-2016\",\"expirydays\":\"10\"`,\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
}

In addition to what #jhilden said , we can indeed update an specific field in a ES document. But you need to enable scripting first.
Directly from the documentation :
#Index a document
curl -XPUT localhost:9200/test/type1/1 -d '{
"counter" : 1,
"tags" : ["red"]
}'
#Increase the count using inline scripting
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : {
"inline": "ctx._source.counter += count",
"params" : {
"count" : 4
}
}
}'
#Add a new field
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : "ctx._source.name_of_new_field = \"value_of_new_field\""
}'
You can also update by query in case that you don't know the id of the document or if you want to do a bulk update.
POST /twitter/_update_by_query
{
"script": {
"inline": "ctx._source.likes++"
},
"query": {
"term": {
"user": "kimchy"
}
}
}
More details of both concepts:
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-update.html
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-update-by-query.html
More information about inline scripting :
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-scripting.html

If I understand your problem correctly you want to UPDATE a record in ElasticSearch. There is no way in ES to do a partial update. What I mean is, there is no equivalent to this:
UPDATE tbl1
SET col1 = 'I am updating only 1 column'
WHERE id = 123
In ElasticSaerch we update a record by:
GET the record you are looking for
update the record
POST the FULL, updated, record back to ElasticSearch specifying the existing _id field.
This will overwrite the old record, something you can verify by looking at the _version property.

elasticsearch - add custom field to a specific index

I hava the JSON for my index that looks like this:
{
"_index": "myindes",
"_type": "external",
"_id": "1",
"_source": {
"id": "1",
"name": "myName",
"description": "myDescription",
"source": "mySource",
}
}
And i want to add a string field in _source named topic
How can i do

You can update the index mapping as
curl -XPUT 'http://localhost:9200/myindex/_mapping/external' -d '
{
"external" : {
"properties" : {
"id": {"type":"string"},
"name": {"type":"string"},
"description": {"type":"string"},
"source": {"type":"string"},
"topic":{"type":"string"} // <---new field
}
}
}'
Although the above step was not necessary but always good to control what you are indexing.
Now, you can index your documents with the new field and it will reflect in new updates. However, old indexed documents will still not contain this new field. You will have to reindex them.

Resources