Get elasticsearch result based on two keys - elasticsearch
I want to get all docs who's "PayerAccountId" should equal to "123" and "UsageStartDate" should be in range [2015-05-01 TO 2015-05-10]
I am expecting something to run like this,
curl -X GET http://192.168.1.3:9200/_all/_search -d '{"query" : {"match" : { "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-01-10]" }}}'
Obviously it's not working any suggestions
Here is my one doc
{
"_index": "logstash-2015.04.01",
"_type": "sam_billing_hourly",
"_id": "ef06aeb5fbab7191a43335740779fc73b667ff0b",
"_score": 1.0,
"_source": {
"#version": "1",
"#timestamp": "2015-04-01T02:00:00.000Z",
"Operation": "A",
"PayerAccountId": "156023466485",
"PricingPlanId": 482457,
"RateId": 3035133,
"RecordType": "LineItem",
"UnBlendedCost": 4.0e-07,
"UnBlendedRate": 4.0e-07,
"UsageEndDate": "2015-04-01T03:00:00Z",
"UsageQuantity": 1,
"UsageStartDate": "2015-04-01T02:00:00Z",
"UsageType": "DNS-Queries",
"fingerprint": "ef06aeb5fbab7191a43335740779fc73b667ff0b"
}
You need a query_string not a match:
{
"query": {
"query_string": {
"query": "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-10-01]"
}
}
}
Related
How can I influence Elasticsearch scoring by using higher score results informations?
I am upgrading my Elasticsearch server from version 1.6.0 to 7.12.1, which made me rewrite every query I had.
Those queries retrieves materials identified by 3 field : nature.idCat, nature.idNat and marque.idMrq (category ID, nature ID and brand ID).
I have a searching field on my application to search for specific materials, so if the user enter "photoc", the query sent to my Elasticsearch server looks like this :
{
"sort": [
"_score"
],
"query": {
"bool": {
"must": [
{
"query_string": {
"default_field": "search",
"query": "*photoc*",
"boost": 10
}
},
[...] // Some more irrelevant conditions for this question like
// if nature.idCat = 26 then idNat must be in some range and idMrq in some other range
]
}
}
}
And 2 examples of "hits" results of this query :
"hits": [
{
"_index": "ref_biens",
"_type": "_doc",
"_id": "T3RrpXsBz_TibRxz0akC",
"_score": 13.0,
"_source": {
"search": "Photocopieur GENERIQUE",
"nature": {
"idCat": 26,
"idNat": 665,
"libelle": "Photocopieur",
"ekip": "U03C",
"codeINSEE": 300121,
"noteMaterielArrondi": 5
},
"marque": {
"idMrq": 16,
"libelle": "GENERIQUE",
"ekip": "Z999",
"idVRDuree": 808
}
}
},
{
"_index": "ref_biens",
"_type": "_doc",
"_id": "UHRrpXsBz_TibRxz0akC",
"_score": 13.0,
"_source": {
"search": "Photocopieur INFOTEC",
"nature": {
"idCat": 26,
"idNat": 665,
"libelle": "Photocopieur",
"ekip": "U03C",
"codeINSEE": 300121,
"noteMaterielArrondi": 5
},
"marque": {
"idMrq": 1244,
"libelle": "INFOTEC",
"ekip": "I091",
"idVRDuree": 808
}
}
}
]
This works perfectly !
My problem appears when the user types more than one word, for example if he is searching specifically for the "Photocopieur PANASONIC", the results of the query shows the right material as the first result with a _score of 23 but then every other match has the same _score of 13 which can bring some totally different material as the next results (matching only on the brand name for example) even though I whish for other "Photocopieur" to be displayed first.
The way I'm thinking of doing it is by adding "score points" to results that have the most similarities to the best match, for instance I would add a 6 point boost for the same nature.idCat, 4 points for the same nature.idNat and finally 2 points for the same marque.idMrq.
Any idea on how I can achieve that ? Is this the correct approach to my problem ?
Find most similar documents in Elasticsearch
How do I find the top 100 most similar documents between two indices in Elasticsearch? Document #1 is in index1, type11, field111. Document #2 is in index2, type21, field211 Edit: Both fields are strings. I looked at the documentation for More Like This query. But it doesn't tell me how I can quickly compare the results for different kinds of similarity metrics and look at the top results.
Try this query, but substitute the id values for your documents:
GET index1,index2/_search
{
"query": {
"more_like_this": {
"fields": [
"field111",
"field211"
],
"like": [
{
"_index": "index1",
"_id": "DOC_1_ID"
},
{
"_index": "index2",
"_id": "DOC_2_ID"
}
],
"min_term_freq": 1,
"max_query_terms": 12
}
}
}
Elasticsearch match phrase prefix not matching all terms
I am having an issue where when I use the match_phrase_prefix query in Elasticsearch, it is not returning all the results I would expect it to, particularly when the query is one word followed by one letter.
Take this index mapping (this is a contrived example to protect sensitive data):
http://localhost:9200/test/drinks/_mapping
returns:
{
"test": {
"mappings": {
"drinks": {
"properties": {
"name": {
"type": "text"
}
}
}
}
}
}
And amongst millions of other records are these:
{
"_index": "test",
"_type": "drinks",
"_id": "2",
"_score": 1,
"_source": {
"name": "Johnnie Walker Black Label"
}
},
{
"_index": "test",
"_type": "drinks",
"_id": "1",
"_score": 1,
"_source": {
"name": "Johnnie Walker Blue Label"
}
}
The following query, which is one word followed by two letters:
POST http://localhost:9200/test/drinks/_search
{
"query": {
"match_phrase_prefix" : {
"name" : "Walker Bl"
}
}
}
returns this:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.5753642,
"hits": [
{
"_index": "test",
"_type": "drinks",
"_id": "2",
"_score": 0.5753642,
"_source": {
"name": "Johnnie Walker Black Label"
}
},
{
"_index": "test",
"_type": "drinks",
"_id": "1",
"_score": 0.5753642,
"_source": {
"name": "Johnnie Walker Blue Label"
}
}
]
}
}
Whereas this query with one word and one letter:
POST http://localhost:9200/test/drinks/_search
{
"query": {
"match_phrase_prefix" : {
"name" : "Walker B"
}
}
}
returns no results. What could be happening here?
I will assume that you are working with Elasticsearch 5.0 and above. I think it might have to be because of the max_expansions default value. As seen in the documentation here, the max_expansions parameters is used to control how many prefixes the last term will be expanded with. The default value is 50 and it might explain why you find "black" and "blue" with the two first letters B and L, but not with the B only. The documentation is pretty clear about it: The match_phrase_prefix query is a poor-man’s autocomplete. It is very easy to use, which let’s you get started quickly with search-as-you-type but it’s results, which usually are good enough, can sometimes be confusing. Consider the query string quick brown f. This query works by creating a phrase query out of quick and brown (i.e. the term quick must exist and must be followed by the term brown). Then it looks at the sorted term dictionary to find the first 50 terms that begin with f, and adds these terms to the phrase query. The problem is that the first 50 terms may not include the term fox so the phase quick brown fox will not be found. This usually isn’t a problem as the user will continue to type more letters until the word they are looking for appears I wouldn't be able to tell you if it's ok to increase this parameter above 50 if you are looking for good performances since I never tried myself.
How to concatenate a string on elastic search
How to concatenate a string on elastic search.
for eg: here dasboradList.views has appended to new fields.
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "35",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
},
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "30",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
}
Above code specifies Elastic search index.
we want to append new field in a dasboradList.dasboradList has string type.
Needed json structure is..
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "35",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",`\"id\":\"name+"_"+userId\",\"createdDate\":\"01-01-2016\",\"expirydays\":\"10\"`,\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
},
{
"_index": "haysbisuitedev",
"_type": "dasboardconfig",
"_id": "30",
"_version": 3,
"found": true,
"_source": {
"userId": 35,
"defaultDashBoard": "testsgare",
"dasboradList": "[{\"Ids\":2,\"views\":[{\"name\":\"test\",`\"id\":\"name+"_"+userId\",\"createdDate\":\"01-01-2016\",\"expirydays\":\"10\"`,\"defaultView\":true,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_23663\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":\"#6FAA87\",\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0},{\"name\":\"check\",\"defaultView\":false,\"layout\":{\"templateType\":\"1\",\"backgroundColor\":\"#DBE3F5\",\"lets\":[{\"id\":\"let_54316\",\"type\":\"\",\"rowNo\":\"0\",\"columnNo\":\"0\",\"colspan\":\"1\",\"rowspan\":\"1\",\"title\":\"\",\"dashlet\":\"\",\"bgColor\":\"\",\"width\":\"32%\",\"height\":\"27%\",\"name\":null,\"catalogId\":\"0\",\"dashletId\":\"0\",\"param\":{\"misID\":null,\"name\":null,\"graphType\":null},\"widget\":{\"headline1\":\"\",\"headline2\":\"\",\"percentage\":\"0\",\"enableWidget\":false,\"hoverOnDashelt\":false,\"chartType\":\"\",\"head1Color\":\"\",\"head2Color\":\"\",\"percentageColor\":\"\"},\"clipHeadline\":false}],\"shared\":false},\"background\":null,\"share\":null,\"comments\":null,\"shareable\":false,\"userId\":0}]}]"
}
}
In addition to what #jhilden said , we can indeed update an specific field in a ES document. But you need to enable scripting first.
Directly from the documentation :
#Index a document
curl -XPUT localhost:9200/test/type1/1 -d '{
"counter" : 1,
"tags" : ["red"]
}'
#Increase the count using inline scripting
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : {
"inline": "ctx._source.counter += count",
"params" : {
"count" : 4
}
}
}'
#Add a new field
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : "ctx._source.name_of_new_field = \"value_of_new_field\""
}'
You can also update by query in case that you don't know the id of the document or if you want to do a bulk update.
POST /twitter/_update_by_query
{
"script": {
"inline": "ctx._source.likes++"
},
"query": {
"term": {
"user": "kimchy"
}
}
}
More details of both concepts:
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-update.html
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-update-by-query.html
More information about inline scripting :
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-scripting.html
If I understand your problem correctly you want to UPDATE a record in ElasticSearch. There is no way in ES to do a partial update. What I mean is, there is no equivalent to this: UPDATE tbl1 SET col1 = 'I am updating only 1 column' WHERE id = 123 In ElasticSaerch we update a record by: GET the record you are looking for update the record POST the FULL, updated, record back to ElasticSearch specifying the existing _id field. This will overwrite the old record, something you can verify by looking at the _version property.
Elastic Search - Querying on values
I have an elasticsearch index with the following values
{
"_index": "article",
"_type": "articleId",
"_id": "10970",
"_score": 1,
"_source": {
"url": "http%3A%2F%2Fwww.tomshardware.com%2Fnews%2FAir-Traffic-Software-DoS-Attacks%2C16471.html%23xtor%3DRSS-181",
"title": "Air%20Traffic%20Software%20Vulnerable%20to%20DoS%20Attacks",
"publicationId": "888",
"text": "%20%3Cp%3E%3Cstrong%3EA%20security%20researcher%20revealed%20a%20flaw%20in%20commonly%20used%20air%20traffic%20control%20software%20that%20would%20allow%20an%20attacker%20to%20create%20an%20unlimited%20number%20of%20phantom%20flights.%3C%2Fstrong%3E%3C%2Fp%3E%20%3Cp%3E%3Ca%20target%3D%22_blank%22%3E%3C%2Fa%3E%3C%2Fp%3E%20%3Cp%3EAccording%20to%20Andrei%20Costin%2C%20%242%2C000%20in%20equipment%20and%20%22modest%20tech%20skills%22%20are%20enough%20to%20throw%20an%20air%20traffic%20control%20system%20of%20virtually%20any%20airport%20into%20complete%20disarray.%20The%20ADS-B%20system%20that%20is%20used%20across%20the%20world%20is%20vulnerable%20as%20it%20does%20not%20verify%20that%20incoming%20traffic%20signals%20as%20genuine.%20%3C%2Fp%3E%20%3Cp%3ECostin%20says%20that%20a%20hacker%20could%20inject%20flights%20that%20do%20not%20exist%20and%20could%20confuse%20an%20air%20controller%20station.%20Air%20controllers%20could%20cross-check%20flights%20with%20flight%20schedules%2C%20but%20if%20the%20number%20of%20phantom%20flights%20is%20high%20enough%2C%20there%20is%20no%20way%20that%20cross-checks%20would%20work.%20Consider%20it%20like%20an%20DoS%20attack%20on%20an%20air%20traffic%20control%20system.%3C%2Fp%3E%20%3Cp%3ECostin%20noted%20that%20rogue%20signals%20from%20the%20ground%20can%20be%20generally%20identified%20and%20ruled%20out%20as%20malicious%20signals%2C%20but%20there%20is%20no%20way%20to%20do%20the%20same%20for%20robotic%20aircraft%2C%20for%20example.%20He%20also%20noted%20that%20data%20sent%20from%20airplanes%20to%20air%20traffic%20controllers%20is%20unencrypted%20and%20can%20be%20captured%20by%20unidentified%20sources.%20Since%20this%20applies%20to%20any%20aircraft%2C%20it%20is%20in%20theory%20possible%20to%20deploy%20airplane%20tracking%20devices%20to%20track%20specific%20aircraft.%3C%2Fp%3E%20%3C%2Fp%3E%3Cp%3E%20%3Cp%3E%3Ca%20target%3D%22_blank%22%20href%3D%22mailto%3Anews-us%40bestofmedia.com%3Fsubject%3DNews%2520Article%2520Feedback%22%3E%3Cem%3E%3Csub%3EContact%20Us%20for%20News%20Tips%2C%20Corrections%20and%20Feedback%3C%2Fsub%3E%3C%2Fem%3E%3C%2Fa%3E%3C%2Fp%3E",
"keywords": {
"air": "3.4965034965034962",
"traffic": "3.4965034965034962",
"flights": "2.797202797202797",
"": "2.797202797202797",
"Costin": "2.097902097902098",
"aircraft": "2.097902097902098",
"signals": "2.097902097902098",
"control": "2.097902097902098",
"system": "2.097902097902098",
"there": "1.3986013986013985"
}
}
}
I am trying to write a query to search does this index have the keyword flights (which it does) but I am having difficulty
Its straightforward running a match query on one of the other fields like text but encountering problems when trying to do the same or similar for keywords
Is there a way of performing this search with the current setup or should I add the keywords in differently?
If I understood you correctly, you would like to find all records that have the field keyword.flights and the value of this field is not important. You can do it using string query:
curl "http://localhost:9200/_search?q=keywords.flights:*"
Or using the exist filter:
curl "http://localhost:9200/_search" -d '{
"query": {
"constant_score" : {
"filter" : {
"exists" : { "field" : "keywords.flights" }
}
}
}
}'