Search by exact match in all fields in Elasticsearch - elasticsearch

Let's say I have 3 documents, each of them only contains one field (but let's imagine that there are more, and we need to search through all fields).
Field value is "first second"
Field value is "second first"
Field value is "first second third"
Here is a script that can be used to create these 3 documents:
# drop the index completely, use with care!
curl -iX DELETE 'http://localhost:9200/test'
curl -H 'content-type: application/json' -iX PUT 'http://localhost:9200/test/_doc/one' -d '{"name":"first second"}'
curl -H 'content-type: application/json' -iX PUT 'http://localhost:9200/test/_doc/two' -d '{"name":"second first"}'
curl -H 'content-type: application/json' -iX PUT 'http://localhost:9200/test/_doc/three' -d '{"name":"first second third"}'
I need to find the only document (document 1) that has exactly "first second" text in one of its fields.
Here is what I tried.
A. Plain search:
curl -H 'Content-Type: application/json' -iX POST 'http://localhost:9200/test/_search' -d '{
"query": {
"query_string": {
"query": "first second"
}
}
}'
returns all 3 documents
B. Quoting
curl -H 'Content-Type: application/json' -iX POST 'http://localhost:9200/test/_search' -d '{
"query": {
"query_string": {
"query": "\"first second\""
}
}
}'
gives 2 documents: 1 and 3, because both contain 'first second'.
Here https://stackoverflow.com/a/28024714/7637120 they suggest to use 'keyword' analyzer to analyze the fields when indexing, but I would like to avoid any customizations to the mapping.
Is it possible to avoid them and still only find document 1?

Yes, you can do that by declaring name mapping type as keyword. The key to solve your problem is just simple -- declare name mapping type:keyword and off you go
to demonstrate it, I have done these
1) created mapping with `keyword` for `name` field`
2) indexed the three documents
3) searched with a `match` query
mappings
PUT so_test16
{
"mappings": {
"_doc":{
"properties":{
"name": {
"type": "keyword"
}
}
}
}
}
Indexing the documents
POST /so_test16/_doc
{
"id": 1,
"name": "first second"
}
POST /so_test16/_doc
{
"id": 2,
"name": "second first"
}
POST /so_test16/_doc
{
"id": 3,
"name": "first second third"
}
The query
GET /so_test16/_search
{
"query": {
"match": {"name": "first second"}
}
}
and the result
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.2876821,
"hits" : [
{
"_index" : "so_test16",
"_type" : "_doc",
"_id" : "m1KXx2sB4TH56W1hdTF9",
"_score" : 0.2876821,
"_source" : {
"id" : 1,
"name" : "first second"
}
}
]
}
}
Adding second solution
( if the name is not a keyword type but a text type. Only thing here is fielddata:true also needed to be added for name field)
Mappings
PUT so_test18
{
"mappings" : {
"_doc" : {
"properties" : {
"id" : {
"type" : "long"
},
"name" : {
"type" : "text",
"fielddata": true
}
}
}
}
}
and the search query
GET /so_test18/_search
{
"query": {
"bool": {
"must": [
{"match_phrase": {"name": "first second"}}
],
"filter": {
"script": {
"script": {
"lang": "painless",
"source": "doc['name'].values.length == 2"
}
}
}
}
}
}
and the response
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.3971361,
"hits" : [
{
"_index" : "so_test18",
"_type" : "_doc",
"_id" : "o1JryGsB4TH56W1hhzGT",
"_score" : 0.3971361,
"_source" : {
"id" : 1,
"name" : "first second"
}
}
]
}
}

In Elasticsearch 7.1.0, it seems that you can use keyword analyzer even without creating a special mapping. At least I didn't, and the following query does what I need:
curl -H 'Content-Type: application/json' -iX POST 'http://localhost:9200/test/_search' -d '{
"query": {
"query_string": {
"query": "first second",
"analyzer": "keyword"
}
}
}'

Related

Update elastic search nested field based on query

I have an Elasticsearch index named pollstat with mapping as follows:
{
"pollstat" : {
"mappings" : {
"dynamic" : "false",
"properties" : {
"dt" : {
"properties" : {
"dte" : {
"type" : "date"
},
"is_polled" : {
"type" : "boolean"
}
}
},
"is_profiled" : {
"type" : "boolean"
},
"maid" : {
"type" : "keyword"
}
}
}
}
}
The above index is created using:
curl -XPUT "http://localhost:9200/pollstat" -H 'Content-Type: application/json' -d'
{
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1
},
"mappings": {
"properties": {
"maid" : {
"type" : "keyword"
},
"dt" : {
"type" : "object",
"properties": {
"dte" : {"type":"date"},
"is_polled" : {"type":"boolean"}
}
},
"is_profiled" : {
"type" : "boolean"
}
},
"dynamic":false
}
}'
To add data into this index, I am using the following code:
curl -X POST "localhost:9200/pollstat/_doc/?pretty" -H 'Content-Type: application/json' -d'{"maid" : "fans", "dt" : [{"dte": "2022-03-19", "is_polled":true } ], "is_profiled":true } '
This is working.
The requirement is to append the dt field when a particular maid polls data on a specific date. In this case, if the maid fans polls data for another day, I want to append the same to the dt field.
I used the following code, which takes the document id to update the document.
curl -X POST "localhost:9200/pollstat/_doc/hQh4oH8BPfXX63hBUbPN/_update?pretty" -H 'Content-Type: application/json' -d'{"script": {"source": "ctx._source.dt.addAll(params.dt)", "params": {"dt": [{ "dte": "2019-07-16", "is_polled": true }, { "dte": "2019-07-17", "is_polled": false } ] } } } '
This is also working
However, my application does not have visibility to the document id but gets the maid. The maid is also as unique as the document id. Hence to update a specific maid, I was trying to do the same with a query on maid.
I used the following code:
curl -X POST "localhost:9200/pollstat/_update_by_query?pretty" -H 'Content-Type: application/json' -d'"query": {"match": { "maid": "fans" }, "script": {"source": "ctx._source.dt.addAll(params.dt)", "params": {"dt": [{ "dte": "2019-07-18", "is_polled": true }, { "dte": "2019-07-19", "is_polled": false } ] } } }'
This code executes without an error and I am getting the following update status as well:
{
"took" : 8,
"timed_out" : false,
"total" : 1,
"updated" : 1,
"deleted" : 0,
"batches" : 1,
"version_conflicts" : 0,
"noops" : 0,
"retries" : {
"bulk" : 0,
"search" : 0
},
"throttled_millis" : 0,
"requests_per_second" : -1.0,
"throttled_until_millis" : 0,
"failures" : [ ]
}
However my index is not getting updated.
Since the maid field has type keyword, I had to use the query->term instead of query->match. The final query is as follows:
curl -X POST "localhost:9200/pollstat/_update_by_query?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"term": { "maid": "fans" }},
"script": {
"source": "ctx._source.dt.addAll(params.dt)",
"params": {
"dt": [
{ "dte": "2019-07-18", "is_polled": true },
{ "dte": "2019-07-19", "is_polled": false }
]
}
}
}
'
Posting this answer for others reference.

How to order search results with related to slop in elasticsearch?

I have an index in ES:
curl -XGET 'http://127.0.0.1:9200/so/_settings?pretty=true'
{
"so" : {
"settings" : {
"index" : {
"number_of_shards" : "1",
"provided_name" : "so",
"creation_date" : "1594912442805",
"analysis" : {
"analyzer" : {
"my_simple_analyzer" : {
"type" : "simple",
"tokenizer" : "lowercase"
}
}
},
"number_of_replicas" : "1",
"uuid" : "8YVu4zU_Sdylr3KhOIwu9Q",
"version" : {
"created" : "7080099"
}
}
}
}
}
It has around 1.5M data.
curl -XGET 'http://127.0.0.1:9200/so/_count?pretty=true'
{
"count" : 15426942,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
}
}
I wanted to perform a full text search, so that the query string first does the phrase match and then followed by results which has slop of 1, then slop of 2 and so on.
So I came up with the below query for the same:
curl -XGET 'http://127.0.0.1:9200/so/_search?pretty=true' -H 'Content-Type: application/json' -d '{
"query": {
"bool": {
"should": [
{
"match_phrase": {
"posts": {
"query": "get the scanner on a specific family like this",
"_name": "exact_match"
}
}
},
{
"match": {
"posts": {
"query": "get the scanner on a specific family like this",
"_name": "partial_match"
}
}
}
]
}
}
}'
Is this the correct query? Because I do see the partial_match doesnt sort from slop of distance 1 and so on. How to achieve it?

Elasticsearch - different searchstring for different fields

is it possible to query different fields with different searchstrings in one query?
For example:
{
"query": [
{ "match": { "name": "Bob" }},
{ "match": { "title": "Awesome Title" }}]
}
where "name" and "title" are fields of documents.
I know there's a multi_match query, but there I query a list of fields all with the same string...
You can try this query to search for documents that match both the conditions ("name": "Bob" and "title": "Awesome Title"). Replace the <index_name> with the name of the index.
$ curl -XGET 'localhost:9200/<index_name>/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"should": [
{
"match_phrase": {
"name": "Bob"
}
},
{
"match_phrase": {
"title": "Awesome Title"
}
}
],
"minimum_should_match": 2
}
}
}
'
Illustration:
(a) Index 4 documents
# Doc 1 - Only "name" matches
$ curl -X PUT "localhost:9200/office/doc/o001" -H 'Content-Type: application/json' -d'
{
"name" : "Bob",
"title" : "Senior Staff",
"description" : "Developing new products"
}
'
# Doc 2 - None of the criteria match
$ curl -X PUT "localhost:9200/office/doc/o002" -H 'Content-Type: application/json' -d'
{
"name" : "Tom",
"title" : "Marketing Manager",
"description" : "Shows and events"
}
'
# Doc 3 - Only "title" matches
$ curl -X PUT "localhost:9200/office/doc/o003" -H 'Content-Type: application/json' -d'
{
"name" : "Liz",
"title" : "Awesome Title",
"description" : "Recruiting talent"
}
'
# Doc 4 - Both "name" and "title" match - Expected in result
$ curl -X PUT "localhost:9200/office/doc/o004" -H 'Content-Type: application/json' -d'
{
"name" : "Bob",
"title" : "Awesome Title",
"description" : "Finance Operations"
}
'
(b) Verify that the documents were indexed
$ curl 'localhost:9200/office/_search?q=*'
# Output
{"took":19,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":4,"max_score":1.0,"hits":[{"_index":"office","_type":"doc","_id":"o003","_score":1.0,"_source":
{
"name" : "Liz",
"title" : "Awesome Title",
"description" : "Recruiting talent"
}
},{"_index":"office","_type":"doc","_id":"o001","_score":1.0,"_source":
{
"name" : "Bob",
"title" : "Senior Staff",
"description" : "Developing new products"
}
},{"_index":"office","_type":"doc","_id":"o004","_score":1.0,"_source":
{
"name" : "Bob",
"title" : "Awesome Title",
"description" : "Finance Operations"
}
},{"_index":"office","_type":"doc","_id":"o002","_score":1.0,"_source":
{
"name" : "Tom",
"title" : "Marketing Manager",
"description" : "Shows and events"
}
(c) Run the query. Result is the one doc (with id=o004) that matches both criteria:
$ curl -XGET 'localhost:9200/office/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"should": [
{
"match_phrase": {
"name": "Bob"
}
},
{
"match_phrase": {
"title": "Awesome Title"
}
}
],
"minimum_should_match": 2
}
}
}
'
(d) Get the query result
{
"took" : 27,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.4261419,
"hits" : [
{
"_index" : "office",
"_type" : "doc",
"_id" : "o004",
"_score" : 1.4261419,
"_source" : {
"name" : "Bob",
"title" : "Awesome Title",
"description" : "Finance Operations"
}
}
]
}
}

Ranged Query with ElasticSearch

I'm testing with ElasticSearch and I'm having problems with ranged queries.
Consider the following document that I've inserted:
curl -XPUT 'localhost:9200/test/test/test?pretty' -d '
{
"name": "John Doe",
"duration" : "10",
"state" : "unknown"
}'
And now I'me trying to do a ranged query that catches all documents whose duration is between 5 and 15:
curl -XPOST 'localhost:9200/test/_search?pretty' -d '
{
"query": {
"range": {
"duration": {
"gte": "5",
"lte": "15"
}
}
}
}'
This returns no hits however if I run the Query like this:
curl -XPOST 'localhost:9200/test/_search?pretty' -d '
{
"query": {
"range": {
"duration": {
"gte": "10"
}
}
}
}'
It returns the Document I've inserted earlier. How can I query ElasticSearch for documents with the duration value between 5 and 15.
The problem is that you are indexing your values as strings. This causes the range query not to work. Try indexing and querying as follows:
curl -XPUT 'localhost:9200/test/test/test?pretty' -d '
{
"name": "John Doe",
"duration" : 10,
"state" : "unknown"
}'
curl -XPOST 'localhost:9200/test/_search?pretty' -d '
{
"query": {
"range": {
"duration": {
"gte": 5,
"lte": 15
}
}
}
}'
This wil yield the following result:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "test",
"_type" : "test",
"_id" : "test",
"_score" : 1.0,
"_source":
{
"name": "John Doe",
"duration" : 10,
"state" : "unknown"
}
} ]
}
}

ElasticSearch - searching different doc_types with the same field name but different analyzers

Let's say I make a simple ElasticSearch index:
curl -XPUT 'http://localhost:9200/test/' -d '{
"settings": {
"analysis": {
"char_filter": {
"de_acronym": {
"type": "mapping",
"mappings": [".=>"]
}
},
"analyzer": {
"analyzer1": {
"type": "custom",
"tokenizer": "keyword",
"char_filter": ["de_acronym"]
}
}
}
}
}'
And I make two doc_types that have the same property name but they are analyzed slightly differently from one another:
curl -XPUT 'http://localhost:9200/test/_mapping/docA' -d '{
"docA": {
"properties": {
"name": {
"type": "string",
"analyzer": "simple"
}
}
}
}'
curl -XPUT 'http://localhost:9200/test/_mapping/docB' -d '{
"docB": {
"properties": {
"name": {
"type": "string",
"analyzer": "analyzer1"
}
}
}
}'
Next, let's say I put a document in each doc_type with the same name:
curl -XPUT 'http://localhost:9200/test/docA/1' -d '{ "name" : "U.S. Army" }'
curl -XPUT 'http://localhost:9200/test/docB/1' -d '{ "name" : "U.S. Army" }'
Let's try to search for "U.S. Army" in both doc types at the same time:
curl -XGET 'http://localhost:9200/test/_search?pretty' -d '{
"query": {
"match_phrase": {
"name": {
"query": "U.S. Army"
}
}
}
}'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.5,
"hits" : [ {
"_index" : "test",
"_type" : "docA",
"_id" : "1",
"_score" : 1.5,
"_source":{ "name" : "U.S. Army" }
} ]
}
}
I only get one result! I get the other result when I specify docB's analyzer:
curl -XGET 'http://localhost:9200/test/_search?pretty' -d '
{
"query": {
"match_phrase": {
"name": {
"query": "U.S. Army",
"analyzer": "analyzer1"
}
}
}
}'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "test",
"_type" : "docB",
"_id" : "1",
"_score" : 1.0,
"_source":{ "name" : "U.S. Army" }
} ]
}
}
I was under the impression that ES would search each doc_type with the appropriate analyzer. Is there a way to do this?
The ElasticSearch docs say that precedence for search analyzer goes:
1) The analyzer defined in the query itself, else
2) The analyzer defined in the field mapping, else
...
In this case, is ElasticSearch arbitrarily choosing which field mapping to use?
Take a look at this issue in github, which seems to have started from this post in ES google groups. I believe it answers your question:
if its in a filtered query, we can't infer it, so we simply pick one of those and use its analysis settings

Resources