ElasticSearch - Match (email value) returns wrong registers - elasticsearch

I'm using match to search for a specific email but the result is wrong. The match property brings me results similar. If the result exists, the result displays on first lines but when the results not exists, it brings me result by same domain.
Here is my query:
{
"query": {
"match" : {
"email" : "placplac#xxx.net"
}
}
}
This email doesn't exist in my base but returning values like banana#xxx.net, ronyvon#xxx.net*, etc.
How can i force to return only if the value is equal from the query?
Thank in advance.

You need to put "index":"not_analyzed" on the "email" field. That way, the only terms that are queried against are the exact values that have been stored to that field (as opposed to the case with the standard analyzer, which is the default used if no analyzer is listed).
To illustrate, I set up a simple mapping with the email field not analyzed, and added two simple docs:
DELETE /test_index
PUT /test_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"doc": {
"properties": {
"email": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
PUT /test_index/doc/1
{"email": "placplac#xxx.net"}
PUT /test_index/doc/2
{"email": "placplac#nowhere.net"}
Now your match query will return only the document that matches the query exactly:
POST /test_index/_search
{
"query": {
"match" : {
"email" : "placplac#xxx.net"
}
}
}
...
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "1",
"_score": 1,
"_source": {
"email": "placplac#xxx.net"
}
}
]
}
}
Here is the code I used:
http://sense.qbox.io/gist/12763f63f2a75bf30ff956c25097b5955074508a
PS: What you actually probably want here is a term query or even term filter, since you don't want any analysis on the query text. So maybe something like:
POST /test_index/_search
{
"query": {
"constant_score": {
"filter": {
"term": {
"email": "placplac#xxx.net"
}
}
}
}
}

Related

Query on Elastic Search on multiple criterias

I have this document in elastic search
{
"_index" : "master",
"_type" : "_doc",
"_id" : "q9IGdXABeXa7ITflapkV",
"_score" : 0.0,
"_source" : {
"customer_acct" : "64876457056",
"ssn_number" : "123456789",
"name" : "Julie",
"city" : "NY"
}
I wanted to query the master index , with the customer_acct and ssn_number to retrive the entire document. I wanted to disable scoring and relevance , I have used the below query
curl -X GET "localhost/master/_search/?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"term": {
"customer_acct": {
"value":"64876457056"
}
}
}
}'
I need to include the second criteria in the term query as well which is the ssn_number, how would I do that? , I want to turn off scoring and relevance would that be possible, I am new to Elastic Search and how would I fit the second criteria on ssn_number in the above query that I have tried?
First, you need to define the proper mapping of your index. your customer_acct and ssn_number are of numeric type but you are storing it as a string. Also looking at your sample I can see you have to use long to store them. and then you can just use filter context in your query as you don't need score and relevance in your result. Read more about filter context in official ES doc as well as below snippet from the link.
In a filter context, a query clause answers the question “Does this
document match this query clause?” The answer is a simple Yes or
No — no scores are calculated. Filter context is mostly used for
filtering structured data,
which is exactly your use-case.
1. Index Mapping
{
"mappings": {
"properties": {
"customer_acct": {
"type": "long"
},
"ssn_number" :{
"type": "long"
},
"name" : {
"type": "text"
},
"city" :{
"type": "text"
}
}
}
}
2. Index sample docs
{
"name": "Smithe John",
"city": "SF",
"customer_acct": 64876457065,
"ssn_number": 123456790
}
{
"name": "Julie",
"city": "NY",
"customer_acct": 64876457056,
"ssn_number": 123456789
}
3. Main search query to filter without the score
{
"query": {
"bool": {
"filter": [ --> only filter clause
{
"term": {
"customer_acct": 64876457056
}
},
{
"term": {
"ssn_number": 123456789
}
}
]
}
}
}
Above search query gives below result:
{
"took": 186,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.0,
"hits": [
{
"_index": "so-master",
"_type": "_doc",
"_id": "1",
"_score": 0.0, --> notice score is 0.
"_source": {
"name": "Smithe John",
"city": "SF",
"customer_acct": 64876457056,
"ssn_number": 123456789
}
}
]
}
}

Elastic Search Case Insensitive query with prefix query

I am new to elastic search. I have below query
GET deals2/_search
{
"size": 200,
"_source": ["acquireInfo"],
"query": {
"bool": {
"must": [
{
"query_string": {
"fields": ["acquireInfo.company_name.keyword"],
"query": "az*"
}
}
]
}
}
}
Here I want Elastic should gives results like case insensitive Like string start with below like
"Az"
"AZ"
"az"
"aZ"
"Az"
But I am not getting all results like this way. So Anyone can please help me on that.
Example:- I have 4 documents
1)Aziia Avto Ust-Kamenogorsk OOO
2)AZ Infotech Inc
3)AZURE Midstream Partners LP
4)State Oil Fund of the Republic of Azerbaijan
Now searching on az , should return only first 3 docs as they start with az ignoring case here and not the 4th one, which also has az but not at the beginning.
This is happening as you are using the keyword field to index the company_name in your application.
The keyword analyzer is a “noop” analyzer which returns the entire input string as a single token for example, company name, consist of foo, Foo, fOo will be stored with case only and searching for foo, will only match foo as elastic search ultimately works on tokens match(which is case sensitive).
What you need is to use a standard analyzer or some other custom analyzer which solves your other use-cases as well and uses lowercase token filter on the field and use the match query which is analyzed, and uses the same analyzer which is used to index the field, this way your search query will generate the same tokens, which is stored in the index and your search will become case-insensitive.
Edit: Had a discussion with the user in chat and updating the answer to suit his requirements, which are below:-
Step 1:- Define settings and mapping for index.
Endpoint :- http://{{hostname}}:{{port}}/{{index}}
{
"settings": {
"analysis": {
"normalizer": {
"my_normalizer": {
"type": "custom",
"char_filter": [],
"filter": "lowercase"
}
}
}
},
"mappings": {
"properties": {
"company_name": {
"type": "keyword",
"normalizer": "my_normalizer"
}
}
}
}
Step 2: Index all the documents
Endpoint: http://{{hostname}}:{{port}}/{{index}}/_doc/ --> 1,2,3,4 etc
{
"company_name" : "State Oil Fund of the Republic of Azerbaijan"
}
Step3 :- Search query
Endpoint:- http://{{hostname}}:{{port}}/{{index}}/_search
{ "query": {
"prefix" : { "company_name" : "az" }
}
}
This would bring the below expected results:-
{
"took": 870,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "prerfixsearch",
"_type": "_doc",
"_id": "2ec9df0fc-dc04-47bb-914f-91a9f20d09efd15f2506-293f-4fb2-bdc3-925684a930b5",
"_score": 1,
"_source": {
"company_name": "AZ Infotech Inc"
}
},
{
"_index": "prerfixsearch",
"_type": "_doc",
"_id": "160d01183-a308-4408-8ac1-a85da950f285edefaca2-0b68-41c6-ba34-21bbef57f84f",
"_score": 1,
"_source": {
"company_name": "Aziia Avto Ust-Kamenogorsk OOO"
}
},
{
"_index": "prerfixsearch",
"_type": "_doc",
"_id": "1da878175-7db5-4332-baa7-ac47bd39b646f81c1770-7ae1-4536-baed-0a4f6b20fa38",
"_score": 1,
"_source": {
"company_name": "AZURE Midstream Partners LP"
}
}
]
}
}
Explanation:, As earlier OP didn;t mention the exclusion of 4th doc in the search result, that's the reason I suggested creating a text field, so that individuals tokens are generated but now as requirement is only the prefix search, we don't need the individual tokens and we would want only 1 token but it should be lowercased to support the case insensitive search, that's the reason I applied the custom normalizer on company_name field.

ElasticSearch Range query

I have created the index by using the following mapping:
put test1
{
"mappings": {
"type1": {
"properties": {
"age": {
"type": "text",
"fields": {
"raw": {
"type": "keyword",
"ignore_above": 32766
}
}
}
}
}
}
}
Added following documents into index:
PUT test1/type1/1/_create
{
"age":50
}
PUT test1/type1/2/_create
{
"age":100
}
PUT test1/type1/3/_create
{
"age":150
}
PUT test1/type1/4/_create
{
"age":200
}
I have used the following range query to fetch result:
GET test1/_search
{
"query": {
"range" : {
"age" : {
"lte" : 150
}
}
}
}
It is giving me the following response :
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test1",
"_type": "type1",
"_id": "2",
"_score": 1,
"_source": {
"age": 100
}
},
{
"_index": "test1",
"_type": "type1",
"_id": "3",
"_score": 1,
"_source": {
"age": 150
}
}
]
}
}
the above response not showing document having age is 50 it is showing only age is 100 and 150. As 50 also less than 200. What is wrong here?
Can anyone help me to get a valid result?
In my schema age field type text, I don't want to change it.
How can I get a valid result?
Because age field type is text, the range query is using alphabetically order. So the results are correct:
"100"<"150"
"150"="150"
"50">"150"
If you are ingesting only numbers in age field, you should change the age field type to number, or add another inner field as number, just you did with raw inner field.
UPDATE: Tested on local system and it is working.
NOTE: Ideally, you would want the mappings to be correct, but if there is no other choice and you are not the person to decide on the mapping then you can still achieve it by following.
For ES version 6.3 onwards, try this.
GET test1/type1/_search
{
"query": {
"bool" : {
"must" : {
"script" : {
"script" : {
"source": "Integer.parseInt(doc['age.raw'].value) <= 150",
"lang": "painless"
}
}
}
}
}
}
Sources to refer:
https://www.elastic.co/guide/en/elasticsearch/reference/6.3/query-dsl-script-query.html
https://discuss.elastic.co/t/painscript-script-cast-string-as-int/97034
Type for your field age in mapping is set to text. That is reason it is doing dictionary sorting where 50 > 150. Please use long data type. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

Latenise token on query time

I need to latenise the query tokens that I use when querying (or filtering). I can do this on application level, but I was wondering if elasticsearch provides an out of the box solution.
I'm using ES 1.7.5 (as a service)
By default elasticsearch will use the same analyzer at index time and query time but it is possible to specify a search_analyzer which will only be used at query time.
Let's take a look at the following example:
# First we define an analyzer which will fold non ascii characters called `latinize`.
PUT books
{
"settings": {
"analysis": {
"analyzer": {
"latinize": {
"tokenizer": "standard",
"filter": ["asciifolding"]
}
}
}
},
"mappings": {
"book": {
"properties": {
"name": {
"type": "string",
"analyzer": "standard", # We use the standard analyzer at index time.
"search_analyzer": "latinize" # But we use the latinize analyzer at query time.
}
}
}
}
}
# Now let's create a document and search for it with a non latinized string.
POST books/book
{
"name": "aaoaao"
}
POST books/_search
{
"query": {
"match": {
"name": "ääöääö"
}
}
}
And bam! There is our document.
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.30685282,
"hits": [
{
"_index": "books",
"_type": "book",
"_id": "AVkIXdNyDpmDHTvI6Cp1",
"_score": 0.30685282,
"_source": {
"name": "aaoaao"
}
}
]
}
}

Elasticsearch aggregation turns results to lowercase

I've been playing with ElasticSearch a little and found an issue when doing aggregations.
I have two endpoints, /A and /B. In the first one I have parents for the second one. So, one or many objects in B must belong to one object in A. Therefore, objects in B have an attribute "parentId" with parent index generated by ElasticSearch.
I want to filter parents in A by children attributes of B. In order to do it, I first filter children in B by attributes and get its unique parent ids that I'll later use to get parents.
I send this request:
POST http://localhost:9200/test/B/_search
{
"query": {
"query_string": {
"default_field": "name",
"query": "derp2*"
}
},
"aggregations": {
"ids": {
"terms": {
"field": "parentId"
}
}
}
}
And get this response:
{
"took": 91,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 3,
"max_score": 1,
"hits": [
{
"_index": "test",
"_type": "child",
"_id": "AU_fjH5u40Hx1Kh6rfQG",
"_score": 1,
"_source": {
"parentId": "AU_ffvwM40Hx1Kh6rfQA",
"name": "derp2child2"
}
},
{
"_index": "test",
"_type": "child",
"_id": "AU_fjD_U40Hx1Kh6rfQF",
"_score": 1,
"_source": {
"parentId": "AU_ffvwM40Hx1Kh6rfQA",
"name": "derp2child1"
}
},
{
"_index": "test",
"_type": "child",
"_id": "AU_fjKqf40Hx1Kh6rfQH",
"_score": 1,
"_source": {
"parentId": "AU_ffvwM40Hx1Kh6rfQA",
"name": "derp2child3"
}
}
]
},
"aggregations": {
"ids": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "au_ffvwm40hx1kh6rfqa",
"doc_count": 3
}
]
}
}
}
For some reason, the filtered key is returned in lowercase, hence not being able to request parent to ElasticSearch
GET http://localhost:9200/test/A/au_ffvwm40hx1kh6rfqa
Response:
{
"_index": "test",
"_type": "A",
"_id": "au_ffvwm40hx1kh6rfqa",
"found": false
}
Any ideas on why is this happening?
The difference between the hits and the results of the aggregations is that the aggregations work on the created terms. They will also return the terms. The hits return the original source.
How are these terms created? Based on the chosen analyser, which in your case is the default one, the standard analyser. One of the things this analyser does is lowercasing all the characters of the terms. Like mentioned by Andrei, you should configure the field parentId to be not_analyzed.
PUT test
{
"mappings": {
"B": {
"properties": {
"parentId": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
I am late from the party but I had the same issue and understood that it caused by the normalization.
You have to change the mapping of the index if you want to prevent any normalization changes the aggregated values to lowercase.
You can check the current mapping in the DevTools console by typing
GET /A/_mapping
GET /B/_mapping
When you see the structure of the index you have to see the setting of the parentId field.
If you don't want to change the behaviour of the field but you also want to avoid the normalization during the aggregation then you can add a sub-field to the parentId field.
For changing the mapping you have to delete the index and recreate it with the new mapping:
creating the index
Adding multi-fields to an existing field
In your case it looks like this (it contains only the parentId field)
PUT /B/_mapping
{
"properties": {
"parentId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
then you have to use the subfield in the query:
POST http://localhost:9200/test/B/_search
{
"query": {
"query_string": {
"default_field": "name",
"query": "derp2*"
}
},
"aggregations": {
"ids": {
"terms": {
"field": "parentId.keyword",
"order": {"_key": "desc"}
}
}
}
}

Resources