How to filter fields in ElasticSearch using GET - elasticsearch

I recently installed ElasticSearch with the Wikipedia river because I'm working on an autocomplete box with article titles. I have been trying to figure out the best way to query the dataset. The following works:
/wikipedia/_search?q=mysearch&fields=title,redirect&size=20
but I would like to add more constraints to the search:
disambiguation=false, redirect=false, stub=false, special=false
I'm new to ElasticSearch and the documentation hasn't gotten me far. From what I've read I need a filtered query; is there a way to do that from a GET request? That would make it much easier for my specific use case. If not, how would the POST request look? Thanks in advance.
For reference, the mapping is:
{
"wikipedia": {
"page": {
"properties": {
"category": {
"type": "string"
},
"disambiguation": {
"type": "boolean"
},
"link": {
"type": "string"
},
"redirect": {
"type": "boolean"
},
"special": {
"type": "boolean"
},
"stub": {
"type": "boolean"
},
"text": {
"type": "string"
},
"title": {
"type": "string"
}
}
}
}
}

For adding more constraints you can continue with the lucene syntax and do something like:
/wikipedia/_search?q=mysearch AND disambiguation:false AND redirect:false AND stub:false AND special:false&fields=title,redirect&size=20
For improving the performance you can use filters using the json API, the query will look like:
curl -XGET 'http://localhost:9200/wikipedia/_search?pretty=true' -d '
{
"from" : 0,
"size" : 20,
"query":{
"filtered" : {
"query" : {
"text" : { "title" : "test" }
},
"filter" : {
"and": [
{
"term" : {
"stub" : false
}
},
{
"term" : {
"disambiguation" : false
}
},
{
"term" : {
"redirect" : false
}
},
{
"term" : {
"special" : false
}
}
]
}
}
}
}
'

Related

Place an Analyzer on a a specific array item in a nested object

I have the following mapping
"mappings":{
"properties":{
"name": {
"type": "text"
},
"age": {
"type": "integer"
},
"customProps":{
"type" : "nested",
"properties": {
"key":{
"type": "keyword"
},
"value": {
"type" : "keyword"
}
}
}
}
}
example data
{
"name" : "person1",
"age" : 10,
"customProps":[
{"hairColor":"blue"},
{"height":"120"}
]
},
{
"name" : "person2",
"age" : 30,
"customProps":[
{"jobTitle" : "software engineer"},
{"salaryAccount" : "AvGhj90AAb"}
]
}
so i want to be able to search for document by salary account case insensitive, i am also searching using wild card
example query is
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "customProps",
"query": {
"bool": {
"must": [
{ "match": { "customProps.key": "salaryAccount" } },
{ "wildcard": { "customProps.value": "*AvG*"
}
}
]}}}}]}}}
i tried adding analyzer with PUT using the following syntax
{
"settings":{
"index":{
"analysis":{
"analyzer":{
"analyzer_case_insensitive" : {
"tokenizer":"keyword",
"filter":"lowercase"
}
}
}
}
},
"mappings":{
"people":{
"properties":{
"customProps":{
"properties":{
"value":{
"type": "keyword",
"analyzer": "analyzer_case_insensitive"
}
}
}
}
}
}
}
im getting the following error
"type" : "mapper_parsing_exception",
"reason" : "Root mapping definition has unsupported parameters: [people: {properties={customProps={properties={value={analyzer=analyzer_case_insensitive, type=keyword}}}}}]"
any idea how to do the analyzer for the salary account object in the array when it exists?
Your use case is quite clear, that you want to search on the value of salaryAccount only when this key exists in customProps array.
There are some issues with your mapping definition :
You cannot define a custom analyzer for keyword type field, instead you can use a normalizer
Based on the mapping definition you added at the beginning of the question, it seems that you are using elasticsearch version 7.x. But the second mapping definition that you provided, in that you have added mapping type also (i.e people), which is deprecated in 7.x
There is no need to add the key and value fields in the index mapping.
Adding a working example with index mapping, search query, and search result
Index Mapping:
PUT myidx
{
"mappings": {
"properties": {
"customProps": {
"type": "nested"
}
}
}
}
Search Query:
You need to use exists query, to check whether a field exists or not. And case_insensitive param in Wildcard query is available since elasticsearch version 7.10. If you are using a version below this, then you need to use a normalizer, to achieve case insensitive scenarios.
POST myidx/_search
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "customProps",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "customProps.salaryAccount"
}
},
{
"wildcard": {
"customProps.salaryAccount.keyword": {
"value": "*aVg*",
"case_insensitive": true
}
}
}
]
}
}
}
}
]
}
}
}
Search Result:
"hits" : [
{
"_index" : "myidx",
"_type" : "_doc",
"_id" : "2",
"_score" : 2.0,
"_source" : {
"name" : "person2",
"age" : 30,
"customProps" : [
{
"jobTitle" : "software engineer"
},
{
"salaryAccount" : "AvGhj90AAb"
}
]
}
}
]

Elasticsearch preform "OR" search on groups of filtering criteria

Overview: I have a situation where within a single index I want to preform a search to return results based on 2 different sets of criteria. Imagine a scenario where where I have an index with a data structure like what is outlined below
I want to preform some sort of query that looks at different "blocks" of criteria. Meaning I want to search by both of the following categories in a single query (if possible):
Category One:
Distance / location
Public = true
--OR--
Category Two:
Distance / location
Public = false
category = "specific category"
(although this is not my exact scenario it is an illustration of what I am facing):
{
"mappings" : {
"properties" : {
"category" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"completed" : {
"type" : "boolean"
},
"deleted" : {
"type" : "boolean"
},
"location" : {
"type" : "geo_point"
},
"public" : {
"type" : "boolean"
},
"uuid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
Please note: I am rather new to Elastic and would appreciate any help with this. I have attempted to search for this question but was not able to find what I was looking for. Please let me know if there is any missing information here I should include.
Sure, you can combine bool queries
POST test_user2737876/_search
{
"query": {
"bool": {
"should": [
{
"bool": {
"filter": [
{
"geo_distance": {
"distance": "200km",
"location": {
"lat": 41.12,
"lon": -71.34
}
}
},
{
"term": {
"public": false
}
}
]
}
},
{
"bool": {
"filter": [
{
"geo_distance": {
"distance": "200km",
"location": {
"lat": 41.12,
"lon": -71.34
}
}
},
{
"term": {
"category.keyword": "specific category"
}
},
{
"term": {
"public": false
}
}
]
}
}
],
"minimum_should_match": 1
}
}
}

How to Query elasticsearch index with nested and non nested fields

I have an elastic search index with the following mapping:
PUT /student_detail
{
"mappings" : {
"properties" : {
"id" : { "type" : "long" },
"name" : { "type" : "text" },
"email" : { "type" : "text" },
"age" : { "type" : "text" },
"status" : { "type" : "text" },
"tests":{ "type" : "nested" }
}
}
}
Data stored is in form below:
{
"id": 123,
"name": "Schwarb",
"email": "abc#gmail.com",
"status": "current",
"age": 14,
"tests": [
{
"test_id": 587,
"test_score": 10
},
{
"test_id": 588,
"test_score": 6
}
]
}
I want to be able to query the students where name like '%warb%' AND email like '%gmail.com%' AND test with id 587 have score > 5 etc. The high level of what is needed can be put something like below, dont know what would be the actual query, apologize for this messy query below
GET developer_search/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"name": "abc"
}
},
{
"nested": {
"path": "tests",
"query": {
"bool": {
"must": [
{
"term": {
"tests.test_id": IN [587]
}
},
{
"term": {
"tests.test_score": >= some value
}
}
]
}
}
}
}
]
}
}
}
The query must be flexible so that we can enter dynamic test Ids and their respective score filters along with the fields out of nested fields like age, name, status
Something like that?
GET student_detail/_search
{
"query": {
"bool": {
"must": [
{
"wildcard": {
"name": {
"value": "*warb*"
}
}
},
{
"wildcard": {
"email": {
"value": "*gmail.com*"
}
}
},
{
"nested": {
"path": "tests",
"query": {
"bool": {
"must": [
{
"term": {
"tests.test_id": 587
}
},
{
"range": {
"tests.test_score": {
"gte": 5
}
}
}
]
}
},
"inner_hits": {}
}
}
]
}
}
}
Inner hits is what you are looking for.
You must make use of Ngram Tokenizer as wildcard search must not be used for performance reasons and I wouldn't recommend using it.
Change your mapping to the below where you can create your own Analyzer which I've done in the below mapping.
How elasticsearch (albiet lucene) indexes a statement is, first it breaks the statement or paragraph into words or tokens, then indexes these words in the inverted index for that particular field. This process is called Analysis and that this would only be applicable on text datatype.
So now you only get the documents if these tokens are available in inverted index.
By default, standard analyzer would be applied. What I've done is I've created my own analyzer and used Ngram Tokenizer which would be creating many more tokens than just simply words.
Default Analyzer on Life is beautiful would be life, is, beautiful.
However using Ngrams, the tokens for Life would be lif, ife & life
Mapping:
PUT student_detail
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "ngram",
"min_gram": 3,
"max_gram": 4,
"token_chars": [
"letter",
"digit"
]
}
}
}
},
"mappings" : {
"properties" : {
"id" : {
"type" : "long"
},
"name" : {
"type" : "text",
"analyzer": "my_analyzer",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"email" : {
"type" : "text",
"analyzer": "my_analyzer",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"age" : {
"type" : "text" <--- I am not sure why this is text. Change it to long or int. Would leave this to you
},
"status" : {
"type" : "text",
"analyzer": "my_analyzer",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"tests":{
"type" : "nested"
}
}
}
}
Note that in the above mapping I've created a sibling field in the form of keyword for name, email and status as below:
"name":{
"type":"text",
"analyzer":"my_analyzer",
"fields":{
"keyword":{
"type":"keyword"
}
}
}
Now your query could be as simple as below.
Query:
POST student_detail/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"name": "war" <---- Note this. This would even return documents having "Schwarb"
}
},
{
"match": {
"email": "gmail" <---- Note this
}
},
{
"nested": {
"path": "tests",
"query": {
"bool": {
"must": [
{
"term": {
"tests.test_id": 587
}
},
{
"range": {
"tests.test_score": {
"gte": 5
}
}
}
]
}
}
}
}
]
}
}
}
Note that for exact matches I would make use of Term Queries on keyword fields while for normal searches or LIKE in SQL I would make use of simple Match Queries on text Fields provided they make use of Ngram Tokenizer.
Also note that for >= and <= you would need to make use of Range Query.
Response:
{
"took" : 233,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 3.7260926,
"hits" : [
{
"_index" : "student_detail",
"_type" : "_doc",
"_id" : "1",
"_score" : 3.7260926,
"_source" : {
"id" : 123,
"name" : "Schwarb",
"email" : "abc#gmail.com",
"status" : "current",
"age" : 14,
"tests" : [
{
"test_id" : 587,
"test_score" : 10
},
{
"test_id" : 588,
"test_score" : 6
}
]
}
}
]
}
}
Note that I observe the document you've mentioned in your question, in my response when I run the query.
Please do read the links I've shared. It is vital that you understand the concepts. Hope this helps!

ElasticSearch mapping "not working" v1.1.1

I have been struggling over what must be a simple syntax issue, trying to make a basic ES mapping work. This is version 1.1.1
I have a gist to be able to create everything from scratch:
https://gist.github.com/jrmadsen67/1fc5e296e26e7a5edae0
The mapping query is:
PUT /movies/movie/_mapping
{
"movie": {
"properties": {
"director": {
"type": "multi_field",
"fields": {
"director": {"type": "string"},
"original": {"type" : "string", "index" : "not_analyzed"}
}
}
}
}
}
I ran:
curl localhost:9200/movies/movie/_mapping?pretty=true
to confirm the mapping is actually there
the query:
POST /movies/movie/_search
{
"query": {
"constant_score": {
"filter": {
"term": { "director.original": "Francis Ford Coppola" }
}
}
}
}
does not get any hits. {"director": "francis"} works as expected.
Greatly appreciate another set of eyes that can tell me what's wrong with this!
The current mapping which match yours documents is :
{
"movie" : {
"properties" : {
"director" : {
"type" : "string"
},
"genres" : {
"type" : "string"
},
"title" : {
"type" : "string"
},
"year" : {
"type" : "long"
}
}
}
}
To match yours mapping and query, documents should be looking like :
{
"director": {
"director": "Francis Ford Coppola",
"original": "xxxxxx"
}
}
This issue was nothing more than the need to re-index the data after adding the mapping.
Here's an excellent walkthrough for others having this trouble:
https://www.found.no/foundation/mapping-workflow/

Exact match a field in Elasticsearch

In each of the documents I am indexing I have a field called "permalink" which I would like to exact match on.
An example document:
{
"entity_type": "company",
"entity_id": 1383221763,
"company_type": "developer",
"name": "Runewaker Entertainment",
"permalink": "runewaker-entertainment"
}
The mapping for these documents is:
{
"properties": {
"entity_id": {
"type": "integer",
"include_in_all": false
},
"name": {
"type": "string",
"include_in_all": true,
},
"permalink": {
"type": "string",
"include_in_all": true,
"index": "not_analyzed"
},
"company_type": {
"type": "string",
"include_in_all": false,
"index": "not_analyzed"
}
}
}
When I run the following query then I don't get any hits:
POST /companies/company/_search HTTP/1.1
Host: localhost:8082
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"term": { "permalink": "runewaker-entertainment" }
}
}
}
}
but I get match with this query:
POST /companies/company/_search HTTP/1.1
Host: localhost:8082
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"term": { "permalink": "runewaker" }
}
}
}
}
It appears any permalink with a hyphen in it results in a failed query but I was under the impression that if the mapping for a property has the index set to not_analyzed then ElasticSearch wouldn't analyze the field at all.
What should the correct query be?
Thank you
UPDATE:
getMapping result on the Companies index:
{
"companies" : {
"company" : {
"properties" : {
"company_type" : {
"type" : "string"
},
"entity_id" : {
"type" : "long"
},
"entity_type" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"node_id" : {
"type" : "long"
},
"permalink" : {
"type" : "string"
}
}
}
}
}
What you described is correct.
I tested and it works as expected. So you probably have some problem with your index. Maybe you indexed the document before you set the mapping?
Try to do it again -
delete your index or create a new one.
do a putMapping with your mapping.
index the document.
The search should work as expected.

Resources