How to use dot notation in geo percolator - elasticsearch

I use Elasticsearch 2.2.1 for searching documents that relate to a specific geo graphic location (within a bouding box). I want to create a percolator that I can use to check if a new document relates to an existing query.
This works fine if I put the percolator into the index containing the documents but because of the issue mention in this document and the workaround mentioned here I need to put the percolate queries into a dedicated percolator index.
When I try to put a percolator into this index:
PUT /mypercindex/.percolator/1
{"query": {"filtered": {"filter":
{"bool":
{"should":
[
{"geo_bounding_box":
{"location.coordinates":
{"bottom_right":
{"lat":50.0,"lon":8.0}
,"top_left":
{"lat":54.0,"lon":3.0}
}
}
}
]
}
}
}}}
I get an error message saying that:
Strict field resolution and no field mapping can be found for the field with name [location.coordinates]
In the percolator documentation is mentioned that, in case of a dedicated percolator index, you need to:
make sure that the mappings from the normal index are also available on the percolate index
This may cause my issue but I cannot find the documentation about how to make the mapping from one index available in the other. I tried to add the dedicated percolator index with the same mapping as my document index but when I do this I still get the same error message.
The mapping of my document index resembles this:
{"my_mapping": {
"dynamic":"strict",
"properties":{
"body":{
"properties":{
"author":{
"type":"string",
"index":"not_analyzed"
},
"hashtags":{
"type":"string",
"index":"not_analyzed"
},
"language":{
"type":"string",
"index":"not_analyzed"
}
,"text":{
"type":"string",
"analyzer":"stopwords"
},
"title":{
"type":"string",
"analyzer":"stopwords"
}
}
},
"location":{
"properties":{
"coordinates":{
"type":"geo_point"
},
"names":{
"type":"string",
"analyzer":"standard"
}
}
}
}
}}
Any help would be greatly appreciated!

Adding a .percolator mapping to the mapping, like mentioned in the Github issue that addresses this workaround, fixed the issue for me:
".percolator": {
"dynamic": true,
"properties": {
"id": {
"type": "integer"
}
}
}

Related

In elasticsearch, what's different between /doc/_mapping and /doc {"mappings"...}

I have confusion about elasticsearch mapping
First, I have a doc created with mapping request as
PUT /person
{
"mappings":{
"properties":{
"firstname":{
"type":"text"
}
}
}
}
Then, I want to add new property "lastname", to do that I PUT
PUT /person
{
"mappings":{
"properties":{
"lastname":{
"type":"text"
}
}
}
}
Unfortunately, the exception stops me
{
"error": {
"root_cause": [
{
"type": "resource_already_exists_exception",
"reason": "index [person/5TWopRJaQ3Clam2TiteuHg] already exists",
"index_uuid": "5TWopRJaQ3Clam2TiteuHg",
"index": "person"
}
],
"type": "resource_already_exists_exception",
"reason": "index [person/5TWopRJaQ3Clam2TiteuHg] already exists",
"index_uuid": "5TWopRJaQ3Clam2TiteuHg",
"index": "person"
},
"status": 400
}
It trys to tell me index existed, I could not update it.
But if I use,
PUT /person/_mapping
{
"properties":{
"lastname":{
"type":"text"
}
}
}
it works, and "lastname" is added into mapping.
What's the difference between /person/ {mappings:...} and /person/_mapping {...}?
They are all PUT method, shouldn't they do the same?
Or /person/ {mappings:...} could only be used as created, and /person/_mapping {...} is only for updating existed ones?
Not very make sense. What's the trick here?
https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html
Indices created in Elasticsearch 7.0.0 or later no longer accept a _default_ mapping. Indices created in 6.x will continue to function as before in Elasticsearch 6.x. Types are deprecated in APIs in 7.0, with breaking changes to the index creation, put mapping, get mapping, put template, get template and get field mappings APIs.
Hence, mappings support is deprecated in ES 7 and will be removed in ES 8

Elasticsearch returning no geo query results in es6.7

I'm mapping congressional districts in elasticsearch but my efforts are returning no results when trying to query by geoshape even though the query is clearly within the boundary of my WKT values. Here's some relevant data:
The mapping:
"mappings":{
"house":{
"properties":{
"state_name":{
"type":"text"
},
"district":{
"type":"integer"
},
"shape":{
"type":"geo_shape",
"strategy":"recursive"
}
}
}
An example of a WKT on the shape property: POLYGON((-122.612285 37.815224,-122.501272 37.821212,-122.499764 37.819724,-122.41871 37.852491,-122.418673 37.852505,-122.430183 37.918425,-122.432283 37.929824,-122.373982 37.883884,-122.373782 37.883725,-122.28246 37.709309,-122.28178 37.70823,-122.419802 37.708231,-122.420082 37.708231,-122.440893 37.716405,-122.440999 37.716488,-122.428203 37.731851,-122.428038 37.732016,-122.451147 37.731567,-122.453411 37.731568,-122.463399 37.752981,-122.463433 37.753028,-122.469667 37.738505,-122.470597 37.737665,-122.512732 37.735087,-122.578294 37.733988,-122.584728 37.779156,-122.588174 37.789362,-122.612285 37.815224))
My search query in kibana that is clearly inside that WKT:
GET /house/_search
{
"query":{
"bool": {
"must": {
"match_all": {}
},
"filter": {
"geo_shape": {
"shape": {
"shape": {
"type": "circle",
"coordinates" : [-122.421151, 37.758442],
"radius": "100m"
}
}
}
}
}
}
I've also tried geoshape type point with intersects and contains relation (which is not currently supported).
What else can I do to debug why these documents are not being found?
I realized that my mapping was incorrect. I was manually setting a geo_shape mapping on shape but when my indexes were created they were creating Shape (note capitalization change) properties with type text. Thus none of my indexes had any geo_shape data to search.
The most useful thing I used to debug and find this issue was a readout of the GET /house/mapping data which showed the two distinct properties that should have been a single property.

Full Text Search as well as Terms Search on same filed of Elasticsearch

I'm from MySql background. So I don't know much about elasticsearch and it's working.
Here is my requirements
There will be table of resulted records with sorting option on all the column. There will be filter option from where user will select multiple values for multiple columns (e.g, City should be from City1, City2, City3 and Category should be from Cat2, Cat22, Cat6). There will be also search bar where user will enter some text and full text search will be applied on some fields (i.e, City, Area etc).
This image will give better understanding.
Where I'm facing problem is Full Text Search. I have tried some mapping but every time I have to compromise either on Full Text Search or Terms Search. So I think there is no any way to apply both search on same field. But as I told, I don;t know much about elasticsearch. So if any one have solution, it will be appreciated.
Here is what I have applied currently which makes sorting and Terms Searching enable but Full Text Search is not working.
{
"mappings":{
"my_type":{
"properties":{
"city":{
"type":"string",
"index":"not_analyzed"
},
"category":{
"type":"string",
"index":"not_analyzed"
},
"area":{
"type":"string",
"index":"not_analyzed"
},
"zip":{
"type":"string",
"index":"not_analyzed"
},
"state":{
"type":"string",
"index":"not_analyzed"
}
}
}
}
}
You can update the mapping with multifields with two mappings one for full text and another for terms search. Here's a sample mapping for city.
{
"city": {
"type": "string",
"index": "not_analyzed",
"fields": {
"fulltext": {
"type": "string"
}
}
}
}
Default mapping is for terms search, so when terms search is required, you could simple query in "city" field. But, you need full-text search, query must be performed on "city.fulltext". Hope this helps.
Full-text search won't work on not_analyzed fields and sorting won't work on analyzed fields.
You need to use multi-fields.
It is often useful to index the same field in different ways for different purposes. This is the purpose of multi-fields. For instance, a string field could be mapped as a text field for full-text search, and as a keyword field for sorting or aggregations:
For example :
{
"mappings": {
"my_type": {
"properties": {
"city": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
} ...
}
}
}
}
Use the dot notation to sort by city.raw :
{
"query": {
"match": {
"city": "york"
}
},
"sort": {
"city.raw": "asc"
}
}

Keep non-stemmed tokens on Elasticsearch

I'm using a stemmer (for the Brazilian Portuguese Language) when I index documents on Elasticsearch. This is what my default analyzer looks like(nvm minor mistakes here because I've copied this by hand from my code in the server):
{
"analysis":{
"filter":{
"my_asciifolding": {
"type": "asciifolding",
"preserve_original": true,
},
"stop_pt":{
"type": "stop",
"ignore_case": true,
"stopwords": "_brazilian_"
},
"stemmer_pt": {
"type": "stemmer",
"language": "brazilian"
}
},
"analyzer": {
"default": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"my_asciifolding",
"stop_pt",
"stemmer_pt"
]
}
}
}
}
I haven't really touched my type mappings (apart from a few numeric fields I've declared "type":"long") so I expect most fields to be using this default analyzer I've specified above.
This works as expected, but the thing is that some users are frustrated because (since tokens are being stemmed), the query "vulnerabilities" and the query "vulnerable" return the same results, which is misleading because they expect the results having an exact match to be ranked first.
Whats is the default way (if any) to do this in elasticsearch? (maybe keep the unstemmed tokens in the index as well as the stemmed tokens?) I'm using version 1.5.1.
I ended up using "fields" field to index my attributes in different ways. Not sure whether this is optimal but this is the way I'm handling it right now:
Add another analyzer (I called it "no_stem_analyzer") with all filters that the "default" analyzer has, minus "stemmer".
For each attribute I want to keep both non stemmed and stemmed variants, I did this (example for field "DESCRIPTION"):
"mappings":{
"_default_":{
"properties":{
"DESCRIPTION":{
"type"=>"string",
"fields":{
"no_stem":{
"type":"string",
"index":"analyzed",
"analyzer":"no_stem_analyzer"
},
"stemmed":{
"type":"string",
"index":"analyzed",
"analyzer":"default"
}
}
}
},//.. other attributes here
}
}
At search time (using query_string_query) I must also indicate (using field "fields") that I want to search all sub-fields (e.g. "DESCRIPTION.*")
I also based my approach upon [this answer].(elasticsearch customize score for synonyms/stemming)

Overriding default keyword analysis for elasticsearch

I am trying to configure an elasticsearch index to have a default indexing policy of analysis with the keyword analyzer, and then overriding it on some fields, to allow them to be free text analyzed. So effectively opt-in free text analysis, where I am explicitly specifying in the mapping which fields are analysed for free text matching. My mapping defintion looks like this:
PUT test_index
{
"mappings":{
"test_type":{
"index_analyzer":"keyword",
"search_analyzer":"standard",
"properties":{
"standard":{
"type":"string",
"index_analyzer":"standard"
},
"keyword":{
"type":"string"
}
}
}
}
}
So standard should be an analyzed field, and keyword should be exact match only. However when I insert some sample data with the following command:
POST test_index/test_type
{
"standard":"a dog in a rug",
"keyword":"sheepdog"
}
I am not getting any matches against the following query:
GET test_index/test_type/_search?q=dog
However I do get matches against:
GET test_index/test_type/_search?q=*dog*
Which makes me think that the standard field is not being analyzed. Does anyone know what I am doing wrong?
Nothing's wrong with the index created. Change your query to GET test_index/test_type/_search?q=standard:dog and it should return the expected results.
If you do not want to specify field name in the query, update your mapping such that you provide the index_analyzer and search_analyzer values explicitly for each field with no default values. See below:
PUT test_index
{
"mappings": {
"test_type": {
"properties": {
"standard": {
"type": "string",
"index_analyzer": "standard",
"search_analyzer": "standard"
},
"keyword": {
"type": "string",
"index_analyzer": "keyword",
"search_analyzer": "standard"
}
}
}
}
}
Now if you try GET test_index/test_type/_search?q=dog, you'll get the desired results.

Resources