Problem with creating roles in open-distro for elasticsearch - elasticsearch

I have 2 roles that are assigned to one user. In the first role, I include field name for documents which have _id 1 and 2
{
"index_permissions": [
{
"index_patterns": [
"test"
],
"dls": "{\n \"terms\": {\n \"_id\": [ \"1\", \"2\"] \n }\n}\n\n",
"fls": [
"name"
],
"masked_fields": [],
"allowed_actions": [
"get",
"crud"
]
}
],
"tenant_permissions": [],
"cluster_permissions": [
"*"
]
}
and in the second role, I include field job_description for document which have _id 3
{
"index_permissions": [
{
"index_patterns": [
"test"
],
"dls": "{\n \"terms\": {\n \"_id\": [\"3\"] \n }\n}\n",
"fls": [
"job_description"
],
"masked_fields": [],
"allowed_actions": []
}
],
"tenant_permissions": [],
"cluster_permissions": []
}
when I try to get data from the index it shows me job_description and name in all documents,
{
"took" : 237,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 2.0,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : 2.0,
"_source" : {
"name" : "John",
"job_description" : "Systems administrator and Linux specialist"
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "2",
"_score" : 2.0,
"_source" : {
"name" : "John",
"job_description" : "Systems administrator and Linux specialist"
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "3",
"_score" : 2.0,
"_source" : {
"name" : "John",
"job_description" : "Systems administrator and Linux specialist"
}
}
]
}
}
but I want to see the only name in two firs records and only job_description in 3 document like that
{
"took" : 237,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 2.0,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : 2.0,
"_source" : {
"name" : "John",
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "2",
"_score" : 2.0,
"_source" : {
"name" : "John",
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "3",
"_score" : 2.0,
"_source" : {
"job_description" : "Systems administrator and Linux specialist"
}
}
]
}
}
does anyone know how to do it?

DLS and FLS do not work in conjunction like that.
DLS is used to only return back a subset of search response based on the DLS query, whereas FLS is used to only include or exclude certain fields from the search response returned from elasticsearch.
All the DLS queries are combined (OR condition) and similarly all FLS input is combined (AND condition) for a user that contains multiple such configurations.
In your case, you have two DLS and two FLS query. The two DLS queries will work as OR conditions, in your case it will return back documents matching 1,2 or 3 doc_id. Similarly, both name and job_description will be returned back.

Related

Elasticsearch returning wrong results upon query

I am new to ElasticSearch and was doing some experiments to learn but I figured out that _search query is returning wrong results. I inserted documents to index by using following code
PUT tryDB/_doc/2
{"personId":"2","minor":true,"money":15 }
PUT tryDB/_doc/3
{"personId":"3","minor":true,"money":20 }
PUT tryDB/_doc/4
{"personId":"4","minor":true,"money":25 }
PUT tryDB/_doc/5
{"personId":"5","minor":true,"money":30 }
PUT tryDB/_doc/6
{"personId":"6","minor":true,"money":35 }
PUT tryDB/_doc/7
{"personId":"7","minor":true,"money":40 }
PUT tryDB/_doc/8
{"personId":"8","minor":true,"money":45 }
PUT tryDB/_doc/9
{"personId":"9","minor":true,"money":55 }
PUT tryDB/_doc/10
{"personId":"10","minor":true,"money":60 }
PUT tryDB/_doc/11
{"personId":"11","minor":true,"money":65 }
PUT tryDB/_doc/12
{"personId":"12","minor":true,"money":70 }
PUT tryDB/_doc/13
{"personId":"2","minor":false,"money":80 }
PUT tryDB/_doc/14
{"personId":"2","minor":false,"money":90 }
PUT tryDB/_doc/15
{"personId":"2","minor":false,"money":100 }
PUT tryDB/_doc/16
{"personId":"2","minor":false,"money":10 }
After which I fired up a GET tryDB/_search query to list all the documents, which in turn returns
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 16,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "tryDB",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"personId" : "1",
"minor" : true,
"money" : 10
}
},
{
"_index" : "tryDB",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"personId" : "2",
"minor" : true,
"money" : 15
}
},
{
"_index" : "tryDB",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"personId" : "3",
"minor" : true,
"money" : 20
}
},
{
"_index" : "tryDB",
"_id" : "4",
"_score" : 1.0,
"_source" : {
"personId" : "4",
"minor" : true,
"money" : 25
}
},
{
"_index" : "tryDB",
"_id" : "5",
"_score" : 1.0,
"_source" : {
"personId" : "5",
"minor" : true,
"money" : 30
}
},
{
"_index" : "tryDB",
"_id" : "6",
"_score" : 1.0,
"_source" : {
"personId" : "6",
"minor" : true,
"money" : 35
}
},
{
"_index" : "tryDB",
"_id" : "7",
"_score" : 1.0,
"_source" : {
"personId" : "7",
"minor" : true,
"money" : 40
}
},
{
"_index" : "tryDB",
"_id" : "8",
"_score" : 1.0,
"_source" : {
"personId" : "8",
"minor" : true,
"money" : 45
}
},
{
"_index" : "tryDB",
"_id" : "9",
"_score" : 1.0,
"_source" : {
"personId" : "9",
"minor" : true,
"money" : 55
}
},
{
"_index" : "tryDB",
"_id" : "10",
"_score" : 1.0,
"_source" : {
"personId" : "10",
"minor" : true,
"money" : 60
}
}
]
}
}
Where are the rest 6 documents ?
Now I went ahead and fired up a range based query
GET tryDB/_search
{
"query": {
"range": {
"money": {
"lte":100
}
}
}
}
Which in turn returned
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "tryDB",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"personId" : "1",
"minor" : true,
"money" : 10
}
},
{
"_index" : "tryDB",
"_id" : "15",
"_score" : 1.0,
"_source" : {
"personId" : "2",
"minor" : false,
"money" : 100
}
},
{
"_index" : "tryDB",
"_id" : "16",
"_score" : 1.0,
"_source" : {
"personId" : "2",
"minor" : false,
"money" : 10
}
}
]
}
}
Which is wrong clearly. Can anyone help me figure out what's going on here?
Where are the rest 6 documents ?
When you do not determine the value of "size", by default elastic returns 10 documents.
Set size like this:
{
"size": 20,
"query": {
"match_all": {}
}
}
POST tryDB/_search
{
"query": {
"bool": {
"filter": [
{
"range": {
"money": {
"lte": 100
}
}
}
]
}
}
}
#rabbitbr Thanks for the quick response!
Hey I figured out the solution (posting here)
Based on the result,
Looks like Elastic Search index money as string.
I tried setting up an explicit mapping to make sure the money field indexed as number.
https://opensearch.org/docs/1.3/opensearch/mappings/
This worked out.

How do I apply reindex to new data values through filters?

This is basic_data(example) Output value
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 163,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "0513_final_test_instgram",
"_type" : "_doc",
"_id" : "6uShY3kBEkIlakOYovrR",
"_score" : 1.0,
"_source" : {
"host" : "DESKTOP-7MDCA36",
"path" : "C:/python_file/20210513_114123_instargram.csv",
"#version" : "1",
"message" : "hello",
"#timestamp" : "2021-05-13T02:50:05.962Z"
},
{
"_index" : "0513_final_test_instgram",
"_type" : "_doc",
"_id" : "EeShY3kBEkIlakOYovvm",
"_score" : 1.0,
"_source" : {
"host" : "DESKTOP-7MDCA36",
"path" : "C:/python_file/20210513_114123_instargram.csv",
"#version" : "1",
"message" : "python,
"#timestamp" : "2021-05-13T02:50:05.947Z"
}
First of all, out of various field values, only message values have been extracted.(under code example)
GET 0513_final_test_instgram/_search?_source=message&filter_path=hits.hits._source
{
"hits" : {
"hits" : [
{
"_source" : {
"message" : "hello"
}
},
{
"_source" : {
"message" : "python"
}
I got to know reindex that stores new indexes.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
However, I don't know even if I look at the document.
0513 attempt code
POST _reindex
{
"source": {
"index": "0513_final_test_instgram"
},
"dest": {
"index": "new_data_index"
}
}
How do you use reindex to store data that only extracted message values in a new index?
update comment attempt
output
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 163,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "new_data_index",
"_type" : "_doc",
"_id" : "6uShY3kBEkIlakOYovrR",
"_score" : 1.0,
"_source" : {
"message" : "hello"
}
},
{
"_index" : "new_data_index",
"_type" : "_doc",
"_id" : "EeShY3kBEkIlakOYovvm",
"_score" : 1.0,
"_source" : {
"message" : "python"
}
}
You simply need to specify which fields you want to reindex into the new index:
{
"source": {
"index": "0513_final_test_instgram",
"_source": ["message"]
},
"dest": {
"index": "new_data_index"
}
}

How to get max_score in a query in elasticsearch

Hi I am new to elastic search, I was wondering how I could get the max_score of my first query and then compare it to the rest of the values. For example if the max_score was 2.6 I would want to take that value and compare it with the _score of all the docs in the query.
GET searchentities/_search
{
"query": {
"multi_match": {
"query": "iron man",
"type": "bool_prefix",
"fields": [
"title",
"title._2gram",
"title._3gram"
],
"minimum_should_match": 2
}
}
}
Gives back the following:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 2.4475412,
"hits" : [
{
"_index" : "searchentities",
"_type" : "_doc",
"_id" : "IronMan",
"_score" : 2.4475412,
"_source" : {
"id" : "Iron Man",
"ad" : false,
"verified" : false,
"clicks" : 2,
"photoID" : "5f8a9dd82ab79c00017722bb",
"title" : "Iron Man "
}
},
{
"_index" : "searchentities",
"_type" : "_doc",
"_id" : "IronMan3",
"_score" : 2.2448254,
"_source" : {
"id" : "Iron Man 3 ",
"ad" : false,
"verified" : false,
"clicks" : 2,
"photoID" : "5f8a9dd82ab79c00017722bb",
"title" : "Iron Man 3 "
}
},
{
"_index" : "searchentities",
"_type" : "_doc",
"_id" : "IronMan2",
"_score" : 2.2448254,
"_source" : {
"id" : "Iron Man 2 ",
"ad" : false,
"verified" : false,
"clicks" : 20,
"photoID" : "5f8a9dd82ab79c00017722bb",
"title" : "Iron Man 2"
}
}
]
}
}
I would want to see
2.44-2.44 = 0
2.44-2.24 = .2
2.44-2.24 = .2
Unfortunately there is no way to post process the score in this way in elastic.
The max score is just the highest score after all scores have been calculated.
This values is not available in the postprocess possibilities that exists in Elasticsearch.
See:
https://discuss.elastic.co/t/using-max-score-inside-script-function/156871/3
I think you have to write your own program processing the result and returning the difference between scores.

Can I bulk delete documents without specifying a routing key?

I'm using python-elasticsearch (https://elasticsearch-py.readthedocs.io/en/master/) to bulk insert documents on an ES cluster with multiple shards. I'm using a routing during the insertion, but I don't have access to that key on deletion
Is there a way to hit all shards? Something like routing=*. If I don't specify a routing key ES is not able to find the document
This
POST _bulk
{ "index" : { "_index" : "test", "_id" : "1", "routing": "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } } // Fails because routing is missing
gives me
{
"took" : 1334,
"errors" : false,
"items" : [
{
"index" : {
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1,
"status" : 201
}
},
{
"delete" : {
"_index" : "test",
"_type" : "_doc",
"_id" : "2",
"_version" : 1,
"result" : "not_found",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1,
"status" : 404
}
}
]
}

Using index sorting by default in _search

I am using ElasticSearch 7.6 and the Index Sorting feature which was introduced in 6.0.
What i am looking to do is to do a GET /myindice/_search without specifying sort and get documents based on Index sorting settings I have specified for my index and NOT insertion order.
My index as per the doc :
PUT twitter
{
"settings" : {
"index" : {
"sort.field" : "date",
"sort.order" : "desc"
}
},
"mappings": {
"properties": {
"date": {
"type": "date"
}
}
}
}
PUT twitter/_doc/a
{
"date": "2015-01-01"
}
PUT twitter/_doc/b
{
"date": "2016-01-01"
}
PUT twitter/_doc/c
{
"date": "2017-01-01"
}
My initial thought is that
GET twitter/_search
Should return doc C, B and A.
I get the following :
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "twitter",
"_type" : "_doc",
"_id" : "a",
"_score" : 1.0,
"_source" : {
"date" : "2015-01-01"
}
},
{
"_index" : "twitter",
"_type" : "_doc",
"_id" : "b",
"_score" : 1.0,
"_source" : {
"date" : "2016-01-01"
}
},
{
"_index" : "twitter",
"_type" : "_doc",
"_id" : "c",
"_score" : 1.0,
"_source" : {
"date" : "2017-01-01"
}
}
]
}
}
As the documentation isn't clear at this particular subject and that all query are using sort :
https://www.elastic.co/guide/en/elasticsearch/reference/6.0/index-modules-index-sorting.html
Am I required to specify the sort order in the GET query (hence repeating the sort specified as the Index Sorting) ?
Thanks in advance for any diligent soul that could help me,

Resources