OpenSearch/ElasticSearch - Authorisation error removing alias using only a wildcard - elasticsearch

I'm trying to do something fairly simple - I have an alias that may be associated with one or more indices and I want to remove all associations and add a single one. Here is an example of what I'm doing (as suggested in here):
POST _aliases
{
"actions": [
{
"remove": {
"index": "*", // or "index": "_all"
"alias": "my_index"
}
},
{
"add": {
"index": "my_index_2023_01_05_17_18_29",
"alias": "my_index"
}
}
]
}
I'm using an admin user and for some reason I'm getting the following error message:
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=]"
}
],
"type" : "security_exception",
"reason" : "no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=]"
},
"status" : 403
}
Kind of weird message - it doesn't specify what's the action that is failing (no permissions for []) so I cannot figure out which permissions I'm missing...
Either way, if instead of using the wildcard like that ("index": "*"), I do something like "index": "my_index*" it works - It is not exactly what I want but it executes.
Any idea why this might be happening and how can I solve this?
Thanks!
(Using OpenSearch 2.3)

When you specify index_pattern as "*", system indexes are also included. Even with the admin user, you cannot change the settings of the system indexes by default. You can delete the alias by using parameters like "index_name"* or "2022" instead of "*".
For example:
POST _aliases
{
"actions": [
{
"remove": {
"index": "index_name*",
"alias": "my_index"
}
}
]
}
Edit:
You can use the following API to remove all aliases without system indices.
DELETE *,.-*/_alias/my-alias
You can check if the API is correct before removing them
GET *,-.*/_alias/my-alias
Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-alias.html
Note: Unfortunately, it's not possible to use an expression like *,.-* in the POST _aliases body.

Related

elascticsearch roll over based on ilm policy with keeping application/consumer as-is

I have an elasticsearch service which is a logging backend for our applications.
Application code is not open to us, so we can not change the application.
applications are calling elasticsearch in certain points in flow via http as below.
POST order-service-2022.04.23/_doc
{
"message": "order created",
"#timestamp": "1591890613"
}
in the later day they are calling the same service like below. only date is changing.
POST order-service-2022.04.24/_doc
{
"message": "order created",
"#timestamp": "1591890613"
}
The problem is we can not set up an ILM policy with out changing the client behaviour.
The aim is roll over the index based on some criteria, since we can not change the application code, http call should be like that in application side.
what we have tried is :
index template :
PUT _index_template/order_template
{
"index_patterns": [
"order-*-*"
],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index.lifecycle.name": "order_policy",
"index.lifecycle.rollover_alias": "order"
}
}
}
ilm policy
PUT _ilm/policy/order_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_primary_shard_size": "1GB",
"max_age": "1d",
"max_docs": 500
}
}
},
"delete": {
"min_age": "2d",
"actions": {
"delete": {}
}
}
}
}
}
seed index :
PUT order-service-2022.04.23
{
"aliases": {
"order": {
"is_write_index": true
}
}
}
during doc insert to elastic successful result.
but during rollup for only one day give below exception
POST /order/_rollover
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "index name [order-service-2022.04.23] does not match pattern '^.*-\\d+$'"
}
],
"type" : "illegal_argument_exception",
"reason" : "index name [order-service-2022.04.23] does not match pattern '^.*-\\d+$'"
},
"status" : 400
}
second issue how the up coming days need to rollup since no alias for them..
First, it looks like the application code is doing the "rollover" by itself since it's changing the index name everyday. So if you can't change the application code, you sort of already have some rollover logic but implemented directly in your application code.
The error you get is because the initial seed index MUST end with a sequential number otherwise it won't work.
The regular expression ^.*-\\d+$ means "any index name that ends with one or more digits"
add this
|
v
PUT order-service-2022.04.23-00001
{
"aliases": {
"order": {
"is_write_index": true
}
}
}
This also means that the application code that makes up the index name would NOT be able to know what the current sequence number is, and hence, MUST write directly to the order write alias instead, otherwise it's not going to work.

ElasticSearch - How create Index template/mapping per alias and perform search against each alias separately

Is is any way in elastic to store index template per alias.
I mean create Index with multiple aliases (alias1 ,alias2 ..) and attach different template to each of them. Then perform Index/Search docs on specific alias.
The reason I'm doing so due to multiple different data-structure (up to 50 types) of documents.
What I did so far is :
1. PUT /dynamic_index
2. POST /_aliases
{ "actions" : [
{ "add" : { "index" : "dynamic_index", "alias" : "alias_type1" } },
{ "add" : { "index" : "dynamic_index", "alias" : "alias_type2" } },
{ "add" : { "index" : "dynamic_index", "alias" : "alias_type3" } }
]}
3.
PUT_template/template1 {
"index_patterns": [
"dynamic_index"
],
"mappings": {
"dynamic_templates": [
{
"strings_as_keywords": {
"match_mapping_type": "string",
"mapping": {
"type": "text",
"analyzer": "standard",
"copy_to": "_all",
"fields": {
"keyword": {
"type": "keyword",
"normalizer": "lowercase_normalizer"
}
}
}
}
}
],
"properties": {
"source": {
"type": "keyword"
}
}
},
"aliases": {
"alias_type1": {
}
}
}
4. same way to alias_type2 , alias_type3 but different fields ...
Indexing/Search : Trying create and search docs per alias like in example:
POST alias_type1/_doc
{
"source": "foo"
, .....
}
POST alias_type2/_doc
{
"source": "foo123"
, .....
}
GET alias_type1/_search
{
"query": {
"match_all": {}
}
}
GET alias_type2/_search
{
"query": {
"match_all": {}
}
}
What I see actually that even if I index documents per alias,
when searching I don't see result per alias ,all results are same on alias_type1,2 and even on index.
Any way I can achieve separation logic on each alias in terms of searches/index docs per type (alias) ?
Any ideas ?
You can’t have separate mapping for aliases pointing to the same index! Aliases are like virtual link pointing to a index so if your aliases pointing to same index you will get the same result back.
If you want to have different mapping based on your data structure you will need to creat multiple indices.
Update
You also can use custom routing based on a field for more information you can check Elastic official documentation here.

how to index a file with elasticsearch 5.5.1

I'm new to Elasticsearch. I have successfully installed Elasticsearch with Kibana, X-pack and ingest-attachment. I have both Elasticsearch and Kibana running. I have kept it simple at the moment with the install using default options on a windows 2012 server. I have a directory on another drive w\mydocs and at the moment it just has 3 plain text files in it, but I will want to add others like pdf and doc file types. So now I want to get these files into Elasticsearches index. I have tried using the following link as a guide Attaching pdf docs in Elasticsearch, however I cannot get it to work.
Here's how I have set up the index and pipeline:
PUT _ingest/pipeline/docs
{
"description": "documents",
"processors" : [
{
"attachment" : {
"field": "data",
"indexed_chars" : -1
}
}]
}
PUT myindex
{
"mappings" : {
"documents" : {
"properties" : {
"attachment.data" : {
"type": "text",
"analyzer": "standard"
}
}
}
}
}
Then to get the first document in I use the following:
PUT localhost:9200/documents/1?pipeline=docs -d #/w/mydocs/README.TXT
and the error that I receive is:
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "request body is required"
}
],
"type": "parse_exception",
"reason": "request body is required"
},
"status": 400
}
you still have to send valid JSON to Elasticsearch, even when indexing binary data. This means, that you have to encode your document as base64 and then put it into a JSON document like this
{
"data" : "base64encodedcontentofyourfile"
}
I was advised not to use the ingest-attachment, but instead to use FsCrawler. I managed to get Fscrawler working without having to convert anything to base64.

How to ignore 404 errors when removing an ElasticSearch alias?

I'm trying to remove the alias my_alias on ElasticSearch 2.3 as:
curl -XPOST 'http://localhost:9200/_aliases' -d '
{
"actions" : [
{ "remove" : { "index" : "*", "alias" : "my_alias" } }
]
}'
However, this alias can not exist. In such case, I get the error:
{
"error": {
"reason": "aliases [my_alias] missing",
"resource.id": "my_alias",
"resource.type": "aliases",
"root_cause": [
{
"reason": "aliases [my_alias] missing",
"resource.id": "my_alias",
"resource.type": "aliases",
"type": "aliases_not_found_exception"
}
],
"type": "aliases_not_found_exception"
},
"status": 404
}
I tried adding ignore_unavailable to the action as { "remove" : { "index" : "*", "alias" : "my_alias", "ignore_unavailable": true } }, or simply ignore: 404, but to no avail. I looked at ElasticSearch's test suite for update_aliases in https://github.com/elastic/elasticsearch/tree/7560101ec73331acb39a042bde6130e59e4bb630/rest-api-spec/src/main/resources/rest-api-spec/test/indices.update_aliases, but couldn't find a test that did this.
I'm starting to think that there's no way to do that. Am I wrong?
I don't think this is possible. There is no delete "if exists" and that is somewhat expected from any HTTP API.
However, you can definitely ignore the response for this particular case in your code instead of treating it as an error.

elasticsearch routing on specific field

hi I want to set custom routing on specific field "userId" on my Es v2.0.
But it giving me error.I don't know how to set custom routing on ES v2.0
Please guys help me out.Thanks in advance.Below is error message, while creating custom routing with existing index.
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [_routing] has unsupported parameters: [path : userId]"
}
],
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [_routing] has unsupported parameters: [path : userId]"
},
"status": 400
}
In ES 2.0, the _routing.path meta-field has been removed. So now you need to do it like this instead:
In your mapping, you can only specify that routing is required (but you cannot specify path anymore):
PUT my_index
{
"mappings": {
"my_type": {
"_routing": {
"required": true
},
"properties": {
"name": {
"type": "string"
}
}
}
}
}
And then when you index a document, you can specify the routing value in the query string like this:
PUT my_index/my_type/1?routing=bar
{
"name": "foo"
}
You can still use custom routing based on a field from the data being indexed. You can setup a simple pipeline and then use the pipeline every time you index a document, or you can also change the index settings to use the pipeline whenever the index receives a document indexing request.
Read about the pipeline here
Do read above and below the docs for more clarity. It's not meant for setting custom routing, but can be used for the purpose. Custom routing was disabled for a good reason that the field to be used can turn out to have null values leading to unexpected behavior. Hence take care of that issue by yourself.
For routing, here is a sample pipeline PUT :
PUT localhost:9200/_ingest/pipeline/nameRouting
Content-Type: application/json
{
"description" : "Set's the routing value from name field in document",
"processors" : [
{
"set" : {
"field": "_routing",
"value": "{{_source.name}}"
}
}
]
}
The index settings will be :
{
"settings" : {
"index" : {
"default_pipeline" : "nameRouting"
}
}
}

Resources