I am trying to create an alias with a filter of an index pattern metrics-* . I was able to do it yesterday and the day before without any problems but I can't do it again today, even if I re-run the same queries as yesterday. I have no problem creating an alias of logs-* . But when I try to create a metrics-* alias, I get an HTTP 400 code with this as response:
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "expressions [metrics-system.filesystem-default, metrics-system.cpu-default, metrics-endpoint.policy-default, metrics-endpoint.metrics-default, metrics-windows.perfmon-default, metrics-azure.compute_vm-default, metrics-system.process.summary-default, metrics-elastic_agent.endpoint_security-default, metrics-endpoint.metadata-default, metrics-endpoint.metadata_current_default, metrics-azure.storage_account-default, metrics-system.memory-default, metrics-system.uptime-default, metrics-elastic_agent.elastic_agent-default, metrics-windows.service-default, metrics-elastic_agent.metricbeat-default, metrics-system.fsstat-default, metrics-system.process-default, metrics-elastic_agent.filebeat-default, metrics-system.network-default, metrics-system.diskio-default, metrics-system.load-default, metrics-system.socket_summary-default] that match with both data streams and regular indices are disallowed"
}
],
"type" : "illegal_argument_exception",
"reason" : "expressions [metrics-system.filesystem-default, metrics-system.cpu-default, metrics-endpoint.policy-default, metrics-endpoint.metrics-default, metrics-windows.perfmon-default, metrics-azure.compute_vm-default, metrics-system.process.summary-default, metrics-elastic_agent.endpoint_security-default, metrics-endpoint.metadata-default, metrics-endpoint.metadata_current_default, metrics-azure.storage_account-default, metrics-system.memory-default, metrics-system.uptime-default, metrics-elastic_agent.elastic_agent-default, metrics-windows.service-default, metrics-elastic_agent.metricbeat-default, metrics-system.fsstat-default, metrics-system.process-default, metrics-elastic_agent.filebeat-default, metrics-system.network-default, metrics-system.diskio-default, metrics-system.load-default, metrics-system.socket_summary-default] that match with both data streams and regular indices are disallowed"
},
"status" : 400
}
Here is the request body :
PUT metrics-*/_alias/perso-metrics
{
"filter": {
"term": {
"agent.name" : "minecraft-server"
}
}
}
Thanks in advance for your help
Looks like some of your indices which are starting with name metrics is not data-streams and are regular indices, in Alias request you can't have both of them, if you try to create aliases separately for data-stream and regular indices it will work.
Related
ES Version : 7.10.2
I have a requirement to show index statistics, I have come across the index stats API which does fulfill my requirement.
But the issue is I don't necessarily need all the fields for a particular metric.
Ex: curl -XGET "http://localhost:9200/order/_stats/docs"
It shows response as below (omitted for brevity)
"docs" : {
"count" : 7,
"deleted" : 0
}
But I only want "count" not "deleted" field, from this.
So, in Index Stats API documentation, i came across a query param as :
fields:
(Optional, string) Comma-separated list or wildcard expressions of fields to include in the statistics.
Used as the default list unless a specific field list is provided in the completion_fields or fielddata_fields parameters
As per above when I perform curl -XGET "http://localhost:9200/order/_stats/docs?fields=count"
It throws an exception
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "request [/order/_stats/docs] contains unrecognized parameter: [fields]"
}
],
"type" : "illegal_argument_exception",
"reason" : "request [/order/_stats/docs] contains unrecognized parameter: [fields]"
},
"status" : 400
}
Am I understanding the usage of fields correctly ?
If yes/no, how can I achieve the above requirement ?
Any help is much appreciated :)
You can use the filter_path argument, like:
curl -XGET "http://localhost:9200/order/_stats?filter_path=_all.primaries.docs.count
This will return you only one field like:
{
"_all" : {
"primaries" : {
"docs" : {
"count" : 10
}
}
}
}
I was asked to migrate to data streams in elasticsearch. I am a newbie in elasticsearch, and still learning about it. Only useful article I could find: https://spinscale.de/posts/2021-07-07-elasticsearch-data-streams-explained.html#data-streams-in-kibana
Since we are using elasticsearch under basic license, I got error when I was following along with tutorial and creating a ILM policy.
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "policy [csc-stream-policy] defines the [searchable_snapshot] action but the current license is non-compliant for [searchable-snapshots]"
}
],
"type" : "illegal_argument_exception",
"reason" : "policy [csc-stream-policy] defines the [searchable_snapshot] action but the current license is non-compliant for [searchable-snapshots]"
},
"status" : 400
}
Can anyone give me an idea what else I could do to active data streams in elasticsearch? I can confirm that searchable snapshots are not supported in free license. Is there another way around it?
Thanks in advance!
I am publishing a data to elasticsearch using fluentd. It has a field Data.CPU which is currently set to string. Index name is health_gateway
I have made some changes in python code which is generating the data so now this field Data.CPU has now become integer. But still elasticsearch is showing it as string. How can I update it data type.
I tried running below commands in kibana dev tools:
PUT health_gateway/doc/_mapping
{
"doc" : {
"properties" : {
"Data.CPU" : {"type" : "integer"}
}
}
}
But it gave me below error:
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
}
],
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
},
"status" : 400
}
There is also this document which says using mutate we can convert the data type but I am not able to understand it properly.
I do not want to delete the index and recreate as I have created a visualization based on this index and after deleting it will also be deleted. Can anyone please help in this.
The short answer is that you can't change the mapping of a field that already exists in a given index, as explained in the official docs.
The specific error you got is because you included /doc/ in your request path (you probably wanted /<index>/_mapping), but fixing this alone won't be sufficient.
Finally, I'm not sure you really have a dot in the field name there. Last I heard it wasn't possible to use dots in field names.
Nevertheless, there are several ways forward in your situation... here are a couple of them:
Use a scripted field
You can add a scripted field to the Kibana index-pattern. It's quick to implement, but has major performance implications. You can read more about them on the Elastic blog here (especially under the heading "Match a number and return that match").
Add a new multi-field
You could add a new multifield. The example below assumes that CPU is a nested field under Data, rather than really being called Data.CPU with a literal .:
PUT health_gateway/_mapping
{
"doc": {
"properties": {
"Data": {
"properties": {
"CPU": {
"type": "keyword",
"fields": {
"int": {
"type": "short"
}
}
}
}
}
}
}
}
Reindex your data within ES
Use the Reindex API. Be sure to set the correct mapping on the target index.
Delete and reindex everything from source
If you are able to regenerate the data from source in a timely manner, without disrupting users, you can simply delete the index and reingest all your data with an updated mapping.
You can update the mapping, by indexing the same field in multiple ways i.e by using multi fields.
Using the below mapping, Data.CPU.raw will be of integer type
{
"mappings": {
"properties": {
"Data": {
"properties": {
"CPU": {
"type": "string",
"fields": {
"raw": {
"type": "integer"
}
}
}
}
}
}
}
}
OR you can create a new index with correct index mapping, and reindex the data in it using the reindex API
I've got an Elastic index transactions-internal and would like to point all the names like transactions-([a-z]+)-internal to this index using alias, so all the requests like
GET /transactions-a-internal/_search
GET /transactions-b-internal/_search
GET /transactions-c-internal/_search
...
etc
should give the same result as
GET /transactions-internal/_search
I've tried
POST /transactions-internal/_alias/transactions-*-internal
but it returned
Invalid alias name [...] must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]
Is there any "smart" solution for that? I would strongly prefer co configure it on Elastic side, not anywhere else.
You're almost there. It's the other way around, i.e. POST /<index>/_alias/<alias>
POST /transactions-*-internal/_alias/transactions-internal
UPDATE:
If you want the other way around, then you can use the following (note that an alias name cannot contain wildcard characters):
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "transactions-internal", "alias" : "transactions-a-internal" } },
{ "add" : { "index" : "transactions-internal", "alias" : "transactions-b-internal" } },
{ "add" : { "index" : "transactions-internal", "alias" : "transactions-c-internal" } }
]
}
Not quite sure if this is applicable to your situation but if you're starting from scratch a possible solution might be to use a index template.
PUT _index_template/transactions-internal
{
"priority": 1,
"template": {
"aliases": {
"transactions-internal": {}
}
},
"index_patterns": [
"transactions-*-internal"
],
"composed_of": []
}
As I'm quite new to elastic I don't know if this template can be applied to an existing index.But this approach will work for new indizes in v 7.12.1
I am going to define mapping with nested fields. according to this documentation, payload to /order-statistics/_mapping/order looks like:
{
"mappings" : {
"order": {
"properties" : {
"order_no" : {
"type" : "string"
},
"order_products" : {
"type" : "nested",
"properties" : {
"order_product_no" : {
"type" : "int"
},
"order_product_options" : {
"type" : "nested",
"properties" : {
"order_product_option_no" : {
"type" : "int"
}
}
}
}
}
}
}
}
}
I've already created the order-statistics index with a call to curl -XPUT 'localhost:9200/order-statistics' and I'm using predefined types such as int, string, double, But I get the following error and can't find what wrong with.
{
"error":{
"root_cause":[
{
"type":"mapper_parsing_exception",
"reason":"Root mapping definition has unsupported parameters: [mappings : {order={properties={order_no={type=string}, order_products={type=nested, properties={order_product_no={type=int}, order_product_options={type=nested, properties={order_product_option_no={type=int}}}}}}}}]"
}
],
"type":"mapper_parsing_exception",
"reason":"Root mapping definition has unsupported parameters: [mappings : {order={properties={order_no={type=string}, order_products={type=nested, properties={order_product_no={type=int}, order_product_options={type=nested, properties={order_product_option_no={type=int}}}}}}}}]"
},
"status":400
}
could someone explain why this not work?
You are using int as type for some fields which is not a valid type in either 2.x or 5.x. For integer values, please use integer or long depending on the values you want to store. For details, please see the docs on core mapping types.
Which version of elasticsearch are you using - 2.x or 5.x? If you are on 5.x already, you should go with keyword or text for your string fields instead of using just string which was the naming up to 2.x. But this is still only a warning.
Additionally, you should be aware of the implications when using nested instead of just object. Using a nested type is necessary if you store an array of objects and want to query for more than one property of such an object with the guarantee that only these documents match where one of the nested objects in the array matches all your conditions. But this comes at a cost, so consider using the simple object type, if this works for you. For more details, please see the docs on nested data type and especially the warning at the end.