Execute this request
PUT _ingest/pipeline/add_test_pipeline
{
"description": "A description for your pipeline",
"processors": [
{
"set": {
"field": "fieldname",
"value": "1"
}
}
]
}
and get error:
{
"error" : {
"root_cause" : [
{
"type" : "parse_exception",
"reason" : "request body or source parameter is required"
}
],
"type" : "parse_exception",
"reason" : "request body or source parameter is required"
},
"status" : 400
}
request works on one cluster but does not on another.
From discuss.elastic.co
this page from google can not open.
When you issue the PUT command to create a new document using your pipeline, you need to specify the document int he request body:
PUT filebeat-7.6.2-2020.05.02-000001/_doc/ny1z03EBxpMbNnRZgGlQ?pipeline= add_test_pipeline
...will not work and produce the error you're seeing because you're not sending any document in the body. The pipeline alone will not create a document out of the blue, you need to provide one, even if it's empty like this:
PUT filebeat-7.6.2-2020.05.02-000001/_doc/ny1z03EBxpMbNnRZgGlQ?pipeline= add_test_pipeline
{
}
Related
When I try to initially POST data in index firs time
POST /my-test-index1/_doc/
{
"foo": {
"bar": [ "99.31.99.33", 862 ]
}
}
i receive error
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "mapper [user.id] cannot be changed from type [long] to [text]"
}
],
"type" : "illegal_argument_exception",
"reason" : "mapper [user.id] cannot be changed from type [long] to [text]"
},
"status" : 400
}
But if I initially post json with quotes numbers it's work, and next numbers without quotes also be work.
POST /my-test-index1/_doc/
{
"foo": {
"bar": [ "99.31.99.33", "862" ]
}
}
{
"_index" : "my-test-index1",
"_type" : "_doc",
"_id" : "OiyhdIABdQBSvDJuTJ4t",
"_version" : 1,
"result" : "created",
}
I know that initially post create mapping, but my question, why numbers in json without quotes doesn't work, not create right mapping when initially post
In first scenario, You are getting exception because array values are not same data type and index mapping is not created. You can check official documentation.
In Elasticsearch, there is no dedicated array data type. Any field can
contain zero or more values by default, however, all values in the
array must be of the same data type.
In second scenario, when initially post json with quotes numbers it's work because it will create index and set text type for bar field. Also, when you send other request with numbers without quotes, it will work as it consider all value of array as text and not integer.
You can use below API to check index mapping:
GET my-test-index1/_mapping
I'm trying to create a pipeline in Elasticsearch.
I have to make a check before to know what pipeline to use, so I've made a pipeline that do the routing, like this :
{
"description": "A pipeline of pipelines for log files",
"version": 1,
"processors": [
{
"pipeline": {
"if": """ctx.applicative_code =~ /AFA[#]959[#]333-SERVICE[#]1.0.0.SIG/ """,
"name": "ged_pipeline"
}
},
{
"fail": {
"message": "This pipeline requires applicative_code to be one of: 'AFA#959#333-SERVICE#1.0.0.SIG'"
}
}
]
}
To test it, I use the _simulate API :
POST _ingest/pipeline/logs_pipeline/_simulate
{
"docs": [
{
"_source": {
"message_log": "127.0.0.1:50613||Download||agent||S0000||PM000||Q||{5C7A4600-C422-4D81-BD02-39072E06F646}",
"applicative_code": "AFA#959#333-SERVICE#1.0.0.SIG"
}
}
]
}
And the response is this :
"error" : {
"root_cause" : [
{
"type" : "exception",
"reason" : "java.lang.IllegalArgumentException: org.elasticsearch.ingest.common.FailProcessorException: This pipeline requires applicative_code to be one of: 'AFA#959#333-SERVICE#1.0.0.SIG'",
"header" : {
"processor_type" : "fail"
}
}
],
"type" : "exception",
"reason" : "java.lang.IllegalArgumentException: org.elasticsearch.ingest.common.FailProcessorException: This pipeline requires applicative_code to be one of: 'AFA#959#333-SERVICE#1.0.0.SIG'",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "org.elasticsearch.ingest.common.FailProcessorException: This pipeline requires applicative_code to be one of: 'AFA#959#333-SERVICE#1.0.0.SIG'",
"caused_by" : {
"type" : "fail_processor_exception",
"reason" : "This pipeline requires applicative_code to be one of: 'AFA#959#333-SERVICE#1.0.0.SIG'"
}
So, it seems that my condition doesn't match.
The reason why i'm using a regex here is that the special character "#" mess up the string comparison. What i've tried so far, in the if condition of the pipeline :
"if": """ctx.applicative_code == "AFA#959#333-SERVICE#1.0.0.SIG" """
Result : doesn't work, the # is interpreted and I can't escape it using \
"if": """ctx.applicative_code.compareTo("AFA#959#333-SERVICE#1.0.0.SIG") """
Result : Same thing, the # is interpreted
Any idea ?
Well, you're almost there, but you are misunderstanding how pipelines work. The fact that the first pipeline processor is matched and does the job, doesn't prevent the second one (i.e. fail) to run. In your above pipeline, both steps always run, hence why you see the error.
What you need to do is to run the fail step only if the condition doesn't match (sort of else condition), like this:
{
"description": "A pipeline of pipelines for log files",
"version": 1,
"processors": [
{
"pipeline": {
"if": """ctx.applicative_code == 'AFA#959#333-SERVICE#1.0.0.SIG' """,
"name": "ged_pipeline"
}
},
{
"fail": {
"if": """ctx.applicative_code != 'AFA#959#333-SERVICE#1.0.0.SIG' """,
"message": "This pipeline requires applicative_code to be one of: 'AFA#959#333-SERVICE#1.0.0.SIG'"
}
}
]
}
I'm new to Elasticsearch. I have successfully installed Elasticsearch with Kibana, X-pack and ingest-attachment. I have both Elasticsearch and Kibana running. I have kept it simple at the moment with the install using default options on a windows 2012 server. I have a directory on another drive w\mydocs and at the moment it just has 3 plain text files in it, but I will want to add others like pdf and doc file types. So now I want to get these files into Elasticsearches index. I have tried using the following link as a guide Attaching pdf docs in Elasticsearch, however I cannot get it to work.
Here's how I have set up the index and pipeline:
PUT _ingest/pipeline/docs
{
"description": "documents",
"processors" : [
{
"attachment" : {
"field": "data",
"indexed_chars" : -1
}
}]
}
PUT myindex
{
"mappings" : {
"documents" : {
"properties" : {
"attachment.data" : {
"type": "text",
"analyzer": "standard"
}
}
}
}
}
Then to get the first document in I use the following:
PUT localhost:9200/documents/1?pipeline=docs -d #/w/mydocs/README.TXT
and the error that I receive is:
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "request body is required"
}
],
"type": "parse_exception",
"reason": "request body is required"
},
"status": 400
}
you still have to send valid JSON to Elasticsearch, even when indexing binary data. This means, that you have to encode your document as base64 and then put it into a JSON document like this
{
"data" : "base64encodedcontentofyourfile"
}
I was advised not to use the ingest-attachment, but instead to use FsCrawler. I managed to get Fscrawler working without having to convert anything to base64.
I'm trying to remove the alias my_alias on ElasticSearch 2.3 as:
curl -XPOST 'http://localhost:9200/_aliases' -d '
{
"actions" : [
{ "remove" : { "index" : "*", "alias" : "my_alias" } }
]
}'
However, this alias can not exist. In such case, I get the error:
{
"error": {
"reason": "aliases [my_alias] missing",
"resource.id": "my_alias",
"resource.type": "aliases",
"root_cause": [
{
"reason": "aliases [my_alias] missing",
"resource.id": "my_alias",
"resource.type": "aliases",
"type": "aliases_not_found_exception"
}
],
"type": "aliases_not_found_exception"
},
"status": 404
}
I tried adding ignore_unavailable to the action as { "remove" : { "index" : "*", "alias" : "my_alias", "ignore_unavailable": true } }, or simply ignore: 404, but to no avail. I looked at ElasticSearch's test suite for update_aliases in https://github.com/elastic/elasticsearch/tree/7560101ec73331acb39a042bde6130e59e4bb630/rest-api-spec/src/main/resources/rest-api-spec/test/indices.update_aliases, but couldn't find a test that did this.
I'm starting to think that there's no way to do that. Am I wrong?
I don't think this is possible. There is no delete "if exists" and that is somewhat expected from any HTTP API.
However, you can definitely ignore the response for this particular case in your code instead of treating it as an error.
hi I want to set custom routing on specific field "userId" on my Es v2.0.
But it giving me error.I don't know how to set custom routing on ES v2.0
Please guys help me out.Thanks in advance.Below is error message, while creating custom routing with existing index.
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [_routing] has unsupported parameters: [path : userId]"
}
],
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [_routing] has unsupported parameters: [path : userId]"
},
"status": 400
}
In ES 2.0, the _routing.path meta-field has been removed. So now you need to do it like this instead:
In your mapping, you can only specify that routing is required (but you cannot specify path anymore):
PUT my_index
{
"mappings": {
"my_type": {
"_routing": {
"required": true
},
"properties": {
"name": {
"type": "string"
}
}
}
}
}
And then when you index a document, you can specify the routing value in the query string like this:
PUT my_index/my_type/1?routing=bar
{
"name": "foo"
}
You can still use custom routing based on a field from the data being indexed. You can setup a simple pipeline and then use the pipeline every time you index a document, or you can also change the index settings to use the pipeline whenever the index receives a document indexing request.
Read about the pipeline here
Do read above and below the docs for more clarity. It's not meant for setting custom routing, but can be used for the purpose. Custom routing was disabled for a good reason that the field to be used can turn out to have null values leading to unexpected behavior. Hence take care of that issue by yourself.
For routing, here is a sample pipeline PUT :
PUT localhost:9200/_ingest/pipeline/nameRouting
Content-Type: application/json
{
"description" : "Set's the routing value from name field in document",
"processors" : [
{
"set" : {
"field": "_routing",
"value": "{{_source.name}}"
}
}
]
}
The index settings will be :
{
"settings" : {
"index" : {
"default_pipeline" : "nameRouting"
}
}
}