How split a field to words by ingest pipeline in Kibana - elasticsearch

Question

I have created an ingest pipeline as below to split a field into words:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": "String cutting processing",
"processors": [
{
"split": {
"field": "foo",
"separator": "|"
}
}
]
},
"docs": [
{
"_source": {
"foo": "apple|time"
}
}
]
}
but it split the field into characters:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
"foo" : [
"a",
"p",
"p",
"l",
"e",
"|",
"t",
"i",
"m",
"e"
]
}
}
}
]
}
If I replace the separator with a comma, the same pipeline split the field to words:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": "String cutting processing",
"processors": [
{
"split": {
"field": "foo",
"separator": ","
}
}
]
},
"docs": [
{
"_source": {
"foo": "apple,time"
}
}
]
}
then the output would be:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
"foo" : [
"apple",
"time"
]
}
}
}
]
}
How can I split the field into words when the separator is "|"?
My next question is how could I apply this ingest pipeline to an existing index?
I tried this solution, but it doesn't work for me.
Edit
Here is the whole pipeline with the document that will assign two parts to two columns:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": """combined fields are text that contain "|" to separate two fields""",
"processors": [
{
"split": {
"field": "dv_m",
"separator": "|",
"target_field": "dv_m_splited"
}
},
{
"set": {
"field": "dv_metric_prod",
"value": "{{dv_m_splited.1}}",
"override": false
}
},
{
"set": {
"field": "dv_metric_section",
"value": "{{dv_m_splited.2}}",
"override": false
}
}
]
},
"docs": [
{
"_source": {
"dv_m": "amaze_inc|Understanding"
}
}
]
}
That generates this response:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
"dv_metric_prod" : "m",
"dv_m_splited" : [
"a",
"m",
"a",
"z",
"e",
"_",
"i",
"n",
"c",
"|",
"U",
"n",
"d",
"e",
"r",
"s",
"t",
"a",
"n",
"d",
"i",
"n",
"g"
],
"dv_metric_section" : "a",
"dv_m" : "amaze_inc|Understanding"
},
"_ingest" : {
"timestamp" : "2021-08-02T08:33:58.2234143Z"
}
}
}
]
}
If I set "separator": "\\|", then I will get this error:
{
"docs" : [
{
"error" : {
"root_cause" : [
{
"type" : "general_script_exception",
"reason" : "Error running com.github.mustachejava.codes.DefaultMustache#776f8239"
}
],
"type" : "general_script_exception",
"reason" : "Error running com.github.mustachejava.codes.DefaultMustache#776f8239",
"caused_by" : {
"type" : "mustache_exception",
"reason" : "Failed to get value for dv_m_splited.2 #[query-template:1]",
"caused_by" : {
"type" : "mustache_exception",
"reason" : "2 #[query-template:1]",
"caused_by" : {
"type" : "index_out_of_bounds_exception",
"reason" : "2"
}
}
}
}
}
]
}

Answer 1

The solution is fairly simple: just escape your separator.
As the separator field in the split processor is a regular expression, you need to escape special characters such as |.
You also need to escape it twice.
So your code only lacks the double escaping part:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": "String cutting processing",
"processors": [
{
"split": {
"field": "foo",
"separator": "\\|"
}
}
]
},
"docs": [
{
"_source": {
"foo": "apple|time"
}
}
]
}
UPDATE
You did not mention or I missed the part where you wanted to assign the values to two separate fields.
In this case, you should use dissect instead of split. It is shorter, simpler, cleaner. See the documentation here.
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": """combined fields are text that contain "|" to separate two fields""",
"processors": [
{
"dissect": {
"field": "dv_m",
"pattern": "%{dv_metric_prod}|%{dv_metric_section}"
}
}
]
},
"docs": [
{
"_source": {
"dv_m": "amaze_inc|Understanding"
}
}
]
}
Result
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
"dv_metric_prod" : "amaze_inc",
"dv_metric_section" : "Understanding",
"dv_m" : "amaze_inc|Understanding"
},
"_ingest" : {
"timestamp" : "2021-08-18T07:39:12.84910326Z"
}
}
}
]
}
ADDENDUM
If using split instead of dissect
You got your array indices wrong. There is no such thing as {{dv_m_splited.2}} as the array index starts from 0 and you only have two results.
This is the correct pipeline when using the split processor.
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": """combined fields are text that contain "|" to separate two fields""",
"processors": [
{
"split": {
"field": "dv_m",
"separator": "\\|",
"target_field": "dv_m_splited"
}
},
{
"set": {
"field": "dv_metric_prod",
"value": "{{dv_m_splited.0}}",
"override": false
}
},
{
"set": {
"field": "dv_metric_section",
"value": "{{dv_m_splited.1}}",
"override": false
}
}
]
},
"docs": [
{
"_source": {
"dv_m": "amaze_inc|Understanding"
}
}
]
}