I set up a ELK environment with 2 different indices(index) streams. both streams have a field with the same value but the filed name is different.
is there a possibility to merge them or something like that so when i use the Kibana filter it shows me the value from both filed.
so i can set up visualizations, but when i filter on stream 1, the visualization of stream 2 is empty.
i also tried to name the indedx the same, but did not help.
Example:
index1 fieldname information.ID = 123
index2 fieldname ID = 123
i want to use the filter on both streams
You can create an alias which contains both indexes. Write a query against the alias using script fields.
In script fields you can define the new field and its source logic from underlying documents.
"script_fields": {
"my_script_field": {
"script": {
"lang": "painless",
"inline": "doc['some_field'].value + doc['another_field'].value"
}
In this way the resultset will have single field “my_script_field”
Related
Is there any way to find and replace the value against particular field of my index with updated value?
In my index, lets say Index name = States_of_India, "State"="Jammu & Kashmir" is populated. I want to replace the '&' sign with "and".
How to do find and replace?
Have a look at the Update by Query API, or the Update API.
For example, if you want to replace the '&' sign with 'and' in the "State" field values, you can try something like this:
POST /index-name/_update_by_query
{
"script":
{
"lang": "painless",
"inline": "ctx._source.State = ctx._source.State.replace('&', 'and')"
}
}
I need to display subtotal columns in a Kibana data table. Not filtering the entire table, but only certain columns.
I've seen posts about doing conditional counts in a metric's JSON input field:
{
"script":{
"inline": "doc['SomeField'].value == 'SomeValue' ? 1 : 0",
"lang": "painless"
}
}
But no reference to conditional sums of numeric data. My loosely expressed need:
sum(btyes) where category = [write]
Alternatively, the Kibana Enhanced Table plugin was suggested as a way to implement computed columns.
Is it possible to achieve conditional sums using JSON input on a specific data table metric? Is anyone using the plugin? Should it be done upstream in an elasticsearch index? What is best practice?
Solution is a simple change to show the actual value in the true condition, rather than a 1 for counting :
{
"script" : "doc['category.keyword'].value == 'write' ? doc['bytes'].value : 0"
}
Recently, I posted the following about adding a string to existing (inconsistent) arrays in documents: ElasticSearch query to populate or append a value to a field
The marked solution is working perfectly.
But now I need to understand how to delete one of the 5-character codes from the arrays. Assuming I now need to delete the code 'ABCDE' from the documents, while leaving the other codes in the array untouched, what would that query look like?
In below script I am looping through array and creating a list by removing the given value.
Please test before running on actual data.
{
"script": {
"source": "ctx._source.customCategories.removeAll(Collections.singleton(params.catg))",
"lang": "painless",
"params": {
"catg": "c"
}
}
}
I have multiple Documents within an Index, each have the following fields:
id serviceName Type
Now, stupidly, id is not unique and I want to change that. I want to use Kibana/Elasticsearch to query the data so that I have id unique and the behaviour I want is that if I have the following Docs:
id serviceName Type
1 A T1
1 B T2
1 D T2
I use a query so that I get this result
1 A,B,C T1,T2,T3
Is there a way for this?
You cannot do this with just Elasticsearch/Kibana, you have to write some code. You can use the scroll api to iterate through all the documents in the index, and then use an upsert query to index them into a new index. I think your upsert request will look something like this:
POST test/type1/1/_update
{
"script" : {
"inline": "ctx._source.serviceName.add(params.serviceName); ctx._source.Type.add(params.Type)",
"lang": "painless",
"params" : {
"serviceName" : "A",
"Type": "T1"
}
},
"upsert" : {
"serviceName": ["A"],
"Type": ["T1"]
}
}
This means in case id 1 doesn't exist yet, add it with the "upsert" value for the document, otherwise do the script (which appends the serviceName and Type values to the existing doc).
This would be pretty straightforward to do with very little code using elasticsearch-py, check out the scan helper and bulk helper
In Elasticsearch, I know I can specify the fields I want to return from documents that match my query using {"fields":["fieldA", "fieldB", ..]}.
But how do I return the sum of all fields that match a particular regular expression (as a new field)?
For example, if my documents look like this:
{"documentid":1,
"documentStats":{
"foo_1_1":1,
"foo_2_1":5,
"boo_1_1:3
}
}
and I want the sum of all stats that match _1_ per document?
You can define an artificial field called script_field that contains a small Groovy script, which will do the job for you.
So after your query, you can add a script_fields section like this:
{
"query" : {
...
},
"script_fields" : {
"sum" : {
"script" : "_source.documentStats.findAll{ it.key =~ '_1_'}.collect{it.value}.sum()"
}
}
}
What the script does is simply to retrieve all the fields in documentStats whose name matches _1_ and sums all their values, in this case, you'll get 4.
Make sure to enable dynamic scripting in elasticsearch.yml and restart your ES node before trying this out.