I am developing a webapp, for which I am pushing my search templates to ES during startup and using them to form the elastic search queries at runtime. I have a requirement wherein, I don't know the number of filters to be applied. Created a search template like -
{
"filters" : {
{{#toJson}}
clauses
{{/toJson}}"
}
}
And search will be made like this -
GET _search/template
{
"id": "template-id",
"params": {
"clauses": {
"filters" : {
{ "match": { "user" : "foo" } },
{ "match": { "user" : "bar" } }
}
}
}
which will render result as -
{
"filters":{
"filters":{
"match" : {
"user" : "foo"
}
},
{
"match" : {
"user" : "bar"
}
}
}
}
as suggested by ES documentation-
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html
But, since it's an invalid JSON, it doesn't allow me to push the template to ES.
My template works well when I use it as stored template in elastic-home/config/scripts. But I want to manage my templates with JAVA and push all templates during startup only.
Can I get any help?
Related
Is it possible to trigger an action for each hit of a given query in a Kibana Monitor? I would like to use a foreach loop to do this as demonstrated here. However, it's unclear how to implement this on the Kibana Monitor page. On the page there is an input field for Trigger Conditions but I'm unsure how to format the foreach within it or if this is supported.
Consider using Elasticsearch watcher (require at least gold licesnse): https://www.elastic.co/guide/en/elasticsearch/reference/current/how-watcher-works.html
Watcher will run on a certain interval and will perform a query against indices (according to your configuration). You will need to create a condition (e.g. hits number is greater than 5) that when it evaluates to true an action will be performed. Elasticsearch allows you to use multiple actions. For example, you can use webhook and receive the data from the last watcher run (you can also use watcher api to transform the data). If you don't have Gold license you can mimic watcher behavior by a script/program that uses Elasticsearch Search API.
Herbeby is a simple example of a watcher checking index named test every minute and sends a webhook with the entire search context in case there is at least one document.
{
"trigger" : {
"schedule" : { "interval" : "1m" }
},
"input" : {
"search" : {
"request" : {
"indices" : [ "test" ],
"body" : {
"query" : {
"bool": {
"must": {
"range": {
"updatedAt": {
"gte": "now-1m"
}
}
}
}
}
}
}
}
},
"condition" : {
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
},
"actions" : {
"sample_webhook" : {
"webhook" : {
"method" : "POST",
"url": "http://b4022015b928.ngrok.io/UtilsService/api/elasticHandler/watcher",
"body" : "{{#toJson}}ctx.payload{{/toJson}}",
"auth": {
"basic": {
"user": "user",
"password": "pass"
}
}
}
}
}
}
An alternative way would be to use Kibana Alerts and Actions.
https://www.elastic.co/guide/en/kibana/current/alerting-getting-started.html
This feature is slightly different from Watcher but basically allows you to perfrom actions upon a query against Elasticsearch. This featrue is only part of Kibana opposing to watcher which is part of Elasticsearch (though it is accessible from Kibana stack management).
I am trying to create a Slack action with a dynamic attachment. My _source looks like this:
{
"user.url": "https://api.github.com/users/...",
"user.gists_url": "https://api.github.com/users/.../gists{/gist_id}",
"user.repos_url": "https://api.github.com/users/.../repos",
"date": "2018-04-27T14:34:10Z",
"user.followers_url": "https://api.github.com/users/.../followers",
"user.following_url": "https://api.github.com/users/.../following{/other_user}",
"user.id": 123456,
"user.avatar_url": "https://avatars0.githubusercontent.com/u/123456?v=4",
"user.events_url": "https://api.github.com/users/.../events{/privacy}",
"user.site_admin": false,
"user.html_url": "https://github.com/...",
"user.starred_url": "https://api.github.com/users/.../starred{/owner}{/repo}",
"user.received_events_url": "https://api.github.com/users/.../received_events",
"metric": "stars",
"user.login": "...",
"user.type": "User",
"user.subscriptions_url": "https://api.github.com/users/.../subscriptions",
"user.organizations_url": "https://api.github.com/users/.../orgs",
"user.gravatar_id": ""
}
and here is my Slack action
"actions": {
"notify-slack": {
"throttle_period_in_millis": 240000,
"slack": {
"account": "monitoring",
"message": {
"from": "Elasticsearch Watcher",
"to": [
"#watcher"
],
"text": "We have {{ctx.payload.new.hits.total}} new stars! And {{ctx.payload.old.hits.total}} in total.",
"dynamic_attachments" : {
"list_path" : "ctx.payload.new.hits.hits",
"attachment_template" : {
"title" : "{{_source.[\"user.login\"]}}",
"text" : "Users Count: {{count}}",
"color" : "{{color}}"
}
}
}
}
}
I can't seem to figure out how to access my _source fields since they have dots in them. I have tried:
"{{_source.[\"user.login\"]}}"
"{{_source.user.login}}"
"{{_source.[user.login]}}"
"{{_source.['user.login']}}"
The answer to my question is that you can't access _source keys with dots in them directly using mustache, you must first transform your data.
Update:
I was able to get this working by using a transform to build a new object. Mustache might not be able to access fields with dots in their names, but painless can! I added this transform to my slack object:
"transform" : {
"script" : {
"source" : "['items': ctx.payload.new.hits.hits.collect(user -> ['userName': user._source['user.login']])]",
"lang" : "painless"
}
}
and now in the slack action dynamic attachments, I can access the items array:
"dynamic_attachments" : {
"list_path" : "ctx.payload.items",
"attachment_template" : {
"title" : "{{userName}}",
"text" : "{{_source}}"
}
}
Old Answer:
So according to this Watcher uses mustache.
and according to this mustache can't access fields with dots in the names.
I'm building app with elasticsearch (5.4) and everything was going well until I try to delete several documents by field values. My x-ndjson looks like this:
{ "delete" : {} }
{ "id" : "109991" }
{ "delete" : {} }
{ "id" : "109992" }
{ "delete" : {} }
{ "id" : "109993" }
<- empty line
and i am POSTing it on http://localhost:9200/someindex/sometype/_bulk, but it responds with "Malformed action/metadata line [2], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]".
Note that my "id" is my custom field, not the _id.
Is something missing in my request?
Thank you
I guess you need to use Delete By Query for this.
POST index/_delete_by_query
{
"query": {
"terms": {
"id": [
109991,
109992
]
}
}
}
I have the currenct structure:
localhost:9200/objects/content
{
id:1,
author:{
name:"john"
},
body:"abc"
}
localhost:9200/objects/reaction
{
content_id:1
message:'I like it'
}
How can I query to get all reactions of contents writed by "john"?
This means a query on reactions, checking the content specified by id, if author is "someone".
This Elasticsearch blog post describes how to manage relationships inside Elasticsearch.
You will need to set a parent mapping between your reactions and your content.
{
"reaction" : {
"_parent" : {
"type" : "content"
}
}
}
You will then index your reaction as a child of content id 1:
curl -XPOST localhost:9200/test/homes?parent=1 -d'
{
message:'I like it'
}
You can then use a Has Parent Query to retrieve all reactions to authors named john:
{
"has_parent" : {
"parent_type" : "content",
"query" : {
"term" : {
"author.name" : "john"
}
}
}
}
I would recommend you to add some redundancy to your model:
localhost:9200/objects/reaction
{
content_id:1
message:'I like it'
author_name:'john'
}
This will increase the index, of course. On the other hand the query to get all reactions of contents writed by "john" will be simple and fast.
I want to create an alias on top of this. Index - test, Type - type
POST /test/type/_search
{
"query": {
"match": {
"brand_name": "xyz"
}
}
}
But I don't see anyway of doing it,since Elasticsearch aliases can only be created on filters and when I try with term filter,I don't get the results which I want.Any trick to achieve this ?
You can use a query filter to use any query as a filter:
"filter" : {
"query" : {
"match" : {
"brand_name" : "xyz"
}
}
}