I would like to find the datetime for 1 day ago so that I can create link to kibana in an email sent from the watcher. Using Elasticsearch 5.0.2
I've tried the watch below but it returns an error of
ScriptException[runtime error]; nested: IllegalArgumentException[Unable to find dynamic method [minusDays] with [1] arguments for class [org.joda.time.DateTime].];
minusDays does exist in the joda DateTime spec
but it doesn't exist in the elastic codebase
here's the watch
PUT /_xpack/watcher/watch/errors-prod
{
"trigger": {
"schedule": {
"daily": {
"at": [
"08:36"
]
}
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"<das-logstash-{now}>",
"<das-logstash-{now-1d}>"
],
"types": [
"redis-input"
],
"body": {
"size": 0,
"query": {
"match_all": {}
}
}
}
}
},
"actions": {
"send_email": {
"transform": {
"script" : "return [ 'from' : ctx.trigger.scheduled_time.minusDays(1) ]"
},
"email": {
"profile": "standard",
"from": "noreply#email.com",
"to": [
"me#email.com"
],
"subject": "errors",
"body": {
"html": "<html><body><p>from {{ctx.payload.from}}</p><p>to {{ctx.trigger.scheduled_time}}</p></body></html>"
}
}
}
}
}
I needed something similar and was able to hack this together by modifying a comment that almost worked from an elastic forum.
"transform": {
"script" : {
"source" : "def payload = ctx.payload; DateFormat df = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\"); ctx.payload.from = df.format(Date.from(Instant.ofEpochMilli(ctx.execution_time.getMillis() - (24 * 60 * 60 * 1000) ))); return payload"
}
},
Hope that helps!
Related
I want to use life cycle management, the goal is to delete messages after 14 days
What should be the action in the first stage? Open or Read_write
What is the difference between the two actions?
{
"policy": {
"policy_id": "delete_after14_days",
"description": "index delete"
"schema_version": 1,
"error_notification": null,
"default_state": "open",
"states": [
{
"name": "hot",
"actions": [
{
**"open": {} or "read_write": {}**
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "14d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"audit-*"
],
"priority": 0
}
]
}
}
I am trying to create a Distill alert and have used the field selector to create the following code. I am trying to exclude the following field from the page comparison:
/div[contains(#class,'MUxGbd')]/span[contains(#class,'MUxGbd')]
This code is the same throughout the page, but the path changes for each result.
How can I modify the code to be 'any' path - for example:
//div[#id='rso']/**any**/div[contains(#class,'MUxGbd')]/span[contains(#class,'MUxGbd')]
Thanks,
{
"selections": [
{
"frames": [
{
"index": 0,
"excludes": [
{
"type": "xpath",
"expr": "//div[#id='rso']/div[#class='hlcw0c']/div[#class='g']//div[#class='tF2Cxc']/div[#class='IsZvec']/div[contains(#class,'MUxGbd')]/span[contains(#class,'MUxGbd')]"
},
{
"type": "xpath",
"expr": "//div[#id='rso']/div[#class='g']//div[#class='tF2Cxc']/div[#class='IsZvec']/div[contains(#class,'MUxGbd')]/span[contains(#class,'MUxGbd')]"
}
],
"includes": [
{
"type": "xpath",
"expr": "//div[#id='rso']"
}
]
}
],
"dynamic": true,
"delay": 0
}
],
"ignoreEmptyText": true,
"includeStyle": false,
"dataAttr": "text"
}
have you tried
"//div[#id='rso']//div[contains(#class,'MUxGbd')]/span[contains(#class,'MUxGbd')]"
?
I have documents in elasticsearch in which each document looks something like as follows:
{
"id": "T12890ADSA12",
"status": "ENDED",
"type": "SAMPLE",
"updatedAt": "2020-05-29T18:18:08.483Z",
"events": [
{
"event": "STARTED",
"version": 1,
"timestamp": "2020-04-30T13:41:25.862Z"
},
{
"event": "INPROGRESS",
"version": 2,
"timestamp": "2020-05-14T17:03:09.137Z"
},
{
"event": "INPROGRESS",
"version": 3,
"timestamp": "2020-05-17T17:03:09.137Z"
},
{
"event": "ENDED",
"version": 4,
"timestamp": "2020-05-29T18:18:08.483Z"
}
],
"createdAt": "2020-04-30T13:41:25.862Z"
}
Now, I wanted to write a query in elasticsearch to get all the documents which are of type "SAMPLE" and I can get the average time between STARTED and ENDED of all those documents. Eg. Avg of (2020-05-29T18:18:08.483Z - 2020-04-30T13:41:25.862Z, ....). Assume that STARTED and ENDED event is present only once in events array. Is there any way I can do that?
You can do something like this. The query selects the events of type SAMPLE and status ENDED (to make sure there is a ENDED event). Then the avg aggregation uses scripting to gather the STARTED and ENDED timestamps and subtracts them to return the number of days:
POST test/_search
{
"query": {
"bool": {
"filter": [
{
"term": {
"status.keyword": "ENDED"
}
},
{
"term": {
"type.keyword": "SAMPLE"
}
}
]
}
},
"aggs": {
"duration": {
"avg": {
"script": "Map findEvent(List events, String type) {return events.find(it -> it.event == type);} def started = Instant.parse(findEvent(params._source.events, 'STARTED').timestamp); def ended = Instant.parse(findEvent(params._source.events, 'ENDED').timestamp); return ChronoUnit.DAYS.between(started, ended);"
}
}
}
}
The script looks like this:
Map findEvent(List events, String type) {
return events.find(it -> it.event == type);
}
def started = Instant.parse(findEvent(params._source.events, 'STARTED').timestamp);
def ended = Instant.parse(findEvent(params._source.events, 'ENDED').timestamp);
return ChronoUnit.DAYS.between(started, ended);
I have the following data:
{
"_id": ObjectID("5e2fa881c3a1a70006c5743c"),
"name": "Some name",
"policies": [
{
"cId": "dasefa-2738-4cf0-90e0d568",
"weight": 12
},
{
"cId": "c640ad67dasd0-92f981583568",
"weight": 50
}
]
}
I'm able to query this with Spring Mongo fine, however I want to be able to order the policies by weight
At the moment I get my results fine with:
return mongoTemplate.find(query, CArea::class.java)
However say I make the following aggregations:
val unwind = Aggregation.unwind("policies")
val sort = Aggregation.sort(Sort.Direction.DESC,"policies.weight")
How can I go and actually apply those to the returned results above? I was hoping that the dot annotation would do the job in my query however didnt do anything e.g. Query().with(Sort.by(options.sortDirection, "policies.weight"))
Any help appreciated.
Thanks.
I am not familier with Spring Mongo, but I guess you can convert the following aggregation to spring code.
db.collection.aggregate([
{
$unwind: "$policies"
},
{
$sort: {
"policies.weight": -1
}
},
{
$group: {
_id: "$_id",
"policies": {
"$push": "$policies"
},
parentFields: {
$first: "$$ROOT"
}
}
},
{
$replaceRoot: {
newRoot: {
$mergeObjects: [
"$parentFields",
{
policies: "$policies"
}
]
}
}
}
])
This will result:
[
{
"_id": "5e2fa881c3a1a70006c5743c",
"name": "Some name",
"policies": [
{
"cId": "c640ad67dasd0-92f981583568",
"weight": 50
},
{
"cId": "dasefa-2738-4cf0-90e0d568",
"weight": 12
}
]
}
]
Playground
I am using ES 5.2. I implemented a watcher.But each time watcher is getting triggered it generate email but google blocks that email due to security concern. So what can be solution for that?
My YML file is as below :
cluster.name: elasticsearch-logging
node.name: "elasticsearch-logging-0"
path.data: /var/lib/elasticsearch/data
xpack.notification.email.account:
gmail_account:
profile: gmail
smtp:
auth: true
starttls.enable: true
host: smtp.gmail.com
port: 587
user: ******.**#gmail.com
password: ******
While doing curl on watcher getting below response :
DOING CURL --
curl -XGET localhost:9200/_xpack/watcher/watch/last_watch
Getting below response:
{
"found": true,
"id": "lastwatch",
"status": {
"version": 5,
"state": {
"active": true,
"timestamp": "2017-06-16T00:39:16.654Z"
},
"lastchecked": "2017-06-16T00:43:00.229Z",
"last_met_condition": "2017-06-16T00:43:00.229Z",
"actions": {
"email_admin": {
"ack": {
"timestamp": "2017-06-16T00:39:16.654Z",
"state": "awaits_successful_execution"
},
"last_execution": {
"timestamp": "2017-06-16T00:43:00.229Z",
"successful": false,
"reason": "MessagingException[failed to send email with subject [404 recently encountered] via account [gmail_account]]; nested: AuthenticationFailedException[534-5.7.14 https://accounts.google.com/signin/continue?sarp=1&scc=1&pltn534-5.7.14 q0WEdpll7GFx7wL5ZoIKlaHy0JIWKkJEAaiNf5hWY11ZPPsJb6u7h9z0Xe\n534-5.7.14 kWiT264a1EJgbKW5ESeccxI0uUZ_3X4klQS4jBjB7dDw6pRU490p-yKtXkL2-Ik\n534-5.7.14 vMoQFBgYsmH2WbbGFC3Z63GBpWVH0O9LmpVsB89ZsSreIXN_bb0AX3UWwoX4dTb4UiXtmi\nQI Please log in via your web browser and\n534-5.7.14 then try again.\n534-5.7.14 Learn more at\n534 5.7.14 https://support.google.com/mail/answer/78754 a22sm752699pfc.115 - gsmtp\n]; "
}
}
}
},
"watch": {
"trigger": {
"schedule": {
"cron": "0 0/1 * * * ?"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logstash*"
],
"types": [],
"body": {
"query": {
"bool": {
"must": {
"match": {
"methodName": "getSSLConnectionSocketFactory"
}
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"email_admin": {
"email": {
"profile": "standard",
"to": [
"****.*****#gmail.com"
],
"subject": "404 recently encountered"
}
}
}
}
}
Looks like a javax.mail issue and you need to turn on less secure apps.