How do I throttle an Elasticsearch Watcher to one alert per event. If an error shows in the logs the watcher will fire until it's acknowledged.
I only want alerts for new errors not a time throttle.
I have found found a reasonable solution for this matter here.
I changed the interval of the watcher to 20s. Then I added a filter for records that are older than 25s.
It does only send one notification per record before it leaves that 5 second window. But I have noticed that this is isn't perfect. It does miss a couple of records but for the most part it works to a satisfactory level.
I added this just in case anyone needed it. If there is a better answer please let me know. I hope this helps.
Related
I recently encountered a thorny problem, while I am using kafkastream's TimeWindowedKStream aggregation method. The phenomenon was that I stopped my program for 5 minutes and then restarted it. I found a small part of my data was lost and got the following prompt, "Skipping record for expired window". All data are normal data that want to be saved, there is no large delay. What can I do to prevent data from being discarded ? It seems that kafkastream got a later time when it got observedstreamtime
The error message means that a window was already closed -- thus you would need to increase GRACE as pointed out by #groo. -- Data expiration is based on event-time so stopping your program and resuming is later should not change much.
However, if there is a repartition topic before the aggregation, if you stop your program for some time, there might be more out-of-order data inside the repartition topic, because the input topic is read much faster than in the "live run" -- this increased unorder during catchup could be the issue.
I've been experiencing a couple issues with Google Sheets & App Scripts lately, and I'm hoping someone can help me out. I'm going to contain this post to one topic / question, but I'm going to mention a couple of other problems that I'm having just for a greater context in case the other issues are related or may be causing the problem specific to this topic.
Problem: My custom app script functions in my Google Speadsheet are currently stuck at "Loading..." and at the time of writing this have been for over an hour. There are no logs of it being executed in MyExecution in my App Script Dashboard, nor are there any errors reported anywhere. These functions were working and have been working for the last couple months until today.
Details:
So, in my Google Sheet I'm using a custom method "findSeenCount". It's used quite a few times throughout a couple sheets, and though I don't think the logic is relevant here, I will say that its purpose was to perform a count on specific things too complex to chain the basic spreadsheet counts and conditions together. The function itself works fine, and has for several months now. However, today as I was working on a separate script (working title: newFunction), I started to notice every time I would save my script project or in the editor run -> newFunction, it would trigger all my findSeenCount functions in the sheet to run (as in it would get an entry in the Execution log), but on the sheet itself (where the calls actually were), it never actually went to recalculate. The return values stayed the same, it never changed to "Loading...", but there were executions clearly happening according to the dashboard. This was quite taxing, and strange as at the time I noticed this happening newFunction was just doing 2 simple requests to get some specific sheets, one of which was a sheet with some findSeenCount functions on it (though I've never had this issue before in some of my other functions).
function newFunction()
{
var attendanceSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Attendance");
// Sheet that contains findSeenCount.
var P1Sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("P1 Prio");
return;
}
At this point, my App Script dashboard -> My Executions page started to become pretty laggy, and eventually just started crashing. No errors were being reported and when the page would load I could see it completing the unwanted executions. It ran a couple hundred times as I was trying to work, which eventually I just removed all the findSeenCount function calls in the sheet entirely. After I did that, any function I tried to trigger would not work. In the sheet, if I called a function it would get stuck on "Loading..." and no execution records would show up on my dashboard -> My Executions. If I ran a function in the editor, it would run, the "running function" box would pop up, eventually go away, and that was it. Again, no execution record would be recorded in the dashboard -> My Executions and it would never actually return any results to the log "Waiting for logs, please wait..." if wrote in a log statement. I'm not seeing any errors anywhere, and I don't think I surpassed any sort of execution limit as I would expect to be told somewhere that I had. On top of this, I've also noticed that I have tons of executions that are recorded, I mean my application sees somewhat heavy traffic daily and has for the last several months. This alone seems to cause a little delay when simply just loading a page of the executions and I'm not sure if there's a way I can clear this list -- but that's getting off topic.
If anyone has any ideas on what's going on, or what I'm supposed to do about this, or where I can find any sort of logging that may be able to give me better details about what the problem is please let me know!
Thanks
A quick fix is to clear browser cache and cookies and clear the sheet and adding your formulas again.
A more reliable way is to ditch the custom formulas and use triggers or menus or buttons. Custom functions are unreliable at large scale.
I have set up a Grafana alert, that alerts when my values in a graph reaches a specific value.
And that value was reached a month ago and ever since then the state has been "ALERTING".
I obviously want to reset this state to "OK", since the incident happened a month ago and has been resolved.
Any idea how to do that?
I've taken some screenshots of the graph/panel and the settings for the alert, so you can see, what I've set it to.
The main goal is to get a alert everytime the values reach over 468.
Well it is obvious and I guess you just get better at Grafana when trying...
I've found out that you should simple go to the page "Alert Rules" and from there pause and then restart the alert.
When you re-enable it it will change status from "alerting" to "pending", the alert will then check after some time (check what time you've set it too) and if the query no longer calls the alert, it will change to "OK", if not - it will again send out a new alert.
Grafana should normally do this itself - but I've experienced that I've had to manually do this.
Should you not have this possibility - meaning that the query should not call the alert, but should show "OK" and it instead keeps showing "Alerting", then I couldn't find another way around than to delete the alert and set it up again. Not the best solution, but pausing and re-enabling didn't work and I haven't had any difficulties since.
If anyone have had the same experience and found a better solution, then please feel free to post it.
I can also see that this has been a bug/known error as stated here
I want to cancel all my events; I have checked the following topic -
Omnet++, How can I get list of all scheduled events of a module?
But in this link author explains how to do it in OMNeT++ for lower version.
I've also used getFES() instead of getMessageQueue(), andcFutureEventSetinstead ofcMessageHeap. But the problem is that the input value of the function {cancelAndDelete(cMessage)}and FES members are cEvent.
Thanks a lot in advance.
cause no one answer to my question I find out a way to solve my problem. I'll share it maybe it will be useful for some one else.
first point : we can't cancel events made by a module with another module;
second : it's more easier to handle our events and our schedule and CancelandDelete them whenever we want;
I need to index/update a document in Elasticsearch and wait until it is searchable (refresh has been done). There is a related issue on Github: https://github.com/elasticsearch/elasticsearch/issues/1063
I won't force the refresh because it impacts indexing performances and I will need to perform this operation really often.
I tried to wait for 1 second as described in the Github issue. It works really well as long as Elasticsearch is not under pressure, but when there is not much RAM left (which might happen occasionally) I have seen the refresh take up to 5 or 6 seconds. Thus I tried another way.
I have written an helper function in my backend that waits for the “searchable” document to reach a given version. It is quite simple:
- GET the document with realtime=false
- if there is a result
- if result.version >= wanted.version.
Return
- else
wait a little more and retry
- else if the doc is not found
- HEAD the document with realtime=true (test if the doc exists in the transaction log)
- if the doc is found (then it has just been created)
wait a little more and retry
- else
Return. (the doc might have been created and deleted really fast)
The wanted version is the version returned by elasticsearch when the document has been indexed.
This algorithm works but you can see that it is far from being perfect.
first it will make more calls to elasticsearch when it is under pressure, which is not a really good idea.
I have seen elastic search reset the version number when a doc has been deleted for some time. If for some reason the function misses that, we might wait until the doc reaches this version again. (that’s why I also added a timeout).
Does someone have a better solution? Scaling automatically is not an acceptable answer right now.
As Guillaume Massé said, a solution is about to be merged in Elasticsearch https://github.com/elastic/elasticsearch/issues/1063#issuecomment-223368867
Thus I would advise to wait for the builtin functionality rather than implementing a custom solution.