I would like to define a query in cep/siddhi that detects a missing event in a sequence.
I have 4 events that must occur within 5 minutes. (p1->p2->p3->p4 within 3000)
How can I write the condition to insert my alert event when one of the p2,P3 or P4 is missing within the 5 minutes time window after p1 occurs ?
Thanks.
Currently there's no direct way of detecting non-occurrences. However, this can be accomplished by writing a custom window and then writing a query that uses the custom window.
Please refer the link below for more info on how to write a custom window for WSO2 CEP:
http://docs.wso2.org/wiki/display/CEP210/Writing+Extentions+to+Siddhi
Related
I am creating a single monitor and trying to monitor a bunch of resources/endpoints.
So that if the alert condition matches for either of these resources, then alert is triggered.
But since I've added other 3 resources , i.e. b, c and d there is no alert notifications triggered.
I feel like I'm adding these wrong way.
How I can I add proper formula for this?
Question:
If an event arrives after the window has closed, then, how do we re-direct it to another topic for handling the correction ?
Context:
We use tumbling windows
We use events source creation time(event-time) for defining windows
thanks
Currently, there is no API to do that. Late events are dropped and you cannot get a hand on them easily.
What you could do is, to have an upstream operator (like a transform()) before the window, the compares the record timestamp to the current "stream time" (you would need to track "stream time" manually within the operator) -- this should help you to detect if the downstream window will drop the record as late and react to it accordingly (for example using a branch() after transform() and before groupByKey().windonwedBy().
I have a requirement as stated # https://kafka.apache.org/21/documentation/streams/developer-guide/dsl-api.html#window-final-results for waiting until window is closed in order to handle late out of order event by buffering it for duration of window.
Per my understanding of this feature is once windowing is created, the window works like wall clock processing, e.g. Creating for 1 hour window, The window starts ticking once first event comes. This 1hr window is closed exactly one hour later and all the events buffered so far will be forwarded to down stream. However, i need to be able to hold this window even longer say conditionally for as long as required e.g. based on state / information in external system such as database.
To be precise my requirement for event forwarding is (windows of 1 hour if external state record says it is good) or (hold for as long as required until external record says it's good and resume tracking of the event until the event make it fully 1hr, disregarding the time when external system is not good)
To elaborate this 2nd condition, e.g. if my window duration 1 1hr , my event starts at 00:00, if on 00:30 it is down and back normal on 00:45, the window should extend until 01:15.
Is it possible to pause and resume the forwarding of events conditionally based on my requirement above ?
Do I have to use transformation / processor and use value store manually to track the first processing time of my event and conditionally forwarding buffered events in punctuator ?
I appreciate all kind of work around and suggestion for this requirement.
the window works like wall clock processing
No. Kafka Streams work on event-time, hence, the timestamps as returned from the TimestampExtractor (by default the embedded record timestamp) are use to advance time.
To be precise my requirement for event forwarding is (windows of 1 hour if external state record says it is good)
This would need a custom solution IMHO.
or (hold for as long as required until external record says it's good and resume tracking of the event until the event make it fully 1hr, disregarding the time when external system is not good)
Not 100% if I understand this part.
Is it possible to pause and resume the forwarding of events conditionally based on my requirement above ?
No.
Do I have to use transformation / processor and use value store manually to track the first processing time of my event and conditionally forwarding buffered events in punctuator ?
I think this might be required.
Check out this blog post, that explains how suppress() work in details, and when it emits based on observed event-time: https://www.confluent.io/blog/kafka-streams-take-on-watermarks-and-triggers
I have a Process deployed on a self-hosted MSSI server. Bound to this Process I have a simple Pass-through query.
Some events gets dropped here "cep:/Server/Application/Erp/Entity/Event_Events_Process1/Query/StreamableBinding_1/Operator/Stream_1_CleanseInput"
I can see the counter of event dropped going up and I cannot find the reason why it's dropping.
Does anyone know how to debug that?
You can use the StreamInsight Event Flow Debugger. Make sure your application exposes the StreamInsight Management Service so you can hook up with the debugger. Then you can record the events which you can debug/step-through in the debugger.
Chances are your events are being dropped because of CTI violations. You might be enqueueing events that based on their start time occurred before the last CTI event.
That's absolutely a CTI violation. You'll see this behavior when you are issuing CTIs declaratively (for example, by specifying AdvanceTimeSettings.IncreasingStartTime or StrictlyIncreasingStartTime). There are a couple of ways that you can handle this:
1) Enqueue your CTIs programmatically. But you'll have to be careful of violations! (They'll cause an exception).
2) Tweak your AdvanceTimeSettings to include a Delay. You won't be able to use IncreasingStartTIme or StrictlyIncreasingStart time but you will be able to specify the CTI span duration or event count and a delay. Keep the delay small enough to keep your stream lively but large enough to not drop events. I can't tell you what that is; it'll depend on your events.
I am a newbie in Developer2000.
I have an Oracle pl/sql procedure (say, proc_submit_request) that fetches thousands of requests and submits them to dbms_job scheduler. The calling of dbms_job is
coded inside a loop for each request fetched.
Currently, i have a button (say, SUBMIT button) in oracle forms screen clicking which calls proc_submit_request.
The problem here is... the control does not return to my screen untill ALL of the requests fetched are submitted to the dbms_job (this takes hours to complete)
The screen grays out and just the hour-glass appears untill the completion of the procedure proc_submit_request.
proc_submit_appears returns a message to screen telling "XXXX requests submitted".
My requirement now is, once the user clicks the SUBMIT button, the screen should no longer gray out. The user should be able to navigate to other screens and not just struck with the submit screen untill the called procedure is completed.
I suggested running listeners (shell scripts and perl stuff) that can listen for any messages in pipe and run requests as back-ground process.
But the user is asking me to fix the issue in the application rather running listeners.
I've heard a little of OPEN_FORM built-in.
Suppose, I have two forms namely Form-1 and Form-2. Form-1 calls Form-2 using OPEN_FORM.
Now are the following things possible using OPEN_FORM?
On calling open_form('Form-2',OTHER-ARGUMENTS...), control must be in Form-1 (i.e. USer should not know that another form is getting opened) and Form-2 should call proc_submit_request.
User must be able to navigate to other screens in the application. But Form-2 must still be running until proc_submit_procedure is completed.
What happens if the user closes (exits) Form-1 ? Will Form-2 be still running?
Please provide me answers or suggest a good solution.
Good thought on the Form-1, Form-2 scenario - I'm not sure if that would work or not. But here is a much easier way without having to fumble around with coordinating forms being hidden and running stuff, and coming to the forefront when the function actually returns...etc, etc.
Rewrite your function that runs the database jobs to run as an AUTONOMOUS_TRANSACTION. Have a look at the compiler directive PRAGMA AUTONOMOUS_TRANSACTION for more details on that. You must use this within a database function/package/procedure - it is not valid with Forms (at least forms 10, not sure about 11).
You can then store the jobs result somewhere from your function (package variable, table, etc) and then use the built-in called CREATE_TIMER in conjunction with the form level trigger WHEN-TIMER-EXPIRED to go and check your storage location every 10 seconds or so - you can then display a message to the user about the jobs, and kill the timer using DELETE_TIMER.
You could create a single DBMS_JOB to call proc_submit_request. That way your form will only have to make one call; and the creation of all the other jobs will be done in a separate session.