how to record the arrival time of an event in esper - time

I want to record a data processing time in esper and I choose Bollinger Band as example. In Bollinger Band, there is called Moving Average (MA). that MA obtained from the result of calculate the stock price average. In this case, I set win:length(20). So, the MA can be obtained from the result of calculate the stock price average from 20 events that exists in the data window view. The following is code that i created.
public class BollingerBand {
static double startTime, finishTime;
public static void main (String [] args){
Configuration configuration = new Configuration();
configuration.addEventType("Stock", Stock.class);
EPServiceProvider epService = EPServiceProviderManager.getDefaultProvider(configuration);
AdapterInputSource source = new AdapterInputSource("BollingerBand.csv");
EPStatement statement = epService.getEPAdministrator().createEPL("insert into Aggregation " +
"select prevcount(symbol), symbol, avg(price) as SimpleMovingAverage, stddev(price) as StandardDeviation, " +
"last(price) as price, last(timestamp) as date from Stock.std:groupwin(symbol).win:length(20)" +
" group by symbol having count(*) >=20");
statement.addListener(new UpdateListener() {
public void update(EventBean[] newEvents, EventBean[] oldEvents) {
// TODO Auto-generated method stub
//System.out.println("Event Receive : "+newEvents[0].getUnderlying());
startTime = System.currentTimeMillis();
System.out.println("\nStart time : " + startTime + " miliseconds\n");
}
});
EPStatement statement2 = epService.getEPAdministrator().createEPL("select symbol, " +
"SimpleMovingAverage + 2*StandardDeviation as UpperBand," +
"SimpleMovingAverage as MiddleBand," +
"SimpleMovingAverage - 2*StandardDeviation as LowerBand," +
"price," +
"4*StandardDeviation/SimpleMovingAverage as Bandwidth," +
"(price - (SimpleMovingAverage - (2 * StandardDeviation))) / ((SimpleMovingAverage + " +
"(2 * StandardDeviation)) - (SimpleMovingAverage - (2 * StandardDeviation))) as PercentB," +
"date from Aggregation");
statement2.addListener(new UpdateListener() {
public void update(EventBean[] newEvents, EventBean[] oldEvents) {
// TODO Auto-generated method stub
//System.out.println("Event Receive : "+newEvents[0].getUnderlying());
finishTime = System.currentTimeMillis();
System.out.println("Start time : " + startTime + " miliseconds");
System.out.println("Finish time : " + finishTime + " miliseconds");
System.out.println("Processing time : " + (finishTime-startTime) + " miliseconds");
}
});
(new CSVInputAdapter(epService, source, "Stock")).start();
}
}
From the above code, time will be recorded if the average has calculated. But what i need is I want the time is recorded when the 20th event and next event enter to data window view. It's as the start time and finish time obtained from bollinger band calculation result. My question is how to record time of the 20th event and in the same time next event enter to the window view data. please help

The CSV adapter doesn't provide a callback when events are sent. You could easily change its code however. Or you could use a different CSV reader and send the events via runtime API.

Maybe have some sort of TickCounter in which there's a map that takes a key value pair of (item_count and timestamp). You update this in your second UpdateListener and of course you can always lookup the item of key 20.
By the way, I've used your Bollinger Band Calculation but using Storm and the EsperBolt. Blogged about it here:
http://chanchal.wordpress.com/2014/07/08/using-esperbolt-and-storm-to-calculate-bollinger-bands/

Related

Can I use variables across all the threads in the thread groups in jmeter?

I'm trying to create a test plan for rate-limiting behavior.
I set a rule that blocks after X requests per minute, and I want to check that I get response code 200 until I reached the X requests, and from then, to get 429. I created a counter that shared between all the threads, but it seems to be a mess because it's not a thread-safe.
This is my beanshell "once only controller":
String props_pre_fix = ${section_id} + "-" + ${START.HMS};
props.remove("props_pre_fix" + ${section_id}, props_pre_fix);
props.put("props_pre_fix" + ${section_id}, props_pre_fix);
props.put(props_pre_fix + "_last_response_code", "200");
props.put(props_pre_fix + "_my_counter", "0");
and this is the beanshell assertion:
String props_pre_fix = props.get("props_pre_fix" + ${section_id});
//log.info("props_pre_fix " + props_pre_fix);
//extract my counter from props
int my_counter = Integer.parseInt(props.get(props_pre_fix + "_my_counter"));
//extract last response code
String last_response_code = props.get(props_pre_fix + "_last_response_code");
log.info("last_response_code " + last_response_code);
//if last seconds is greater than current seconds it means we are in a new minute - set counter to zero
if(last_response_code.equals("429") && ResponseCode.equals("200")){
log.info("we moved to a new minute - my_counter should be zero");
my_counter = 0;
}
//increase counter
my_counter++;
log.info("set counter with value: " + my_counter);
//save counter
props.put(props_pre_fix + "_my_counter", my_counter + "");
log.info("counter has set with value: " + my_counter);
if (ResponseCode.equals("200")) {
props.put(props_pre_fix + "_last_response_code", "200");
if(my_counter <= ${current_limit}){
Failure = false;
}
else {
Failure = true;
FailureMessage = "leakage of " + (my_counter - ${current_limit}) + " requests";
}
}
else if (ResponseCode.equals("429")) {
props.put(props_pre_fix + "_last_response_code", "429");
if(my_counter > ${current_limit}){
Failure = false;
}
}
I'm using props to share the counter, but I obviously feel that this is not the right way to do it.
Can you suggest me how to do that?
I don't think that it is possible to automatically test this requirement using JMeter Assertions because you don't have access to the current throughput so I would rather recommend considering cross-checking Response Codes per Second and Transactions per Second charts (can be installed using JMeter Plugins Manager)
All the 200 and 429 responses can be marked as successful using Response Assertion configured like:
If for some reason you still want to do this programmatically you might want to take a look at Summariser class source which is used for displaying current throughput in the STDOUT.
Also be informed that starting from JMeter 3.1 you should be using JSR223 Test Elements and Groovy language for scripting.

Execution time reactive programming

Is this an ideal way to find execution time of method (getFavouriteDetails()), in reactive programming ?
public List<Favourites> getFavouriteDetails(String userId){
userService.getFavorites(userId)
.flatMap(favoriteService::getDetails)
.switchIfEmpty(suggestionService.getSuggestions())
.take(5)
.publishOn(UiUtils.uiThreadScheduler())
.subscribe(uiList::show, UiUtils::errorPopup)
.flatMap(a -> Mono.subscriberContext().map(ctx -> {
log.info("Time taken : " + Duration.between(ctx.get(key), Instant.now()).toMillis() + " milliseconds.");
return a;
}))
.subscriberContext(ctx -> ctx.put(key, Instant.now()))
}
Two approaches to ensure that you only measure execution time when you subscribe -
Wrap a Mono around the Flux using flatMapMany. This returns a Flux as well.
Use an AtomicReference, set time in onSubscribe and log elapsed time in doFinally.
Sample code -
timeFluxV1(getFavouriteDetails(userId)).subscribe(uiList::show, UiUtils::errorPopup);
timeFluxV1(getFavouriteDetails(userId)).subscribe(uiList::show, UiUtils::errorPopup);
private <T> Flux<T> timeFluxV1(Flux<T> flux) {
return Mono.fromSupplier(System::nanoTime)
.flatMapMany(time -> flux.doFinally(sig -> log.info("Time taken : " + TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - time) + " milliseconds.")));
}
private <T> Flux<T> timeFluxV2(Flux<T> flux) {
AtomicReference<Long> startTime = new AtomicReference<>();
return flux.doOnSubscribe(x -> startTime.set(System.nanoTime()))
.doFinally(x -> log.info("Time taken : " + TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTime.get()) + " milliseconds."));
}
public Flux<Favourites> getFavouriteDetails(String userId) {
return userService.getFavorites(userId)
.flatMap(favoriteService::getDetails)
.switchIfEmpty(suggestionService.getSuggestions())
.take(5)
.publishOn(UiUtils.uiThreadScheduler());
}
To time a method, the most basic way in Java is to use long System.nanoTime(). Instant and System.currentTimeMillis are for wall-clock operations and are not guaranteed to be monotonous nor precise enough...
In Reactor, to measure the time a sequence takes to complete, you would usually need to start the timing on subscription (nothing happens until you subscribe) and stop the timing within a doFinally (which execute some code on the side of the main sequence whenever it completes, errors or is cancelled).
Here however you are subscribing yourself, so there is no risk to be multiple subscriptions. You can thus do away with the "start timing on subscription" constraint.
It gives us something like this:
public List<Favourites> getFavouriteDetails(String userId){
final long start = System.nanoTime();
userService.getFavorites(userId)
.flatMap(favoriteService::getDetails)
.switchIfEmpty(suggestionService.getSuggestions())
.take(5)
.publishOn(UiUtils.uiThreadScheduler())
.doFinally(endType -> log.info("Time taken : " + TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) + " milliseconds."))
.subscribe(uiList::show, UiUtils::errorPopup);
//return needed!
}
Note that there is also a elapsed() operator, which measures the time between subscription and 1st onNext, then between subsequent onNexts. It outputs a Flux<Tuple2<Long, T>>, and you could aggregate the longs to get overall timing, but that would lose you the "realtime" nature of Ts in that case.

Is it good design to loop through many users? - SpringBoot

I am looking into good design principles and would just like some feedback.
I want to go through every user in my application that is a patient, check their scheduled activities, see if there is one upcoming and send them an email. It looks a little like this:
List<Patient> listOfAllPatients = patientRepositoryJPA.findAll();
if (listOfAllPatients.size() != 0) {
for (Patient patient : listOfAllPatients) {
user = userRepositoryJPA.findByUsername(patient.getUsername());
Timestamp currentTimeAndDate = new Timestamp(System.currentTimeMillis());
Long closestDate = Integer.toUnsignedLong(31);;
List<Activity> activities = activityRepositoryJPA.findByPatientAndStartDateTimeAfter(patient, currentTimeAndDate);
for (Activity activity : activities) {
Timestamp activityDateAndTime = activity.getStartDateTime();
Long difference = getTimeDifferenceByMinutes(currentTimeAndDate, activityDateAndTime);
if (difference < closestDate) {
LocalDateTime upcomingDate = activity.getStartDateTime().toLocalDateTime();
emailHandler.sendEmail(javaMailSender, "Upcoming Activity Reminder", "The activity, " + activity.getName() + " is starting soon! (" +
upcomingDate.getHour() + ":" + upcomingDate.getMinute() + ")", user);
}
}
}
Now here I loop through every patient in the application and then through each of their activities. If there were many user (millions) surely there would be some bottlenecks or something here.
Does anybody have any advice on how big companies would handle data like this?
Or is what I am doing fine? Thanks.
After some advice, I have improved it to this:
Timestamp currentTimeAndDate = new Timestamp(System.currentTimeMillis());
LocalDateTime add31Minutes = LocalDateTime.now().plus(31, ChronoUnit.MINUTES);
Timestamp noLaterThanDateAndTime = Timestamp.valueOf(add31Minutes);
for (Activity activity : activityRepositoryJPA.findByStartDateTimeBetween(currentTimeAndDate, noLaterThanDateAndTime)) {
Patient patient = activity.getPatient();
LocalDateTime upcomingDate = activity.getStartDateTime().toLocalDateTime();
emailHandler.sendEmail(javaMailSender, "BAA - Upcoming Activity Reminder", "The activity, " + activity.getName() + " is starting soon! (" +
upcomingDate.getHour() + ":" + upcomingDate.getMinute() + ")", patient.getUser());
}
Now I find all activities between now and 31 minutes time in 1 query.

JDBC - How do I accumulate data from ResultSet

How would I accumulate data from a ResultSet? What I'm trying to do is get the price from the ResultSet and it only gets the first value, but what I want is every price for e.g 25.0, 10.00, and 15.00 I have stored them in a Double datatype, but the Result Set only gets the last value when I have a while(rs.next() and if I have if(rs.next()) it will get 25.00 but I want all the prices to be added. How would I do this exactly? Thanks!
if(rs2.next()) {
jTextArea3.append("\n");
jTextArea3.append("\n");
jTextArea3.append("Part Type: " + rs2.getString("PartType"));
jTextArea3.append("\n");
jTextArea3.append("Description: " + rs2.getString("Description"));
jTextArea3.append("\n");
jTextArea3.append("Price: " + rs2.getString("Price"));
partsCost = rs2.getDouble("Price");
System.out.println(partsCost);
}
Maybe something like this is what you mean?
double total = 0;
while(rs.next()) {
jTextArea3.append("\n");
jTextArea3.append("\n");
jTextArea3.append("Part Type: " + rs2.getString("PartType"));
jTextArea3.append("\n");
jTextArea3.append("Description: " + rs2.getString("Description"));
jTextArea3.append("\n");
jTextArea3.append("Price: " + rs2.getString("Price"));
partsCost = rs2.getDouble("Price");
System.out.println(partsCost);
total += partsCost;
}
System.out.println(total);
You don't show the SQL query. If you only care about the total of partsCosts, and not the other details, you would need to change your query to have an aggregate function such as, "select sum(partsCost)... "

Entity Framework SaveChanges() first call is very slow

I appreciate that this issue has been raised a couple of times before, but I can't find a definitive answer (maybe there isn't one!).
Anyway the title tells it all really. Create a new context, add a new entity, SaveChanges() takes 20 seconds. Add second entity in same context, SaveChanges() instant.
Any thoughts on this? :-)
============ UPDATE =============
I've created a very simple app running against my existing model to show the issue...
public void Go()
{
ModelContainer context = new ModelContainer(DbHelper.GenerateConnectionString());
for (int i = 1; i <= 5; i++)
{
DateTime start = DateTime.Now;
Order order = context.Orders.Single(c => c.Reference == "AA05056");
DateTime end = DateTime.Now;
double millisecs = (end - start).TotalMilliseconds;
Console.WriteLine("Query " + i + " = " + millisecs + "ms (" + millisecs / 1000 + "s)");
start = DateTime.Now;
order.Note = start.ToLongTimeString();
context.SaveChanges();
end = DateTime.Now;
millisecs = (end - start).TotalMilliseconds;
Console.WriteLine("SaveChanges " + i + " = " + millisecs + "ms (" + millisecs / 1000 + "s)");
Thread.Sleep(1000);
}
Console.ReadKey();
}
Please do not comment on my code - unless it is an invalid test ;)
The results are:
Query 1 = 3999.2288ms (3.9992288s)
SaveChanges 1 = 3391.194ms (3.391194s)
Query 2 = 18.001ms (0.018001s)
SaveChanges 2 = 4.0002ms (0.0040002s)
Query 3 = 14.0008ms (0.0140008s)
SaveChanges 3 = 3.0002ms (0.0030002s)
Query 4 = 13.0008ms (0.0130008s)
SaveChanges 4 = 3.0002ms (0.0030002s)
Query 5 = 10.0005ms (0.0100005s)
SaveChanges 5 = 3.0002ms (0.0030002s)
The first query takes time which I assume is the view generation? Or db connection?
The first save takes nearly 4 seconds which for the more complex save in my app takes over 20 seconds which is not acceptable.
Not sure where to go with this now :-(
UPDATE...
SQL Profiler shows first query and update are fast and are not different for first. So I know delay is Entity Framework as suspected.
It might not be the SaveChanges call - the first time you make any call to the database in EF, it has to do some initial code generation from the metadata. You can pre-generate this though at compile-time: http://msdn.microsoft.com/en-us/library/bb896240.aspx
I would be surprised if that's the only problem, but it might help.
Also have a look here: http://msdn.microsoft.com/en-us/library/cc853327.aspx
I would run the following code on app start up and see how long it takes and if after that the first SaveChanges is fast.
public static void UpdateDatabase()
{
//Note: Using SetInitializer is reconnended by Ladislav Mrnka with reputation 275k
//http://stackoverflow.com/questions/9281423/entity-framework-4-3-run-migrations-at-application-start
Database.SetInitializer<DAL.MyDbContext>(
new MigrateDatabaseToLatestVersion<DAL.MyDbContext,
Migrations.MyDbContext.Configuration>());
using (var db = new DAL.MyDbContext()) {
db.Database.Initialize(false);//Execute the migrations now, not at the first access
}
}

Resources