I've add AbstractPagingItemReader to read by packet from api by 100 items
The execution is OK until the size of returned items is less than 100
Here is my code :
public static final int ITEMS_BY_PAGE = 10;
#Override
protected void doReadPage() {
setPageSize(ITEMS_BY_PAGE);
results = this.documentService.searchRecordsWithAwaitingDocument(getPage(), ITEMS_BY_PAGE);
}
#Override
protected void doJumpToPage(int itemIndex) {
}
Is there a way to not stop the execution when the size of returned items is less than ITEMS_BY_PAGE ? i want to stop the execution only when there is no result on return (size 0) ?
Related
0
I have an assignment which asks for everything I have in the code below. That all works fine - I just need to calculate any monthly hours over 160 hours to be paid at 1.5 times the normal hourly rate. My math seems sound and calculates fine:
((hours - 160) * overtime) + (160 * hourlyRate)
But I dont know if I'm putting this if statement in the right method or if it even should be an if statement. My increase/decreasePay methods are working prior to this and they need to stay. I removed some things so it's easier to read.
HourlyWorker Class:
public class HourlyWorker extends Employee
{
private int hours;
private double hourlyRate;
private double monthlyPay;
private double overtime = (1.5 * hourlyRate);
public HourlyWorker(String last, String first, String ID, double rate)
{
super(last, first, ID);
hourlyRate = rate;
}
public void setHours(int hours)
{
this.hours = hours;
}
public int getHours()
{
return hours;
}
public void setHourlyRate(double rate)
{
this.hourlyRate = rate;
}
public double getHourlyRate()
{
return hourlyRate;
}
public double getMonthlyPay()
{
if (hours > 160)
{
monthlyPay = ((hours - 160) * overtime) + (160 * hourlyRate);
}
else
{
monthlyPay = hourlyRate * hours;
}
return monthlyPay;
}
public void increasePay(double percentage)
{
hourlyRate *= 1 + percentage / 100;
}
public void decreasePay(double percentage)
{
hourlyRate *= 1 - percentage / 100;
}
}
What I'm testing with:
public class TestEmployee2
{
public static void main(String[] args)
{
Employee [] staff = new Employee[3];
HourlyWorker hw1 = new HourlyWorker("Bee", "Busy", "BB1265", 10);
hw1.setHours(200);
staff[0] = hw1;
System.out.println(staff[0].getMonthlyPay());
staff[0].increasePay(10);
System.out.println(staff[0].getMonthlyPay());
}
}
Output is:
1600 (initial monthly rate, with 40 overtime hours and 160 regular hours)
1760 (10% increase to the monthlyPay)
Should be:
2006
2206.6
String.split() will do the trick.
go over the list of artists you have a split each row to artist/genre.
for (String artist : artists) {
String[] split = artist.split(" ");
// add some data validation to avoid ArrayIndexOutOfBounds
String name = split[0];
String genre = split[1];
}
You can use Files.readAllLines(myPath) to read from File.
If you are familiar with Streams from Java 8 you can use Streams on read Lines from File.
Using .stream() and collecting them in a format you want. Either as list or joining them to a single String.
I have a collection of data like dummy below
class Place {
userId,
price
}
That means a collection of some places.
Use-case:
There is a user with userId and login.
How to calc average place-price that equal to userId ?
RxJava is nice and I have tried filter and toList, however it is not so performance nice.
Observable.fromIterable(places)
.subscribeOn(Schedulers.newThread())
.filter(new Predicate<Place>() {
#Override
public boolean test(Place place) throws Exception {
return place.userId == global.login.userId;
}
})
.toList()
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Consumer<List<Place>>() {
#Override
public void accept(List<Place> filteredPlace) throws Exception {
//Here I have to use loop to do math-average, it is not nice to average.
}
});
If the places is something that is already available in-memory, you can rearrange the evaluation such as this:
Observable.just(places)
.subscribeOn(Schedulers.computation())
.map((Iterable<Place> list) -> {
double sum = 0.0;
int count = 0;
for (Place p : list) {
if (p.userId == global.login.userId) {
sum += p.price;
count++;
}
}
return sum / count;
})
.observeOn(AndroidSchedulers.mainThread())
.subscribe(average -> { /* display average */ });
If the sequence of places becomes available over time (through an Observable):
Observable<Place> places = ...
places
.observeOn(Schedulers.computation())
.filter((Place p) -> p.userId == global.login.userId)
.compose(o -> MathObservable.averageDouble(o.map(p -> p.price)))
.observeOn(AndroidSchedulers.mainThread())
.subscribe(average -> { /* display average */ });
MathObservable is part of the RxJava 2 Extensions library.
My program has 3 functions. Each function takes a list of Items and fill certain information.
For example
class Item {
String sku,upc,competitorName;
double price;
}
function F1 takes a List and fills upc
function F2 takes List (output of F1) and fills price.
function F3 takes List (output of F2) and fills competitorName
F1 can process 5 items at a time,
F2 can process 20 items at a time,
F3 also 20.
Right now I am running F1 -> F2 -> F3 in serial because F2 needs info(UPC code) from F1. F3 needs price from F2.
I would like to make this process efficient by running F1 run continuously instead of waiting for F2 and F3 to be completed. F1 executes and output into queue then F2 takes 20 items at a time and process them. and then follows F3.
How can i achieve this by using BlockingCollection and Queue?
This is a typical use case of Apache Storm in case you've continuous items coming in to F1. You can implement this in Storm in matter of minutes and you'll have fast and perfectly parallel system in place. Your F1, F2 and F3 will become bolts and your Items producer will become spout.
Since you asked how to do it using BlockingCollections here is an implementation. You'll need 3 threads in total.
ItemsProducer: It is producing 5 items at a time and feeding it to F1.
F2ExecutorThread: It is consuming 20 items at a time and feeding it to F2.
F3ExecutorThread: It is consuming 20 items at a time and feeding it to F3.
You also have 2 blocking queues one is used to transfer data from F1->F2 and one from F2->F3. You can also have a queue to feed data to F1 in similar fashion if required. It depends upon how you are getting the items. I've used Thread.sleep to simulate the time required to execute the function.
Each function will keep looking for items in their assigned queue, irrespective of what other functions are doing and wait until the queue has items. Once they've processed the item they'll put it in another queue for another function. They'll wait until the other queue has space if it is full.
Since all your functions are running in different threads, F1 won't be waiting for F2 or F3 to finish. If your F2 and F3 are significantly faster then F1 you can assign more threads to F1 and keep pushing to same f2Queue.
public class App {
final BlockingQueue<Item> f2Queue = new ArrayBlockingQueue<>(100);
final BlockingQueue<Item> f3Queue = new ArrayBlockingQueue<>(100);
public static void main(String[] args) throws InterruptedException {
App app = new App();
app.start();
}
public void start() throws InterruptedException {
Thread t1 = new ItemsProducer(f2Queue);
Thread t2 = new F2ExecutorThread(f2Queue, f3Queue);
Thread t3 = new F3ExecutorThread(f3Queue);
t1.start();
t2.start();
t3.start();
t1.join();
t2.join();
t3.join();
}
}
/**
* Thread producing 5 items at a time and feeding it to f1()
*/
class ItemsProducer extends Thread {
private BlockingQueue<Item> f2Queue;
private static final int F1_BATCH_SIZE = 5;
public ItemsProducer(BlockingQueue<Item> f2Queue) {
this.f2Queue = f2Queue;
}
public void run() {
Random random = new Random();
while (true) {
try {
List<Item> items = new ArrayList<>();
for (int i = 0; i < F1_BATCH_SIZE; i++) {
Item item = new Item(String.valueOf(random.nextInt(100)));
Thread.sleep(20);
items.add(item);
System.out.println("Item produced: " + item);
}
// Feed items to f1
f1(items);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
void f1(List<Item> items) throws InterruptedException {
Random random = new Random();
for (Item item : items) {
Thread.sleep(100);
item.upc = String.valueOf(random.nextInt(100));
f2Queue.put(item);
}
}
}
/**
* Thread consuming items produced by f1(). It takes 20 items at a time, but if they are not
* available it waits and starts processesing as soon as one gets available
*/
class F2ExecutorThread extends Thread {
static final int F2_BATCH_SIZE = 20;
private BlockingQueue<Item> f2Queue;
private BlockingQueue<Item> f3Queue;
public F2ExecutorThread(BlockingQueue<Item> f2Queue, BlockingQueue<Item> f3Queue) {
this.f2Queue = f2Queue;
this.f3Queue = f3Queue;
}
public void run() {
try {
List<Item> items = new ArrayList<>();
while (true) {
items.clear();
if (f2Queue.drainTo(items, F2_BATCH_SIZE) == 0) {
items.add(f2Queue.take());
}
f2(items);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
void f2(List<Item> items) throws InterruptedException {
Random random = new Random();
for (Item item : items) {
Thread.sleep(100);
item.price = random.nextInt(100);
f3Queue.put(item);
}
}
}
/**
* Thread consuming items produced by f2(). It takes 20 items at a time, but if they are not
* available it waits and starts processesing as soon as one gets available.
*/
class F3ExecutorThread extends Thread {
static final int F3_BATCH_SIZE = 20;
private BlockingQueue<Item> f3Queue;
public F3ExecutorThread(BlockingQueue<Item> f3Queue) {
this.f3Queue = f3Queue;
}
public void run() {
try {
List<Item> items = new ArrayList<>();
while (true) {
items.clear();
if (f3Queue.drainTo(items, F3_BATCH_SIZE) == 0) {
items.add(f3Queue.take());
}
f3(items);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void f3(List<Item> items) throws InterruptedException {
Random random = new Random();
for (Item item : items) {
Thread.sleep(100);
item.competitorName = String.valueOf(random.nextInt(100));
System.out.println("Item done: " + item);
}
}
}
class Item {
String sku, upc, competitorName;
double price;
public Item(String sku) {
this.sku = sku;
}
public String toString() {
return "sku: " + sku + " upc: " + upc + " price: " + price + " compName: " + competitorName;
}
}
I guess you can follow the exact same approach in .Net as well. For better understanding I suggest you to go through basic architecture of http://storm.apache.org/releases/current/Tutorial.html
I tried to do same thing in .NET and i think it is working.
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
namespace BlockingCollectionExample
{
class Program
{
static void Main(string[] args)
{
BlockingCollection<Listing> needUPCJobs = new BlockingCollection<Listing>();
BlockingCollection<Listing> needPricingJobs = new BlockingCollection<Listing>();
// This will have final output
List<Listing> output = new List<Listing>();
// start executor 1 which waits for data until available
var executor1 = Task.Factory.StartNew(() =>
{
int maxSimutenousLimit = 5;
int gg = 0;
while (true)
{
while (needUPCJobs.Count >= maxSimutenousLimit)
{
List<Listing> tempListings = new List<Listing>();
for (int i = 0; i < maxSimutenousLimit; i++)
{
Listing listing = new Listing();
if (needUPCJobs.TryTake(out listing))
tempListings.Add(listing);
}
// Simulating some delay for first executor
Thread.Sleep(1000);
foreach (var eachId in tempListings)
{
eachId.UPC = gg.ToString();
gg++;
needPricingJobs.Add(eachId);
}
}
if (needUPCJobs.IsAddingCompleted)
{
if (needUPCJobs.Count == 0)
break;
else
maxSimutenousLimit = needUPCJobs.Count;
}
}
needPricingJobs.CompleteAdding();
});
// start executor 2 which waits for data until available
var executor2 = Task.Factory.StartNew(() =>
{
int maxSimutenousLimit = 10;
int gg = 10;
while (true)
{
while (needPricingJobs.Count >= maxSimutenousLimit)
{
List<Listing> tempListings = new List<Listing>();
for (int i = 0; i < maxSimutenousLimit; i++)
{
Listing listing = new Listing();
if (needPricingJobs.TryTake(out listing))
tempListings.Add(listing);
}
// Simulating more delay for second executor
Thread.Sleep(10000);
foreach (var eachId in tempListings)
{
eachId.Price = gg;
gg++;
output.Add(eachId);
}
}
if (needPricingJobs.IsAddingCompleted)
{
if(needPricingJobs.Count==0)
break;
else
maxSimutenousLimit = needPricingJobs.Count;
}
}
});
// producer thread
var producer = Task.Factory.StartNew(() =>
{
for (int i = 0; i < 100; i++)
{
needUPCJobs.Add(new Listing() { ID = i });
}
needUPCJobs.CompleteAdding();
});
// wait for producer to finish producing
producer.Wait();
// wait for all executors to finish executing
Task.WaitAll(executor1, executor2);
Console.WriteLine();
Console.WriteLine();
}
}
public class Listing
{
public int ID;
public string UPC;
public double Price;
public Listing() { }
}
}
I need some help with RxJava. I have an expensive network call which returns an Observable (stream of Adverts from elasticsearch). I want to cache the ID property of each emitted item (Advert) for 10 minutes (in Redis), so that subsequent calls in the following 10 minutes use the IDs from Cache to fetch the adverts from Elasticsearch.
I've got some code - which goes some way towards achieving the desired outcome, (credit to following blog ..
http://blog.danlew.net/2015/06/22/loading-data-from-multiple-sources-with-rxjava/)
It can cache each emitted item from the stream, what I need is it to cache all the IDs from those items in the stream as 1 cache entry
Code so far is here https://github.com/tonymurphy/rxjava for anyone interested, snippets below
#Component
public class CachingObservable {
private final Logger logger = LoggerFactory.getLogger(CachingObservable.class);
#Autowired
private AdvertService advertService;
// Each "network" response is different
private Long requestNumber = 0L;
public Observable<Advert> getAdverts(final String location) {
Observable<Advert> memory = memory(location);
Observable<Advert> network = network(location);
Observable<Advert> networkWithSave = network.doOnNext(new Action1<Advert>() {
#Override
public void call(Advert advert) {
List<Long> ids = new ArrayList<Long>();
ids.add(advert.getId());
advertService.cache(location, ids);
}
});
// Retrieve the first source with data - concat checks in order
Observable<Advert> source = Observable.concat(memory,
networkWithSave)
.first();
return source;
}
From my understanding, the concat method is not really useful for my use case. I need to know if/when the network observable completes, I need to get the list of advert id's returned and I need to store them in the cache. I could subscribe to the network observable - but I want this to be lazy - only called if no data is found in the cache. So the following updated code doesn't work.. any ideas appreciated
public Observable<Advert> getAdverts(final String location) {
Observable<Advert> memory = memory(location);
Observable<Advert> network = network(location);
Observable<Advert> networkWithSave = network.doOnNext(new Action1<Advert>() {
#Override
public void call(Advert advert) {
List<Long> ids = new ArrayList<Long>();
ids.add(advert.getId());
advertService.cache(location, ids);
}
});
// Retrieve the first source with data - concat checks in order
Observable<Advert> source = Observable.concat(memory,
networkWithSave)
.first();
Observable<List<Advert>> listObservable = networkWithSave.toList();
final Func1<List<Advert>, List<Long>> transformer = new Func1<List<Advert>, List<Long>>() {
#Override
public List<Long> call(List<Advert> adverts) {
List<Long> ids = new ArrayList<Long>();
for (Advert advert : adverts) {
ids.add(advert.getId());
}
return ids;
}
};
listObservable.map(transformer).subscribe(new Action1<List<Long>>() {
#Override
public void call(List<Long> ids) {
logger.info("ids {}", ids);
}
});
return source;
}
What I would do is use filter to make sure the old content of the cache is not emitted so the concat jumps to the network call:
Subject<Pair<Long, List<Advert>>, Pair<Long, List<Advert>>> cache =
BehaviorSubject.create().toSerialized();
static final long RETENTION_TIME = 10L * 60 * 1000;
Observable<Advert> memory = cache.filter(v ->
v.first + RETENTION_TIME > System.currentTimeMillis()).flatMapIterable(v -> v);
Observable<Advert> network = ...
Observable<Advert> networkWithSave = network.toList().doOnNext(v ->
cache.onNext(Pair.of(System.currentTimeMillis(), v)).flatMapIterable(v -> v)
);
return memory.switchIfEmpty(network);
Ok, I think I have a solution which will work for me. I may have overlooked something, but it should be easy?
public Observable<Advert> getAdverts(final String location) {
Observable<Advert> memory = memory(location);
final Observable<Advert> network = network(location);
final Func1<List<Advert>, List<Long>> advertToIdTransformer = convertAdvertsToIds();
memory.isEmpty().subscribe(new Action1<Boolean>() {
#Override
public void call(Boolean aBoolean) {
if (aBoolean.equals(Boolean.TRUE)) {
Observable<List<Long>> listObservable = network.toList().map(advertToIdTransformer);
listObservable.subscribe(new Action1<List<Long>>() {
#Override
public void call(List<Long> ids) {
logger.info("Caching ids {}", ids);
advertService.cache(location, ids);
}
});
}
}
});
// Retrieve the first source with data - concat checks in order
Observable<Advert> source = Observable.concat(memory,
network)
.first();
return source;
}
I have to implement a limitation algorithm in order to avoid to reach a throughput limit imposed by the service I'm interacting with.
The limit is specified as «N requests over 1 day» where N is of the order of magnitude of 10^6.
I have a distributed system of clients interacting with the service so they should share the measure.
An exact solution should involve to record all the events and than computing the limit «when» the event of calling the service occur: of course this approach is too expensive and so I'm looking for an approximate solution.
The first one I devised imply to discretize the detection of the events: for example maintaing 24 counters at most and recording the number of requests occurred within an hour.
Acceptable.
But I feel that a more elegant, even if leaded by different «forces», is to declinate the approach to the continuum.
Let's say recording the last N events I could easily infer the «current» throughput. Of course this algorithm suffer for missing consideration of the past events occurred the hours before. I could improve with with an aging algorithm but… and here follow my question:
Q: «There's an elegant approximate solution to the problem of estimating the throughput of a service over a long period with and high rate of events?»
As per my comments, you should use a monitor and have it sample the values every 15 minutes or something to get a reasonable guess of the number of requests.
I mocked something up here but haven't tested it, should give you a starter.
import java.util.LinkedList;
import java.util.Queue;
import java.util.Timer;
import java.util.TimerTask;
public class TestCounter {
private final Monitor monitor;
private TestCounter() {
monitor = new Monitor();
}
/** The thing you are limiting */
public void myService() {
if (monitor.isThresholdExceeded()) {
//Return error
} else {
monitor.incremenetCounter();
//do stuff
}
}
public static void main(String[] args) {
TestCounter t = new TestCounter();
for (int i = 0; i < 100000; i++) {
t.myService();
}
for (int i = 0; i < 100000; i++) {
t.myService();
}
}
private class Monitor {
private final Queue<Integer> queue = new LinkedList<Integer>();
private int counter = 1;
/** Number of 15 minute periods in a day. */
private final int numberOfSamples = 76;
private final int threshold = 1000000;
private boolean thresholdExceeded;
public Monitor() {
//Schedule a sample every 15 minutes.
Timer t = new Timer();
t.scheduleAtFixedRate(new TimerTask() {
#Override
public void run() {
sampleCounter();
}
}, 0l, 900000 /** ms in 15 minutes */
);
}
/** Could synchroinise */
void incremenetCounter() {
counter++;
}
/** Could synchroinise */
void sampleCounter() {
int tempCount = counter;
counter = 0;
queue.add(tempCount);
if (queue.size() > numberOfSamples) {
queue.poll();
}
int totalCount = 0;
for (Integer value : queue) {
totalCount += value;
}
if (totalCount > threshold) {
thresholdExceeded = true;
} else {
thresholdExceeded = false;
}
}
public boolean isThresholdExceeded() {
return thresholdExceeded;
}
}
}