We have a Windows Phone 7 application which uses a set of 3 service methods using Reactive Extensions, defined as follows:
public static class ServiceClient
{
public static IObservable<string> LookupImage(byte[] image) {...}
public static IObservable<XDocument> GetDefinition(string id) {...}
public static IObservable<Dictionary<string, byte[]>> GetFiles(string id, string[] fileNames) {...}
}
We need the WP7 application to keep calling LookupImage in the above client (each time with different set of byte[] image data) until the returned IObservable<string> is nonempty. After we get the Observable string we have to call GetDefinition and GetFiles methods (in that order).
The calls to LookupImage should happen as often as the service response is returned as opposed to being controlled by a timer as it will vary depending on network connection speed and we need to be able to send as many of these as possible.
I'd appreciate any pointers to what might be a solution to the above. As a start I have the following
private void RunLookupAndRenderLogic()
{
byte[] imageBytes = GetImageBytes();
// There are some cases where the image was not 'interesting' enough in which case GetImageBytes() returns null
if (pictureBytes != null)
{
// Where we have image data, send this to LookupImage service method
var markerLookup = ServiceClient.LookupImage(imageBytes);
markerLookup.Subscribe(id =>
{
// If the id is empty, we need to call this again.
if (String.IsNullOrEmpty(id))
{
???
}
// If we have an id, call GetDefinition and GetFiles methods of the service. No further calls to LookupImage should take place.
RenderLogic(id);
});
}
else
// If no interesting image was returned, try again
RunRecognitionAndRenderLogic();
}
Apologies if I get this wrong, but if I understand it correctly you want to Retry the call to LookupImage with the exact same argument, until it returns a value?
A naive way of solving this would be to simply call repeat and then take(1):
ServiceClient.LookupImage(imageBytes)
.Repeat()
.Take(1)
.Subscribe(id => ....);
However as Rx is single threaded by default, there is no point in this context that allows us to inject our disposal call (implicit from the Take(1)-->OnComplete()-->Auto disposal of subscription).
You can dodge this by offering some breathing space between subsequent re-subscriptions by using the CurrentThread Scheduler.
Observable.Defer(()=>
ServiceClient.LookupImage(imageBytes)
.ObserveOn(Scheduler.CurrentThread)
)
.Repeat()
.Take(1)
.Subscribe(id => ....);
There are other ways of achieving this with some good understanding of Rx and some creativity. (Most I would imagine a Scheduler)
To give you some inspriation check out the chapter on Scheduling and Threading. It covers recursion and building your own iterator which is effectively what you are trying to do.
Full code sample:
private void RunLookupAndRenderLogic()
{
byte[] imageBytes = GetImageBytes();
// There are some cases where the image was not 'interesting' enough in which case GetImageBytes() returns null
if (pictureBytes != null)
{
// Where we have image data, send this to LookupImage service method
var subscription = Observable
.Defer(()=>
ServiceClient.LookupImage(imageBytes)
.ObserveOn(Scheduler.CurrentThread)
)
.Where(id=>!String.IsNullOrEmpty(id))
.Repeat()
.Take(1)
.Subscribe(id =>
{
// If we have an id, call GetDefinition and GetFiles methods of the service. No further calls to LookupImage should take place.
RenderLogic(id);
});
//TODO: You dont offer any way to cancel this (dispose of the suscription).
//This means you could loop forever :-(
}
else
{
// If no interesting image was returned, try again
RunRecognitionAndRenderLogic();
}
}
(Disclosure: I am the author of IntroToRx.com)
Related
Up until now I have done very basic things with smallrye Mutiny in Quarkus. Basically, I have one or two very small web services which only interact with a web application. These services return a Uni<Response>.
Now I'm writing a logging service I want my others to pass information to. In this logging service, I need to return a value to calling services. The logging service will return this value as a Uni<Integer>. What I'm struggling with is how to extract the return value in the calling service as an int.
Here is the function in the logging service
#GET
#Path("/requestid")
#Produces(MediaType.TEXT_PLAIN)
public Uni<Integer> getMaxRequestId(){
return service.getMaxRequestId();
}
public Uni<Integer> getMaxRequestId() {
Integer result = Integer.valueOf(em.createQuery("select MAX(request_id) from service_requests").getFirstResult());
if(result == null) {
result = 0;
}
return Uni.createFrom().item(result += 1);
}
And here is the client side code in the calling service
#Path("/requests")
public class RequestIdResource {
#RestClient
RequestIdServices service;
#GET
#Path("/requestid")
#Produces(MediaType.TEXT_PLAIN)
public Uni<Integer> getMaxRequestId(){
return service.getMaxRequestId();
}
}
public void filter(ContainerRequestContext requestContext) throws IOException {
int requestid = client.getMaxRequestId();
rm.name = ConfigProvider.getConfig().getValue("quarkus.application.name", String.class);
rm.server = requestContext.getUriInfo().getBaseUri().getHost();
rm.text = requestContext.getUriInfo().getPath(true);
rm.requestid = requestid;
}
Basically everything I have tried creates another Uni. Maybe I am simply using the concept all wrong. But how do I get the Integer out of the Uni so I can get the intValue?
You need to invoke a terminal operation, or use the value and continue the chain.
If you want to invoke a terminal operator you can invoke the await operation to make your code blocking and wait for the response.
If you want to merge this reactive invocation with another that is present in your client code, you can join or combine your actual Mutiny stream with the on coming from the response by using the combine method.
If you just want to use the value and do not retrieve it, you can suscribe and get the result.
If you have a multi you can call directly the method toList
Assuming that you want to have some timeouts involved and you want to get the actual Integer, you can go with the await method and a timeout.
In one documentation they say IHandleMessages handler hast to be written this way (signature is automatically generated when I choose to "Implement interface" option in Visual Studio):
public class PlaceOrderHandler : IHandleMessages<PlaceOrder>
{
public Task Handle(PlaceOrder message, IMessageHandlerContext context)
{
var orderPlaced = new OrderPlaced { OrderId = message.OrderId };
return context.Publish(orderPlaced);
}
}
While another documentation says it has to be written this way:
public class PlaceOrderHandler : IHandleMessages<PlaceOrder>
{
public async Task Handle(PlaceOrder message, IMessageHandlerContext context)
{
var orderPlaced = new OrderPlaced { OrderId = message.OrderId };
await context.Publish<OrderPlaced>(e => { e.OrderId = message.OrderId; });
}
}
I wonder what is the difference between these two statements, can someone explain in simple language?
Which option is the right one?
Both are correct options. The difference between the two is how a single asynchronous operation is handles in the Handle method.
In the first case, a Task is returned as-is. In the second case, publishing is awaited within the Handle method. The difference? In the first case no async state machine is created by the compiler as the task of publishing returned back. In the second scenario, a state machine is created.
Which option is the right one to use? They are both correct options. If a method is called frequently and you care for the unnecessary allocations not to take place, returnng a single task without awaiting is more efficient.
I'm new to ReactiveX/RxJs and I'm wondering if my use-case is feasible smoothly with RxJs, preferably with a combination of built-in operators. Here's what I want to achieve:
I have an Angular2 application that communicates with a REST API. Different parts of the application need to access the same information at different times. To avoid hammering the servers by firing the same request over and over, I'd like to add client-side caching. The caching should happen in a service layer, where the network calls are actually made. This service layer then just hands out Observables. The caching must be transparent to the rest of the application: it should only be aware of Observables, not the caching.
So initially, a particular piece of information from the REST API should be retrieved only once per, let's say, 60 seconds, even if there's a dozen components requesting this information from the service within those 60 seconds. Each subscriber must be given the (single) last value from the Observable upon subscription.
Currently, I managed to achieve exactly that with an approach like this:
public getInformation(): Observable<Information> {
if (!this.information) {
this.information = this.restService.get('/information/')
.cache(1, 60000);
}
return this.information;
}
In this example, restService.get(...) performs the actual network call and returns an Observable, much like Angular's http Service.
The problem with this approach is refreshing the cache: While it makes sure the network call is executed exactly once, and that the cached value will no longer be pushed to new subscribers after 60 seconds, it doesn't re-execute the initial request after the cache expires. So subscriptions that occur after the 60sec cache will not be given any value from the Observable.
Would it be possible to re-execute the initial request if a new subscription happens after the cache timed out, and to re-cache the new value for 60sec again?
As a bonus: it would be even cooler if existing subscriptions (e.g. those who initiated the first network call) would get the refreshed value whose fetching had been initiated by the newer subscription, so that once the information is refreshed, it is immediately passed through the whole Observable-aware application.
I figured out a solution to achieve exactly what I was looking for. It might go against ReactiveX nomenclature and best practices, but technically, it does exactly what I want it to. That being said, if someone still finds a way to achieve the same with just built-in operators, I'll be happy to accept a better answer.
So basically since I need a way to re-trigger the network call upon subscription (no polling, no timer), I looked at how the ReplaySubject is implemented and even used it as my base class. I then created a callback-based class RefreshingReplaySubject (naming improvements welcome!). Here it is:
export class RefreshingReplaySubject<T> extends ReplaySubject<T> {
private providerCallback: () => Observable<T>;
private lastProviderTrigger: number;
private windowTime;
constructor(providerCallback: () => Observable<T>, windowTime?: number) {
// Cache exactly 1 item forever in the ReplaySubject
super(1);
this.windowTime = windowTime || 60000;
this.lastProviderTrigger = 0;
this.providerCallback = providerCallback;
}
protected _subscribe(subscriber: Subscriber<T>): Subscription {
// Hook into the subscribe method to trigger refreshing
this._triggerProviderIfRequired();
return super._subscribe(subscriber);
}
protected _triggerProviderIfRequired() {
let now = this._getNow();
if ((now - this.lastProviderTrigger) > this.windowTime) {
// Data considered stale, provider triggering required...
this.lastProviderTrigger = now;
this.providerCallback().first().subscribe((t: T) => this.next(t));
}
}
}
And here is the resulting usage:
public getInformation(): Observable<Information> {
if (!this.information) {
this.information = new RefreshingReplaySubject(
() => this.restService.get('/information/'),
60000
);
}
return this.information;
}
To implement this, you will need to create your own observable with custom logic on subscribtion:
function createTimedCache(doRequest, expireTime) {
let lastCallTime = 0;
let lastResult = null;
const result$ = new Rx.Subject();
return Rx.Observable.create(observer => {
const time = Date.now();
if (time - lastCallTime < expireTime) {
return (lastResult
// when result already received
? result$.startWith(lastResult)
// still waiting for result
: result$
).subscribe(observer);
}
const disposable = result$.subscribe(observer);
lastCallTime = time;
lastResult = null;
doRequest()
.do(result => {
lastResult = result;
})
.subscribe(v => result$.next(v), e => result$.error(e));
return disposable;
});
}
and resulting usage would be following:
this.information = createTimedCache(
() => this.restService.get('/information/'),
60000
);
usage example: https://jsbin.com/hutikesoqa/edit?js,console
how to use dispatcher.BeginInvoke in for loop( httpwebrequest).With each dispatcher.BeginInvoke have complete before call another dispatcher.BeginInvoke. Because objects return by httpwerequest are wrong position.
No, BeginInvoke is asynchronous - you're basically adding delegates to a queue of items to be executed on the UI thread.
If you need to wait until the delegate has executed before you continue work in your background thread, you'll need to do a bit of work yourself, as Silverlight doesn't support the synchronous Dispatcher.Invoke method, or the DispatcherOperation.Wait() method. Silverlight tries to avoid synchronous approaches like this - if you can possibly redesign your code so that you don't need to wait, that would be preferable.
Being able to easily convert a synchronous sequence of operations into asynchrounous code has been a subject I've blogged about a fair bit. If you want to take up my approach you will need to add the following (relatively small) chunks of code:
The core AsyncOperationService
Code to create an AsyncOperation from the .NET Async Pattern
A couple of Extension methods for WebRequest
Here is some example code that has the flavour of what you describe in your question:-
IEnumerable<AsyncOperation> LoadSomeStuff(IList<string> urls)
{
for (string url in urls)
{
yield return AsyncOperationService.SwitchToBackgroundThread();
WebRequest req = WebRequest.Create(url);
WebResponse resp = null;
yield return req.GetResponseAsyncOp(r => resp = r);
using (resp)
{
// Do stuff with the Web Response such as construct model class instances from a stream.
}
// When ready to actually start touching the UI
yield return AsyncOperationService.SwitchToUIThread();
// Do stuff to the UI
}
}
usage:
List<string> urls = new List<string> {"pinkElephants.xml", "whileElephants.xml"}
LoadSomeStuff(urls).Run(err =>
{
if (err == null)
{
// Cool, it all worked and I probably don't need to do anything
}
else
{
// Something bad happened, lets tell the user about it in the UI somehow.
}
});
Note that this isn't the most efficient code possible. However in many cases the time it takes HTTP response to be delivered massively out-weighs the time the rest of the code uses up so the inefficiency can be quite small and well worth the reduced complexity of code.
Boot Pros,
I recently started to program in spring-boot and I stumbled upon a question where I would like to get your opinion on.
What I try to achieve:
I created a Controller that exposes a GET endpoint, named nonBlockingEndpoint. This nonBlockingEndpoint executes a pretty long operation that is resource heavy and can run between 20 and 40 seconds.(in the attached code, it is mocked by a Thread.sleep())
Whenever the nonBlockingEndpoint is called, the spring application should register that call and immediatelly return an Operation ID to the caller.
The caller can then use this ID to query on another endpoint queryOpStatus the status of this operation. At the beginning it will be started, and once the controller is done serving the reuqest it will be to a code such as SERVICE_OK. The caller then knows that his request was successfully completed on the server.
The solution that I found:
I have the following controller (note that it is explicitely not tagged with #Async)
It uses an APIOperationsManager to register that a new operation was started
I use the CompletableFuture java construct to supply the long running code as a new asynch process by using CompletableFuture.supplyAsync(() -> {}
I immdiatelly return a response to the caller, telling that the operation is in progress
Once the Async Task has finished, i use cf.thenRun() to update the Operation status via the API Operations Manager
Here is the code:
#GetMapping(path="/nonBlockingEndpoint")
public #ResponseBody ResponseOperation nonBlocking() {
// Register a new operation
APIOperationsManager apiOpsManager = APIOperationsManager.getInstance();
final int operationID = apiOpsManager.registerNewOperation(Constants.OpStatus.PROCESSING);
ResponseOperation response = new ResponseOperation();
response.setMessage("Triggered non-blocking call, use the operation id to check status");
response.setOperationID(operationID);
response.setOpRes(Constants.OpStatus.PROCESSING);
CompletableFuture<Boolean> cf = CompletableFuture.supplyAsync(() -> {
try {
// Here we will
Thread.sleep(10000L);
} catch (InterruptedException e) {}
// whatever the return value was
return true;
});
cf.thenRun(() ->{
// We are done with the super long process, so update our Operations Manager
APIOperationsManager a = APIOperationsManager.getInstance();
boolean asyncSuccess = false;
try {asyncSuccess = cf.get();}
catch (Exception e) {}
if(true == asyncSuccess) {
a.updateOperationStatus(operationID, Constants.OpStatus.OK);
a.updateOperationMessage(operationID, "success: The long running process has finished and this is your result: SOME RESULT" );
}
else {
a.updateOperationStatus(operationID, Constants.OpStatus.INTERNAL_ERROR);
a.updateOperationMessage(operationID, "error: The long running process has failed.");
}
});
return response;
}
Here is also the APIOperationsManager.java for completness:
public class APIOperationsManager {
private static APIOperationsManager instance = null;
private Vector<Operation> operations;
private int currentOperationId;
private static final Logger log = LoggerFactory.getLogger(Application.class);
protected APIOperationsManager() {}
public static APIOperationsManager getInstance() {
if(instance == null) {
synchronized(APIOperationsManager.class) {
if(instance == null) {
instance = new APIOperationsManager();
instance.operations = new Vector<Operation>();
instance.currentOperationId = 1;
}
}
}
return instance;
}
public synchronized int registerNewOperation(OpStatus status) {
cleanOperationsList();
currentOperationId = currentOperationId + 1;
Operation newOperation = new Operation(currentOperationId, status);
operations.add(newOperation);
log.info("Registered new Operation to watch: " + newOperation.toString());
return newOperation.getId();
}
public synchronized Operation getOperation(int id) {
for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
return op;
}
}
Operation notFound = new Operation(-1, OpStatus.INTERNAL_ERROR);
notFound.setCrated(null);
return notFound;
}
public synchronized void updateOperationStatus (int id, OpStatus newStatus) {
iteration : for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
op.setStatus(newStatus);
log.info("Updated Operation status: " + op.toString());
break iteration;
}
}
}
public synchronized void updateOperationMessage (int id, String message) {
iteration : for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
op.setMessage(message);
log.info("Updated Operation status: " + op.toString());
break iteration;
}
}
}
private synchronized void cleanOperationsList() {
Date now = new Date();
for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if((now.getTime() - op.getCrated().getTime()) >= Constants.MIN_HOLD_DURATION_OPERATIONS ) {
log.info("Removed operation from watchlist: " + op.toString());
iterator.remove();
}
}
}
}
The questions that I have
Is that concept a valid one that also scales? What could be improved?
Will i run into concurrency issues / race conditions?
Is there a better way to achieve the same in boot spring, but I just didn't find that yet? (maybe with the #Async directive?)
I would be very happy to get your feedback.
Thank you so much,
Peter P
It is a valid pattern to submit a long running task with one request, returning an id that allows the client to ask for the result later.
But there are some things I would suggest to reconsider :
do not use an Integer as id, as it allows an attacker to guess ids and to get the results for those ids. Instead use a random UUID.
if you need to restart your application, all ids and their results will be lost. You should persist them to a database.
Your solution will not work in a cluster with many instances of your application, as each instance would only know its 'own' ids and results. This could also be solved by persisting them to a database or Reddis store.
The way you are using CompletableFuture gives you no control over the number of threads used for the asynchronous operation. It is possible to do this with standard Java, but I would suggest to use Spring to configure the thread pool
Annotating the controller method with #Async is not an option, this does not work no way. Instead put all asynchronous operations into a simple service and annotate this with #Async. This has some advantages :
You can use this service also synchronously, which makes testing a lot easier
You can configure the thread pool with Spring
The /nonBlockingEndpoint should not return the id, but a complete link to the queryOpStatus, including id. The client than can directly use this link without any additional information.
Additionally there are some low level implementation issues which you may also want to change :
Do not use Vector, it synchronizes on every operation. Use a List instead. Iterating over a List is also much easier, you can use for-loops or streams.
If you need to lookup a value, do not iterate over a Vector or List, use a Map instead.
APIOperationsManager is a singleton. That makes no sense in a Spring application. Make it a normal PoJo and create a bean of it, get it autowired into the controller. Spring beans by default are singletons.
You should avoid to do complicated operations in a controller method. Instead move anything into a service (which may be annotated with #Async). This makes testing easier, as you can test this service without a web context
Hope this helps.
Do I need to make database access transactional ?
As long as you write/update only one row, there is no need to make this transactional as this is indeed 'atomic'.
If you write/update many rows at once you should make it transactional to guarantee, that either all rows are updated or none.
However, if two operations (may be from two clients) update the same row, always the last one will win.