I need to create a queue where when an item is added to the queue a subscriber should be notified. Only solution comes to my brain is to use a while loop to poll the queue or use peek the queue compare whether a new object is added to the queue, if new item found then use onNext() to forward the item to subscriber. It would be an endless loop and that is what I need to observe the queue.
You could sub-class one of the queues and have the add/offer/etc... methods call your Observable.
public class SubscriberQueue<E> extends LinkedList<E> {
private Observable<E> mObservable = PublishSubscriber.create();
#Override
public boolean add(E e) {
if (super.add()) {
mObservable.onNext(e);
}
}
public Observable<E> getObservable() {return mObservable;}
}
// Test code
SubscriberQueue<Integer> myQueue = new SubscriberQueue<>();
myQueue.getObservable().subscribe(i -> System.out.println("Added: " + i));
myQueue.add(1);
Related
I have a Spring Boot application that spins consumers for a section of queues and I want to be able to add queues to those consumers at run time.
I've installed the event exchange plugin (https://www.rabbitmq.com/event-exchange.html) and I created a dedicated queue that is bound to the amq.rabbitmq.event exchange. I can see the events coming in when I declare the queues statically.
How would I accomplish this run time magic? I've seen people use the property file, but I would prefer not to have to modify the property file during run time as I add more queues
#Component
public class MessageConsumer {
List<String> allQueues = new ArrayList<String>();
public MessageConsumer() {
allQueues.add("queue1");
allQueues.add("queue2");
allQueues.add("queue3");
}
#RabbitListener(id = "event", queues = {"custom-emp-queue-events"}) // create this queue in rabbitmq management, bound to amqp exchange
public void processQueueEvents(Message message) {
... add the queue to the allQueues list on queue.created ...
}
#RabbitListener(id = "process", queues = allQueues.stream().toArray(String[]::new) ) // this is where the "issue" is
public void processMessageFromQueues(String messageAsJson) {
... process message ...
}
}
This can be done with a SpEL expression over there:
#RabbitListener(id = "process", queues = "#{messageConsumer.allQueues}" )
But you have to add a public getter for that allQueues.
See more info in the Reference Manual: https://docs.spring.io/spring-amqp/docs/2.1.3.RELEASE/reference/html/_reference.html#async-annotation-driven
UPDATE
#Autowired
private RabbitListenerEndpointRegistry listenerEdnpointRegistry;
#RabbitListener(id = "event", queues = {"custom-emp-queue-events"}) // create this queue in rabbitmq management, bound to amqp exchange
public void processQueueEvents(Message message) {
((AbstractMessageListenerContainer) this.listenerEdnpointRegistry.getListenerContainer("process")).addQueueNames(...);
}
Complete structure of the program
Annotation:
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.METHOD)
public #interface UserAnnotation {
}
Then created a Interceptor:
public class UserInterceptor implements MethodInterceptor {
private static final Logger logger = LoggerFactory.getLogger(UserInterceptor.class);
#Inject
UserService userService; // this is not working
public Object invoke(MethodInvocation invocation) throws Throwable {
logger.info("UserInterceptor : Interceptor Invoked");
Object result = invocation.proceed();
Observable<List<User>> observable = (Observable<List<Sample>>) result;
observable.flatMap(Observable::from).subscribe(object -> {
User user = (User)object
SampleSender sender = new SampleSender();
sender.setBoolean(user.isBoolean());
logger.info("Pushing Data into Sender");
userService.insert(String.join("_", "key", "value"), sender);
}
return result;
}
}
Then I created a GuiceModule as below:-
public class UserModule extends AbstractModule {
#Override
protected void configure() {
SampleInterceptor interceptor = new SampleInterceptor()
requestInjection(interceptor);
bindInterceptor(Matchers.any(), Matchers.annotatedWith(SampleAnnotation.class), interceptor);
}
}
Class in which I am using the above annotation is
// This class also have so many method and this was already declared and using in another services, I created a sample class here
class UserClassForInterceptor {
#Inject
AnotherClass anotherClass;
// this userMethod() is not a new method, its already created,
// now I am adding annotation to it, because after finishing this functionality,
// I want something should be done, so created annotation and added here
#UserAnnotation
public Observable<List<Sample>> userMethod() {
logger.info("This is printing only once");
return anotherClass.getUser().flatMap(user ->{
logger.info("This is also printing twice");
// this logger printed twise means, this code snippet is getting executed twise
});
}
}
public class AnotherClass{
public Observable<User> getUser(){
Observable<Sample> observableSample = methodReturnsObservableSample();
logger.info("Getting this logger only once");
return observableSample.map(response-> {
logger.info("This logger is printing twice");
//here have code to return observable of User
});
}
}
If I remove annotation loggers inside the observable are printing only one time but when I use annotation those loggers are getting printed twise. Why it is behaving like this I dont know.
I have a RestModule using which I am binding UserClassForInterceptor as follows
public final class RestModule extends JerseyServletModule {
// other classes binding
bind(UserClassForInterceptor.class).in(Scopes.SINGLETON);
// other classes binding
install(new SampleModule());
}
Now I have a bootsrap class in which I am binding RestModule
public class Bootstrap extends ServerBootstrap {
binder.install(new RestModule());
}
Usage:-
#Path("service/sample")
public class SampleRS {
#Inject
UserClassForInterceptor userClassForInterceptor;
public void someMethod() {
userClassForInterceptor.sampleMethod();
}
}
You created an annotation, #UserAnnotation, and an interceptor class to go with the annotation. You attach the annotation to a method, userMethod().
The first thing your interceptor routine does is invoke userMethod() to get the observable that it returns and then the interceptor subscribes to the returned observable, causing the first log messages to appear. Eventually, the interceptor returns the observable to the original caller. When something else subscribes to the returned observable, the observer chain is activated a second time, hence the log messages appear twice.
RxJava Has Side Effects
While RxJava is an implementation of the "functional reactive programming" concept, the observer chains that you construct (in a functional manner) only work when they are subscribed to, and those subscriptions have side effects. Logging output is one side effect, and probably the most benign; changes to variables or invocations of methods that have side effects have a wider impact.
When an observer chain is constructed (properly), it acts as a potential computation until there is a subscriber. If you need to have more than one subscriber, as you might for your problem domain, then you have to decide whether the observer chain needs to be activated for each subscription, the normal case, or only once for all overlapping subscriptions.
If you want all overlapping subscriptions to share the same observable, then you can use the share() operator. There are a number of related operators that affect the lifetime of observables and subscriptions. Here is an overview: How to use RxJava share() operator?
Aspect Oriented Programming: Interceptors And Guice
Your code is using Guice to provide a capability called "aspect oriented programming". This allows you to introduce code into your program to address cross-cutting concerns, or to enhance its functionality by setting up controlled gateways. Using Guice, or similar AOP approaches, requires discipline.
In your case, you used the interception process to cause unexplained (until now) side effects by subscribing to an observer chain that has non-trivial side effects. Imagine that the method you intercepted set up a one-time connection and that your interceptor used up that connection doing its work, leaving the original caller unable to use the connection.
The discipline you need is to understand the rules that the interceptor must follow. Think of rules such as "First, do no harm".
Doing Things The FRP Way
If you need to add an extra step when handling user information, then you should construct a new observable in your interceptor that does that, but only when the original caller subscribed to the observable:
Object result = invocation.proceed();
Observable<List<User>> observable = (Observable<List<Sample>>) result;
Observable<List<User>> newObservable = observable
.doOnNext( sampleList ->
Observable.fromIterable( sampleList )
.subscribe(object -> {
User user = (User)object
SampleSender sender = new SampleSender();
sender.setBoolean(user.isBoolean());
logger.info("Pushing Data into Sender");
userService.insert(String.join("_", "key", "value"), sender);
}));
return newObservable;
By returning a modified observer chain, you don't introduce side effects from the original observer chain, and ensure that the side effects you introduce in your own code will only be triggered when the original observer chain is subscribed to.
This code also helped me
public Object invoke(MethodInvocation invocation) throws Throwable {
Object result = null;
try{
logger.debug("Interceptor Invoked");
result = invocation.proceed();
Observable<List<User>> observable = (Observable<List<User>>)result;
return observable
.doOnNext(this::updateUser);
}
catch(Exception ex){
logger.error("Error: ",ex);
}
return result;
}
private void updateUser(List<User> users) {
if(CollectionUtils.isNotEmpty(users)) {
for(User user: users) {
SampleSender sender = new SampleSender();
sender.setBoolean(user.isBoolean());
logger.info("Pushing Data into Sender");
userService.insert(String.join("_", "key", "value"), sender);
}
}
}
I am able to stop the consuming and restart the consuming but the problem is that when I am restarting the consuming, I am able to process the already published message but when I publish the new messages those are not able to process.
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Consumer;
#Component
public class RabbitMqueue implements Consumer {
int count = 0;
#RabbitListener(queues="dataQueue")
public void receivedData(#Payload Event msg, Channel channel,
#Header(AmqpHeaders.CONSUMER_TAG) String tag) throws IOException,
InterruptedException {
count++;
System.out.println("\n Message recieved from the Dataqueue is " + msg);
//Canceling consuming working fine.
if(count == 1) {
channel.basicCancel(tag);
System.out.println("Consumer is cancle");
}
count++;
System.out.println("\n count is " + count + "\n");
Thread.sleep(5000);
//restarting consumer. able to process already consumed messages
//but not able to see the newly published messages to the queue I mean
//newly published message is moving from ready to unack state but nothing
//happening on the consumer side.
if(count == 2) {
channel.basicConsume("dataQueue", this);
System.out.println("Consumer is started ");
}
}
}
You must not do this channel.basicCancel(tag).
The channel/consumer are managed by Spring; the only thing you should do with the consumer argument is ack or nack messages (and even that is rarely needed - it's better to let the container do the acks).
To stop/start the consumer, use the endpoint registry as described in the documentation.
Containers created for annotations are not registered with the application context. You can obtain a collection of all containers by invoking getListenerContainers() on the RabbitListenerEndpointRegistry bean. You can then iterate over this collection, for example, to stop/start all containers or invoke the Lifecycle methods on the registry itself which will invoke the operations on each container.
e.g. registry.stop() will stop all the listeners.
You can also get a reference to an individual container using its id, using getListenerContainer(String id); for example registry.getListenerContainer("multi") for the container created by the snippet above.
If your are using AMQP/Rabbit, you can try one of these:
1) Prevent starting at startup in code:
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
//
//autoStartup = false, prevents handling messages immedeatly. You need to start each listener itselve.
//
factory.setAutoStartup(false);
factory.setMessageConverter(new Jackson2JsonMessageConverter());
return factory;
}
2) Prevent starting at startup in in app.yml/props:
rabbitmq.listener.auto-startup: false
rabbitmq.listener.simple.auto-startup: false
3) Start/stop individual listeners
give your #RabbitListener a id:
#RabbitListener(queues = "myQ", id = "myQ")
...
and :
#Autowired
private RabbitListenerEndpointRegistry rabbitListenerEndpointRegistry;
MessageListenerContainer listener =
rabbitListenerEndpointRegistry.getListenerContainer("myQ");
...
listener.start();
...
listener.stop();
I wish to batch and process items as they come along so i created a UnicastProcessor and subscribed to it like this
UnicastProcessor<String> processor = UnicastProcessor.create()
processor
.bufferTimeout(10, Duration.ofMillis(500))
.subscribe(new Subscriber<List<String>>() {
#Override
public void onSubscribe(Subscription subscription) {
System.out.println("OnSubscribe");
}
#Override
public void onNext(List<String> strings) {
System.out.println("OnNext");
}
#Override
public void onError(Throwable throwable) {
System.out.println("OnError");
}
#Override
public void onComplete() {
System.out.println("OnComplete");
}
});
And then for testing purposes i created a new thread and started adding items in a loop
new Thread(() -> {
int limit = 100
i = 0
while(i < limit) {
++i
processor.sink().next("Hello $i")
}
System.out.println("Published all")
}).start()
After running this (and letting the main thread sleep for 5 seconds) i can see that all item have been published, but the subscriber does not trigger on any of the events so i can't process any of the published items.
What am I doing wrong here?
Reactive Streams specification is the answer!
The total number of onNext´s signalled by a Publisher to a Subscriber
MUST be less than or equal to the total number of elements requested
by that Subscriber´s Subscription at all times. [Rule 1.1]
In your example, you just simply provide a subscriber who does nothing in any sense. In turn, Reactive Streams specification, directly says that nothing will happen (there will be no onNext invocation) if you have not called Subscription#request method
A Subscriber MUST signal demand via Subscription.request(long n) to
receive onNext signals. [Rule 2.1]
Thus, to fix your problem, one of the possible solutions is changing the code in the following way:
UnicastProcessor<String> processor = UnicastProcessor.create()
processor
.bufferTimeout(10, Duration.ofMillis(500))
.subscribe(new Subscriber<List<String>>() {
#Override
public void onSubscribe(Subscription subscription) {
System.out.println("OnSubscribe");
subscription.request(Long.MAX_VALUE);
}
#Override
public void onNext(List<String> strings) {
System.out.println("OnNext");
}
#Override
public void onError(Throwable throwable) {
System.out.println("OnError");
}
#Override
public void onComplete() {
System.out.println("OnComplete");
}
});
Note, in this example demand in size Long.MAX_VALUE means an unbounded demand so that all messages will be directly pushed to the given Subscriber [Rule 3.17]
Use UnicatProcessor correctly
On the one hand, your example will work correctly with mentioned fixes. However, on the other hand, each invocation of FluxProcessor#sink() (yeah sink is FluxProcessor's method) will lead to a redundant calling of UnicastProcessor's onSubscribe method, which under the hood cause a few atomic reads and writes which might be avoided if create FluxSink once and safely use it as many tame as needed. For example:
UnicastProcessor<String> processor = UnicastProcessor.create()
FluxSink<String> sink = processor.serialize().sink();
...
new Thread(() -> {
int limit = 100
i = 0
while(i < limit) {
++i
sink.next("Hello $i")
}
System.out.println("Published all")
}).start()
Note, in this example, I executed an additional method serialize which provide thread-safe sink and ensure that the calling of FluxSink#next concurrently will not cause a violation of the ReactiveStreams spec.
I am running RxJava and creating a subject to use onNext() method to produce data. I am using Spring.
This is my setup:
#Component
public class SubjectObserver {
private SerializedSubject<SomeObj, SomeObj> safeSource;
public SubjectObserver() {
safeSource = PublishSubject.<SomeObj>create().toSerialized();
**safeSource.subscribeOn(<my taskthreadExecutor>);**
**safeSource.observeOn(<my taskthreadExecutor>);**
safeSource.subscribe(new Subscriber<AsyncRemoteRequest>() {
#Override
public void onNext(AsyncRemoteRequest asyncRemoteRequest) {
LOGGER.debug("{} invoked.", Thread.currentThread().getName());
doSomething();
}
}
}
public void publish(SomeObj myObj) {
safeSource.onNext(myObj);
}
}
The way new data is generated on the RxJava stream is by #Autowire private SubjectObserver subjectObserver
and then calling subjectObserver.publish(newDataObjGenerated)
No matter what I specify for subscribeOn() & observeOn():
Schedulers.io()
Schedulers.computation()
my threads
Schedulers.newThread
The onNext() and the actual work inside it is done on the same thread that actually calls the onNext() on the subject to generate/produce data.
Is this correct? If so, what am I missing? I was expecting the doSomething() to be done on a different thread.
Update
In my calling class, if I change the way I am invoking the publish method, then of course a new thread is allocated for the subscriber to run on.
taskExecutor.execute(() -> subjectObserver.publish(newlyGeneratedObj));
Thanks,
Each operator on Observable/Subject return a new instance with the extra behavior, however, your code just applies the subscribeOn and observeOn then throws away whatever they produced and subscribes to the raw Subject. You should chain the method calls and then subscribe:
safeSource = PublishSubject.<AsyncRemoteRequest>create().toSerialized();
safeSource
.subscribeOn(<my taskthreadExecutor>)
.observeOn(<my taskthreadExecutor>)
.subscribe(new Subscriber<AsyncRemoteRequest>() {
#Override
public void onNext(AsyncRemoteRequest asyncRemoteRequest) {
LOGGER.debug("{} invoked.", Thread.currentThread().getName());
doSomething();
}
});
Note that subscribeOn has no practical effect on a PublishSubject because there is no subscription side-effect happening in its subscribe() method.