I'm trying to configure a lambda function to consume from an SQS queue that I've been given read and delete permissions to, that I do not own/have configuration of. Is there a way to use lambda's SQS trigger functionality for a queue that doesn't exist inside my AWS account?
If not, what are some alternatives that don't include checking the queue on a scheduled event.
If the owner of the SQS queue gives you the necessary permissions (see the setup docs for what those permissions are), you can do this. But, you shouldn't.
Subscribing to someone else's SQS queue is an anti-pattern. This is because a queue represents a backlog of work, and the implicit functionality is that everything that goes in eventually comes out. All the queue does is separate input flow from output flow (data can flow in both faster and slower than they flow out).
This idea of flow, however, means that when something comes out, it's no longer in the queue. (Caveat here: there are work-arounds to this, but they're usually not preferred). A consumer, however, always has the goal of processing everything in the queue. This may be done by multiple threads under the control of one consumer, but the end result is still that everything is processed. If there are multiple consumers, then they by necessity compete with one another, and none of them get to process everything in the queue.
How do we ensure there aren't multiple consumers? Simple: the consumer owns the queue. No other consumer is granted read permissions. It might well be the case that someone other than the consumer controls the filling of the queue (receiving write permissions) - and AWS has the perfect solution for this:
SNS Topics: An SNS topic is a source of data. It is, in effect, a publisher. When someone else wants you to have access to their data, they allow you to become a subscriber to their topic. When a new message is published to the SNS topic, everyone who is subscribed to the topic gets a copy. What happens to that copy is decided by the subscriber: it may be acted upon directly, stored for later action, or acted on indirectly, e.g. by being placed in a queue. This is the Pub-Sub model. It separates the details of one entity (the publisher) creating messages and sending them out to many others, from each recipient's (subscriber's) individual decision about how to consume those messages.
TL;DR: get whoever is currently owning the queue to publish to an SNS topic instead, then set up a queue (or whatever you prefer) subscribed to that topic.
Related
I'm looking for a method to programmatically identify the triggers associated with an SQS queue. Looking through the SQS sdk docs, it doesn't seem this is possible. I had thought instead to try from the other end, and it appears the Lambda ListEventSourceMappings function would likely do what I want, since I'm able to provide it with the queue ARN. However, this requires the ListSourceMappings permission on all lambdas (*), which isn't really ideal - though it shouldn't really hurt, just not what I want. Is there another mechanism for this that I'm missing, or another approach?
Lambda polls SQS queues. It doesn't appear that way in the console, because they hide some of the details from you, but behind the scenes there is a process running within the AWS Lambda system that is polling your SQS queue and invoking your Lambda function when a message is available.
SQS doesn't push messages to Lambda (or anywhere else). SQS just holds messages and hands them out to anything that asks for them. So from an SQS perspective, there is no knowledge of who the message consumers are.
Given the above, the only way to find what you want is to use the Lambda ListEventSourceMappings API.
I have 2 systems where System A has to send messages to System B. I am new to JMS so I don't have a big idea on how to implement this. I was thinking of using a message broker (ActiveMQ) to send messages. So, System A will send messages to a queue and the Message Listener in B will consume those messages. There are many users in System B and I want these messages to be shown whenever the user log into the system. So my problem is, if System B keeps consuming messages even when the users are not logged in how can they see the messages which are already consumed?
Should I store the consumed messages in a database? I don't understand how this works.
You could let SystemB read all the messages and act as a gateway between JMS and the system itself, i.e. store each user message in the database and when a user logs in, read those messages from the database and display them. If the user has to acknowledge they've read each message that might be a better solution as you can then track if they've read them all and delete each one from the database as they acknowledge they've read it.
Another solution might be virtual topics and queues. A single topic that messages are sent to that is split into queues, one per user. When a user logs in, SystemB reads from that user's queue. This is separate from the application's domain (it's JMS at this point) so the message is marked as consumed and is taken off the queue by ActiveMQ. If the user doesn't read it and needs to see it the next time they login then you need the database solution.
It's essentially where two domains meet. The information the user needs to see comes in on JMS, which has its own rules (message consumed, remove from queue etc). The information then enters your application's domain which might have different rules (must read, save for next login etc).
If a user doesn't login for a long period of time, their queue might fill up and not be able to receive any more messages, whereas, if the messages are always read and stored in a database it doesn't matter how frequently they login as the database should be better at holding large amounts of messages.
Another option is one topic per user and messages are sent to those topics but other systems would need to know which users are in your system, which probably isn't a good idea. Or you could use Apache Camel to route incoming messages on the main topic to user topics. The messages would need to be durable and transacted in case the broker went down. When a user logs in, read from the topic to get all their messages. You can route based on content or headers.
Your problem is one of message persistence and or re-delivery. There are a couple of approaches:
JMS Durable subscription: you could make a durable subscription on a topic from system B and only consume messages while users are logged in. When you don't receive() messages, your messages will be held at the broker for you until you call receive() again. In case A sent messages persistently, all of this is saved by the ActiveMQ broker on disk.
JMS Queue: system A puts messages into a queue and System B doesn't pick up the messages unless users are logged in. The queue will get bigger until you call receive() again from system B. Similar to durable subscriptions, but with a queue you can only have one consumer for each message. With durable subscriptions it's a easier to configure a fault-tolerant version of system B...
Add a 'replay server' for n-times delivery: system A publishes to a topic (could be non-persistent) and a third component (C) would also subscribe to every message and persist to disk. When system B needs to see a message again, it could ask system C for those messages, ideally supporting from_time, or similar.
I have a Topic that has two subscribers. However, one of the subscribers will be deployed in a cluster. So, in effect (I think?) the cluster will be seen as multiple subscribers.
I need to make sure only one message is consumed by the cluster.
I do not know the options to do this.
Is it possible to insert a queue in between the cluster and the topic, and then have the clustered application only work with the inserted queue?
If this is possible, how, in general, to make the queue "subscribe" to the Topic?
Another idea I heard is to make cloned subscribers in the cluster, anyone hear of this before to solve this issue?
Another idea is to do away with the Topic and just use two queues in its place, this seems to be a safe solution?
Using WebSphere MQ
You can use a durable subscription to achieve this.
Administratively create a durable subscription and specify a destination, basically a queue, which will receive the publications that are published on the specified topic. Your clustered application can then get messages from that destination. Since the publications are being received from a queue, only one instance of your application will get message(s).
The command below creates durable subscription for topic "/SPORTS/HOCKEY" with destination queue as Q1.
DEFINE SUB(SPORTSUB) TOPICSTR(/SPORTS/HOCKEY) TOPICOBJ(SPORT) DESTQ(Q1)
In JMS there are Queues and Topics. As I understand it so far queues are best used for producer/consumer scenarios, where as topics can be used for publish/subscribe. However in my scenario I need a way to combine both approaches and create a producer-consumer-observer architecture.
Particularly I have producers which write to some queues and workers, which read from these queues and process the messages in those queues, then write it to a different queue (or topic). Whenever a worker has done a job my GUI should be notified and update its representation of the current system state. Since workers and GUI are different processes I cannot apply a simple observer pattern or notify the GUI directly.
What is the best way to realize this using a combination of queues and/or topics? The GUI should always be notified, but it should never consume anything from a queue?
I would like to solve this with JMS directly and not use any additional technology such as RMI to implement the observer part.
To give a more concrete example:
I have a queue with packages (PACKAGEQUEUE), produced by machine (PackageProducer)
I have a worker which takes a package from the PACKAGEQUEUE adds an address and then writes it to a MAILQUEUE (AddressWorker)
Another worker processes the MAILQUEUE and sends the packages out by mail (MailWorker).
After step 2. when a message is written to the MAILQUEUE, I want to notify the GUI and update the status of the package. Of course the GUI should not consume the messages in the MAILQUEUE, only the MailWorker must consume them.
You can use a combination of queue and topic for your solution.
Your GUI application can subscribe to a topic, say MAILQUEUE_NOTIFICATION. Every time (i.e at step 2) PackageProducer writes message to MAILQUEUE, a copy of that message should be published to MAILQUEUE_NOTIFICATION topic. Since the GUI application has subscribed to the topic, it will get that publication containing information on status of the package. GUI can be updated with the contents of that publication.
HTH
I used the Subscriber SYSTEM.JMS.D.SUBSCRIBER.QUEUE and Client ID as setClientID("USER1") and used topicSubscriber = topicSession.createDurableSubscriber(topic,"SUB1");
The topicSubscriber is created and while trying to receive using this topicSubscriber.receive(); , it is not receiving the messages from topic , but there are messages in topic.
Can any one say why its not receiving messages and whether i need to chek any queue configurations.
Any help is appreciated.
Thanks in advance.
I already had topicConnection.Start() in my coding , also i checked in TopicSession there is no Start() Method.
The same code with Non durable subscriber method topicSession.createSubscriber(topic); is working , but for durable it is not working.
Thanks
Sorry, yes I meant topicConnection.Start(). It was worth a check.
I got the answer for durable subscriber not working ,
My queue depth has reached the max queue depth , so the subscriber is not able to subscribe the message.
Eg . my max queue depth for queue SYSTEM.JMS.D.SUBSCRIBER.QUEUE is set to 100 , and if we check our current queue depth and if it reaches 100 the subscriber will not work.
As an alternative way iam trying to create with Temporary Topic , here iam getting an error while creating the durable subscriber topicSession.createDurableSubscriber(topic,"SUB1");
JMS Exception :: javax.jms.InvalidDestinationException: MQJMS0003:
Destination not understood or no longer valid
Can anyone help to solve this error.
Thanks in Advance.
The problem seems to be how you are using SYSTEM.JMS.D.SUBSCRIBER.QUEUE. You appear to be directing messages and subscribers to this queue as the destination for a durable subscription. IBM MQ uses that queue to manage durable subscriptions.
As a general rule, queues whose names begin with SYSTEM are for internal system use by MQ. Some of them, such as SYSTEM.ADMIN.*.EVENT are OK to get messages from but you would not use these as a subscription destination for unrelated messages either.
Many tutorials use SYSTEM.DEFAULT.LOCAL.QUEUE as a destination for messages but this is only because the queue is known to exist on all versions of MQ and MQ uses only the definition of the queue and never the content of that queue. It is easier for the tutorial writer (and IBM is just as guilty here) to point at SYSTEM.DEFAULT.LOCAL.QUEUE then to walk the student through the need for and means to create their own queue. So although best practices suggest it should not be an exception to the "do not use SYSTEM objects" rule, common usage makes SYSTEM.DEFAULT.LOCAL.QUEUE the de facto exception.
The other exceptions are, of course, the command queues for MQ, MFT and IIB. These are also names SYSTEM.* but are designed for users to communicate with the software listening on the queue.
Note that all the exceptions "do not use SYSTEM objects" rule are interfaces between applications and MQ system resources. The event queues are MQ sending information to the user. The command queues are the user sending information to the system components. A subscription is neither of these categories. A destination for a subscription is considered an application-owned object, even when the system manages it on behalf of the subscriber.
When you want a durable subscription, either let the system assign a permanent queue and use that, or else pre-define a queue (that is not named SYSTEM.*) and use that. Whatever else you do, please do NOT try to hijack MQ's internal system queues for application-level purposes.