Imagine an application(A1) running on a server(S1) makes a client connection to a Queue Manager(QM1) running on a different server(S2). Is it possible to extend the functionality of Queue Manager so that when the application makes a MQGET or MQPUT call on a queue on the Queue Manager, a message can be logged to a log file or some custom action can be performed?
Thanks,
Yash
An API Exit can be used to log the API calls made by an application. This API exit can run on either S1 or S2 from your question.
The sample API Exit supplied with MQ (amqsaxe) would be a good starting point as that's all it does.
Remember that this will have a detrimental effect on the performance of your application.
Further Reading
The API exit sample program
Related
Problem:
Suppose there are two services A and B. Service A makes an API call to service B.
After a while service A falls down or to be lost due to network errors.
How another services will guess that an outbound call from service A is lost / never happen? I need some another concurrent app that will automatically react (run emergency code) if service A outbound CALL is lost.
What are cutting-edge solutions exist?
My thoughts, for example:
service A registers a call event in some middleware (event info, "running" status, timestamp, etc).
If this call is not completed after N seconds, some "call timeout" event in the middleware automatically starts the emergency code.
If the call is completed at the proper time service A marks the call status as "completed" in the same middleware and the emergency code will not be run.
P.S. I'm on Java stack.
Thanks!
I recommend to look into patterns such as Retry, Timeout, Circuit Breaker, Fallback and Healthcheck. Or you can also look into the Bulkhead pattern if concurrent calls and fault isolation are your concern.
There are many resources where these well-known patterns are explained, for instance:
https://www.infoworld.com/article/3310946/how-to-build-resilient-microservices.html
https://blog.codecentric.de/en/2019/06/resilience-design-patterns-retry-fallback-timeout-circuit-breaker/
I don't know which technology stack you are on but usually there is already some functionality for these concerns provided already that you can incorporate into your solution. There are libraries that already take care of this resilience functionality and you can, for instance, set it up so that your custom code is executed when some events such as failed retries, timeouts, activated circuit breakers, etc. occur.
E.g. for the Java stack Hystrix is widely used, for .Net you can look into Polly .Net to make use of retry, timeout, circuit breaker, bulkhead or fallback functionality.
Concerning health checks you can look into Actuator for Java and .Net core already provides a health check middleware that more or less provides that functionality out-of-the box.
But before using any libraries I suggest to first get familiar with the purpose and concepts of the listed patterns to choose and integrate those that best fit your use cases and major concerns.
Update
We have to differentiate between two well-known problems here:
1.) How can service A robustly handle temporary outages of service B (or the network connection between service A and B which comes down to the same problem)?
To address the related problems the above mentioned patterns will help.
2.) How to make sure that the request that should be sent to service B will not get lost if service A itself goes down?
To address this kind of problem there are different options at hand.
2a.) The component that performed the request to service A (which than triggers service B) also applies the resilience patterns mentioned and will retry its request until service A successfully answers that it has performed its tasks (which also includes the successful request to service B).
There can also be several instances of each service and some kind of load balancer in front of these instances which will distribute and direct the requests to an available instance (based on regular performed healthchecks) of the specific service. Or you can use a service registry (see https://microservices.io/patterns/service-registry.html).
You can of course chain several API calls after another but this can lead to cascading failures. So I would rather go with an asynchronous communication approach as described in the next option.
2b.) Let's consider that it is of utmost importance that some instance of service A will reliably perform the request to service B.
You can use message queues in this case as follows:
Let's say you have a queue where jobs to be performed by service A are collected.
Then you have several instances of service A running (see horizontal scaling) where each instance will consume the same queue.
You will use message locking features by the message queue service which makes sure that as soon one instance of service A reads a message from the queue the other instances won't see it. If service A was able to complete it's job (i.e. call service B, save some state in service A's persistence and whatever other tasks you need to be included for a succesfull procesing) it will delete the message from the queue afterwards so no other instance of service A will also process the same message.
If service A goes down during the processing the queue service will automatically unlock the message for you and another instance A (or the same instance after it has restarted) of service A will try to read the message (i.e. the job) from the queue and try to perform all the tasks (call service B, etc.)
You can combine several queues e.g. also to send a message to service B asynchronously instead of directly performing some kind of API call to it.
The catch is, that the queue service is some highly available and redundant service which will already make sure that no message is getting lost once published to a queue.
Of course you also could handle jobs to be performed in your own database of service A but consider that when service A receives a request there is always a chance that it goes down before it can save that status of the job to it's persistent storage for later processing. Queue services already address that problem for you if chosen thoughtfully and used correctly.
For instance, if look into Kafka as messaging service you can look into this stack overflow answer which relates to the problem solution when using this specific technology: https://stackoverflow.com/a/44589842/7730554
There is many way to solve your problem.
I guess you are talk about 2 topics Design Pattern in Microservices and Cicruit Breaker
https://dzone.com/articles/design-patterns-for-microservices
To solve your problem, Normally I put a message queue between services and use Service Discovery to detect which service is live and If your service die or orverload then use Cicruit Breaker methods
I have a application where I am using IBM MQ in a Pseudo-Synchronous way. I have request Q and Response Q. This will called through service. Now there is a time out period for my service to show the response.
But the actual response can be pushed to the response Q after the timeout also. Now I want to move such all delayed messages(timed out) to a separate message Q, such that they can be consumed by a different process.
Is this is possible through setting by time to live? Is any approach available in IBM MQ for such movement.
Please refer below image for the scenario.
Don't use time to live because the message will expire and be deleted - unless that is what you want. From your description/picture, it sounds like you want another application to process late responses.
There is nothing in MQ that will move messages older then "x" seconds/minutes to another queue. You will need to write a program to do that. If you write a program, make sure you do the move under a unit of work.
There are several commercial programs that can do it: (1) GUI program: MQ Visual Edit and (2) Command Line (shell): MQ Batch Toolkit.
MQPUT returns a successful post response (00) on IBM ZOS IMS online service, but the message is not getting inserted into the Remote queue. The queue connection was successful as well.
Program is written in cobol with IMS interface and the module is invoked through the IMS Websphere bridge interface.
By default for MQ on z/OS, MQPUTs are done under a local UOW. i.e. MQPMO-SYNCPOINT is set for Put Message Options. Hence, the code must perform an MQCMIT API call.
Update the code to use MQPMO-NO-SYNCPOINT and then the message will not be held waiting for an MQCMIT.
Thanks for your response, actually, the program had a rollback on the Logical unit of work when the program faces a failure situation, so in this case the program updated to the MQ but the roll back happened at the end of the processing ...
We have an environment where MQ acts as an interface between
Websites and Micro Focus. Sometimes a message gets stuck in a queue,
thereby blocking all the communications over that particular queue. If
the queue depth increases greatly, all the communication stops in the
queue manager.
When we check the status of queue, we see that microfocus process is present there.
Is there are way to automatically clear all applications connected to the queue?
I don't think its possible to close an applications handle on a given queue but you could have a script that runs a couple of MQSC commands against the queue manager to first get the connection identifier using the DISPLAY CONN command and then close the connection using the STOP CONN command. You could then setup a trigger on the queue that executes the script once a certain queue depth has been reached.
What is the best approach to connect to websphere mq v7.1 and clear all the messages of one or more specified queues using Java and JMS? Do I need to use Websphere MQ specific java API? Thanks.
Like all good questions, "it depends."
The queue can be cleared with a command only if there are no open handles on the queue. In that case sending a PCF command to clear the queue is quite effective, but if there are open handles you get back an error. PCF commands are of course a Java feature and not JMS because they are proprietary to WebSphere MQ.
On the other hand, any program authorized to perform destructive gets off a queue can clear the queue. In this case, just loop over a get until you get the 2033 return code indicating the queue is empty. This can be performed using JMS or Java but both of these manage the input buffer for you. If the queue is REALLY deep then you end up moving all that data and if the app is client connected, you are moving it at network speed instead of in memory.
To get around this, you need to specify a minimal amount of buffer and as one of the GET options also specify MQGMO.TRUNCATED_MSG_ACCEPTED. This moves only the message header during the get calls and can be significantly faster.
Finally, if you are doing this programamtically and regardless of which method you use, spin off several threads and don't use syncpoint. You actually have to go out of your way to get exclusive input on a queue so once you get a session, just spawn many threads off of it. Close each thread gracefully and shut down the the session once all the threads are closed.