I'm new to WebSphere MQ and I am trying to move files through WebSphere MQ as messages using c# code. Can WebSphere MQ transfer files (e.g. *.txt) directly?
Thanks in Advance.
WebSphere MQ File Transfer Edition (FTE) is a product built on top of WebSphere MQ for transferring files. No coding is required to for transferring files. There are commands and MQ Explorer through which you can configure your file transfers. You can also start file transfers through scripting.
Read through this developerWorks article
No, base WebSphere MQ does not directly transfer files. The semantics for moving files include such things as file names, directory paths, conversion of names across different types of filesystem (i.e. a UNIX filetree versus a z/OS Partitioned Dataset). Messaging uses semantics specific to queues and topics and these are drastically different from those for files.
Accordingly, base WMQ does not attempt to perform file semantics as part of its core functionality. The WebSphere MQ File Transfer Edition and IBM Sterling Connect:Direct provide that functionality. In the case of FTE, managed file transfer is natively built on top of WebSphere MQ.
Many people have in the past written solutions to move files using messaging, whether that's WebSphere MQ or another transport provider. It turns out that doing so in a robust way is a non-trivial task. Here are some of the issues:
If you make each file a single message then message sizes vary considerably. It is very difficult to tune the messaging engine to simultaneously perform with very large and very small messages.
If you break the files up into chunks then you cna tune chunk size for optimal messaging performance but now you must deal with sequence preservation and resend of missing messages.
The general problem of whether to append, replace or rename a destination file with the same name.
The general problem of how to make the file available at the destination in a transactional manner. You don't want the recipient to grab half a file.
File integrity (FTE hashes the file to insure it doesn't get changed in transit).
These are a sampling of the issues to do this in a general-purpose way. However if all you need is a one-time, point-to-point solution then have a look at SupportPac MA01, the Q program. It will take files and either put a line at a time into a message or make the entire file a single message. It can also be used as a pipe so it's possible to have two instances of Q where one pipes messages to a queue and the other retrieves them.
Related
We give support to an organization that uses IBM MQ (v9.0) to communicate to an external partner. It's a simple layout, a single local queue manager connected to an external one trough one single channel. Various applications use different queues to communicate trough this QM.
Last week, after solving a support ticket, my boss came to the conclusion that having one single channel to funnel all the messages from the different queues (and applications that use such queues) is not good.
I have to write a report detailing what criteria should we use to separate the message flows into different channels, and what benefits and inconveniences would this entail.
I'm fairly new to IBM MQ, and I'm having trouble finding information on this topic. So far I've read the official documentation and other bibliography and i have a superficial idea about it, but i cant find concrete information about this topic.
Is separating message flows into different channels a good practice? What would be a good criteria to use?
Here is a good write up on channels and it discusses reasons for parallel channels. You may be able to use this in your report. MQ Channels
If you have applications with large messages that are of the fire-and-forget pattern it is common to put these on a different path than the real time apps that are of the request-response pattern.
Whether or not this configuration change will lessen the chance of problem tickets depends largely on what led to the issue. Good luck.
The Context:
I'm hoping I'm missing something simple. I am a novice in IBM MQ
My organisation distributes XML messages via IBM Websphere MQ. Each message represents a record that I simply want to save to a file and clear from the queue.
The current mechanism we have is a poorly understood and overly complex Java program that seems to use both MQ libraries and JMS to do this.
I am hoping to move this to something simpler and easier to maintain, but I can't seem to find a generic 'IBM MQ Client' that can do things like connect to a queue and save the messages. It seems like you have to use binaries or DLLS and build an application around it.
I tried Python as that's closer to my understanding but PyMQI is only for Python 2 (which we don't use) and requires a C compiler which always seems to refuse to work.
The question:
Is there a program out there in windows to simply access a queue and save the messages to a file? We connect periodically so I can just wrap a scheduled task around it.
IBM MQ queue manager and clients communicate using a proprietary protocol, so you definitely need to use IBM MQ Client binaries (dlls or lib or jars depending on language you are comfortable with) to put/get messages to/from a queue.
I am sure the JMSToolbox mentioned by #titou10 runs on top of IBM MQ JMS jars.
Yes
If you deal with JMS messages in IBM MQ then JMSToolBox on SourceForge is a tool that will fulfill your needs.
You can download it here
I am trying to understand the persistence model of IBM MQ. I realize that the messages are stored in file based system but is there a way I can store the files in DB. I have come across article that states that the messages can be stored in the DB2 database, but I would like to know if I can store in other databases such as Oracle DB or SQLite or No-SQL Db's.
You may be misinterpreting the article. (A link would be nice.) Early versions of MQ were closely intertwined with IBM's database at the time and much of the product retains traces of that heritage in names, sizes of data structures, and some functional constraints.
However modern MQ's primary use of database is in the coupling facility in support of shared queues, and this is z/OS only. However that's the exception and for the vast majority of MQ versions and platforms you don't have the ability to select the queue persistence mechanism.
Remember that IBM's MQ was never intended to be a queueing front-end database client. MQ was invented to address problems of synchronous communication in which the outage of a single component caused entire systems to fail. The purpose of MQ was so that the application could treat the network as if it was 100% reliable (which was far from the case back in the 1980's). In order to achieve that MQ kept all critical operations local to the host on which it was installed and minimized external dependencies to things that were the most reliable: POSIX IPC and locally-mounted filesystems.
Some more recent message transports, especially JMS providers, give you the option of selecting a persistence store and even allowing it to be remote. Some are barely more than a JMS API over a database client. Though each of these approaches has valid use cases, IBM MQ has retained its focus on ultimate speed and enterprise-grade reliability.
Other IBM products such as the WAS Messaging Engine, MQTT, MQ Light, Sterling and others exist to fill some of the other requirement spaces. As for MQ, no there's no option to pick your persistence store other than the on the z/OS platform where your options are mostly related to the coupling facility.
We have two systems, one based on JMS and another based on WebSphere MQ.
There is client A which sends a message to a topic configured in JMS. Another client B which receives this message through the Topic configured in WebSphere MQ.
How can I make this communication happen? What are the considerations while building this bridge? If bridge is the solution, how can I build it?
I assume from your description that one of the clients is written in Java (JMS) and the other one is written in an other language and both have access to the same queue. MQ-Series is a queuing product, JMS is a Java API (like JDBC is to relational databases). MQ-Series supports the JMS API so there is no problem in communciating messages. JMS will probably be a subset of possible MQ-Series features.
Make sure that the content can be interpreted by both parties. The standard way is to use XML in the message. But you could use any other format that both clients can understand. You could also use CSV (comma separated values), JSON (JavaScript object notation) and there are even cross platform binary formats like Hessian.
But if you could be more specific about the participants and the kind of information you want to communicate, you probably would get more specific answers.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
We are facing a choice to use IBM MQ over SFTP for file transfer. I've heard advantages of such approach, but I've never see anyone actually using it for a large files.
So main question: how well IMB MQ can handle transfer of large files (up to 100 MB)? Is it stable? It's from mainframe to UNIX server, if it does matter.
Thanks a lot.
I've used MQ with files up to 8GB in size, without incident. You have to allocate enough space for MQ to manage them, but it works.
MQ itself offers message-based communication between programs. You can't use it directly to transfer files. People talking about "sending files via MQ" often mean that they will write the programs themselves to read from a file, send over MQ, receive from MQ and write to another file. This sounds temptingly trivial at first but quickly becomes very complex if you want it to be robust. Not recommended.
What you should be doing if you want to involve MQ in your file transfers is use one of the file-transfer products that sit upon MQ. There are a number of mature offerings, about which I know next to nothing, together with a new product from IBM itself ("Websphere MQ File Transfer Edition") which is being actively developed and recently announced an updated version (7.0.1).
To answer your specific question, I've personally used WMQFTE to transfer a file of 1000GB from an AIX machine to a Windows one, so I don't think your "large" 100MB files will be a problem :-). I don't get involved with the mainframe as much but it is very much a core platform for FTE.
Vendors have been doing File transfer over a messaging system such as WebSphere MQ for over 12 years. I have not used the IBM product. Yet, I know until recently IBM resold Metastorm’s product (under the brand PM4Data) and only recently entered the market with a first generation product.
I’ve seen Metastorm Integration Manager handle files of terabyte sizes all of the time; breaking up the messages across WebSphere MQ and performing buffers acknowledgements so not to overflow message queues or having to do a lot of MQ Administration.
Other worthwhile considerations of using messaging systems for file transfer vs. SFTP include:
Pre-and Post-Data Movement Handling - As a network transport, SFTP moves data (as can file transfer products over WebSphere MQ). To do pre- or post-movement processing, you’d have to write your own monitoring, triggering, error handling, etc. Mature vendors in the file transfer over messaging space provide infrastructure out-of-the-box for you. Depending on your application, they can also help
Audit Trail - SFTP provides often provide system-level logging. Often time, vendors providing file integration over MQ have greater audit capabilities, including the Pre- and Post-Data Movement Handling. Some vendors also offer secure web-based systems to expose certain integrations to authorized users.
Exception Handling – SFTP will give synchronous error messages from which you then code your error logic. Vendors providing file integration over MQ often provide exception capabilities ranging from "here’s a message on queue signifying a problem happened... good luck" to built-in integration to Web-based consoles or even BPM system.
Treating File Transfer as a Service – Often times, File Transfer is part of a larger business process. Some vendors leverage this to chain file transfers and even have them collaborate with human-centric processes to provide end-to-end visibility around the context of a point-to-point file transfer. Additionally, using messaging systems for file transfer helps move an organization to a more real-time, message-driven approach.
I hope the additional details give some details in your assessment. Good luck!
In one of the previous answers, there is the followng statement:
People talking about "sending files via MQ" often mean that they will write the programs themselves to read from a file, send over MQ, receive from MQ and write to another file. This sounds temptingly trivial at first but quickly becomes very complex if you want it to be robust. Not recommended.
In theory it would be nice to "sending files via MQ" because you would receive a Confirmation on Delivery (CoD) message when the recipient got (read) the file from the queue. Another benefit would be that the file would be automatically removed from the queue when the recipient got (read) it from the queue.
Do you perhaps know whether the above (or a similar) is offered by MQFTE?
Capitalwares Universal File Mover is an option here.
Also WebSphere Message Broker for Remote Adapter Deployment and WebSphere MQ File Transfer Edition are options to consider.