We have a Mainframe Batch job which runs every 15 mins and looks for a file on FTP server. If File is found job executes and retrieves the file. If file is not found then job gets cancelled. 90% of the time, job does not find any file on FTP server and gets cancelled.
Is there any way to avoid running job every 15 mins to check file on FTP server? Please provide suggestions. Non-Mainframe option is also open.
Thanks,
Rahul
This seems like an ideal situation for WebSphere MQ. Message queues ensure the data file is processed once and only once after it has been received. It has a richer security and much better performance than a solution involving FTP and programs cobbled around it. When the message is received (i.e. the file that you were trying to send), MQ can trigger a batch job to process the file.
Here's a COBOL example of how to get a message off the queue:
https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ref.dev.doc/q089830_.htm
Here's a good article on configuring MQ to trigger the batch job:
http://thebigoldmainframe.blogspot.com/2012/09/mq-batch-triggering-process.html
Related
I believe I've got a scoping issue here.
Project explanation:
The goal is to process any incoming file (on disk), including meta data (which is stored in an SQL database). For this I have two tasklets (FileReservation and FileProcessorTask) which are the steps in the overarching "worker" jobs. They wait for an event to start their work. There are several threads dealing with jobs for concurrency. The FileReservation tasklet sends the fileId to FileProcessorTask using the job context.
A separate job (which runs indefinitely) checks for new file meta data records in the database and upon discovering new records "wakes up" the FileReservationTask tasklets using a published event.
With the current configuration the second step in a job can receive a null message when the FileReservation tasklets are awoken.
If you uncomment the code in BatchConfiguration you'll see that it works when we have separate instances of the beans.
Any pointers are greatly appreciated.
Thanks!
Polling a folder for new files is not suitable for a batch job. So using a Spring Batch job (filePollingJob) is not a good idea IMO.
Any pointers are greatly appreciated.
Polling a folder for new files and running a job for each incoming file is a common use case, which can be implemented using a java.nio.file.WatchService or a FileInboundChannelAdapter from Spring integration. See How do I kickoff a batch job when input file arrives? for more details.
I have a spring-batch job scanning the SFTP server at a given interval. When it finds a new file, it starts the processing.
It works fine for most cases, but there is one case when it doesn't work:
User starts uploading a new file to the SFTP server
Batch job checks the server and finds a new file
It start processing it
But since the file is still being uploaded, during the processing it encounters unexpected end of input block, and the error occurs.
How can I check that file was fully uploaded to the SFTP server before batch job processing starts?
Locking files while uploading / Upload to temporary file name
You may have an automated system monitoring a remote folder and you want to prevent it from accidentally picking a file that has not finished uploading yet. As majority of SFTP and FTP servers (WebDAV being an exception) do not support file locking, you need to prevent the automated system from picking the file otherwise.
Common workarounds are:
Upload “done” file once an upload of data files finishes and have
the automated system wait for the “done” file before processing the
data files. This is easy solution, but won’t work in multi-user
environment.
Upload data files to temporary (“upload”) folder and move them atomically to target folder once the upload finishes.
Upload data files to distinct temporary name, e.g. with .filepart extension, and rename them atomically once the upload finishes. Have the automated system ignore the .filepart files.
Got from here
We had similar problem, Our solution was, we configured spring-batch cron trigger to trigger the job every 10min(though we could configure for 5min, as file transfer was taking less than 3min), then we read/process all the files created prior to 10 minutes. We assume the FTP operation completes within 3 minutes. This gave us some additional flexibility such as when spring-batch app was down etc.
For example if the batch job triggered at 10:20AM we read all the files that were created before 10:10AM, like-wise job that runs at 10:30, reads all the files created before 10:20.
Note: Once Read you need to either delete or move to history folder for duplicate reads.
I am converting my Windows script script that uses FTP to SFTP.
To trigger the mainframe job we had below command:
quote site filetype=jes
put C:\Test\test.dat
bye
sftp.exe uname#servername
But site filetype=jes does not work in SFTP. What will be the equivalent command for SFTP to trigger the mainframe job by sending a trigger file?
There are several options:
You can use a different FTP server (such as the Co:Z product mentioned in an earlier response.
You can wrap a conventional FTP session in a secure network session (VPN, SSH, etc) in a way that keeps the connection secure, but doesn't require SFTP. This gives you the security of SFTP while letting you continue to use your existing FTP scripting unchanged.
You can swap FTP for more of a shell approach (SSH) to login to the mainframe and submit your JCL. Once you have any sort of shell session, there are many ways to submit JCL - see http://www-01.ibm.com/support/knowledgecenter/SSLTBW_1.13.0/com.ibm.zos.r13.bpxa500/submit.htm%23submit for an example.
A slight variant on #3 (above) is that you can have a "submit JCL" transaction in something like a web server, if you're running one on z/OS. This gives you a way to submit JCL using an HTTP request, say through CURL or WGET (if you go this way, be sure someone carefully reviews the security around this transaction...you probably don't want it open to the outside world!).
If this is something you do over and over, and if your site uses job scheduling software (CA-7, Control-M, OPC, Zeke, etc...most sites have one of these), almost all these products can monitor for file activity and launch batch jobs when a file is created. You'd simply create a file with SFTP "PUT", and the job scheduling software would do its thing.
Good luck!
If you're using the Co:Z SFTP server on z/OS you can submit mainframe batch jobs directly.
Strictly speaking this isn't a trigger file, but it does appear to be the equivalent of what you describe as your current FTP process.
I am using Spring-Batch to execute a batch that creates some objects in the database, creates a file from these objects and then sends the file to a FTP server.
Thus, I have 2 steps : One that reads conf from DB, insert into the DB and creates the file ; the second sends the file to the FTP server.
The problem is when there is a problem with the FTP server, I can't rollback the transaction (to cancel the new inserts into the DB).
How can I configure my Job to use just one transaction over the different steps?
This is a bad idea due to transactional nature of spring-batch.
IMHO a simple solution should be to mark data saved in step 1 with a token generated when job starts and, if your FTP upload will fail, move to a cleanup step to delete all data with token.
A agree with bellabax: this is a bad idea.
But I wouldn't do a 3rd cleanup step because this step may also fail, letting the transaction not rollbacked.
You could mark the inserted entries with a flag that indicates the entries has not yet been sent to the FTP.
The 3rd step would switch the flag to indicate that these entries has been sent to the FTP.
Then you just need a cron/batch/4th cleaning step/whatever that would remove all entries that haven't been sent to the FTP
Following process in our linux server is taking 100% of CPU
java -DMQJMS_LOG_DIR=/opt/hd/ca/mars/tmp/logs/log -DMQJMS_TRACE_DIR=/opt/hd/ca/mars/tmp/logs/trace -DMQJMS_INSTALL_PATH=/opt/isv/mqm/java com.ibm.mq.jms.admin.JMSAdmin -v -cfg /opt/hd/ca/mars/mqm/data/JMSAdmin.config
I forcibly killed the process and bounced MQ then i don't see this. What might be the reason for this to happen?
The java process com.ibm.mq.jms.admin.JMSAdmin is normally executed via the IBM MQ script /opt/mqm/java/bin/JMSAdmin.
The purpose of JMSAdmin is to create JNDI resources for connecting to IBM MQ, these are normally file based and stored in a file called .binding, the location of the .binding file would be found in configuration file that is passed to the command. In your output above the configuration file is /opt/hd/ca/mars/mqm/data/JMSAdmin.config.
JMSAdmin is an interactive process where you run commands such as:
DEFINE QCF(QueueConnectionFactory1) +
QMANAGER(XYZ) +
...
I would be unable to tell you why it was taking 100% CPU, but the process itself does not directly interact with or connect to the queue manager and it would be safe to kill off the process with out needing to restart the queue manager. The .binding file that JMSAdmin generates is used by JMS applications in some configurations to find details of how to connect to MQ and the names of queues and topics to access.
In July 2011 you would have been using IBM MQ v7.0 or lower all of which are out of support, if anyone should come across a similar issue with a recent supported version of MQ I would suggest you take a java thread dump and open a case with IBM to investigate why it is taking up 100% of the CPU.
*PS I know this is a 9 year old question, but I thought an answer may be helpful to someone who finds this when searching for a similar problem.