I have a question, When fte picks a file, convert the record delimiters into messages. At the end of the message, can we add one EOF message?
FTE provides several call-out points at which you can perform almost any manipulation. In this case, the post-transfer destination call can be used to append any data you wish to the end of the file prior to marking the transfer complete.
Please see Running programs before or after a transfer in the Infocenter.
Related
I think title is confusing. I try to explain it in detail.
I have a big xml file that contains a lot of messages for queue. The structure looks like: <'objects'><...>.....<...><'/objects'>. It's an array of objects tags and one item of objects is a message. I want to put this file that should be broken into a separate messages into queue(mq) by using rfhutil.
I know it's possible but I'm confused about delimiter. The whole menu looks puzzled.
Is anyone using rfhutil and mq? Maybe you have a guide for that. I couldn't find any information about rfhutil.
I hope I explained well.there is a pic of rfhutil menu. load Q
"Load Q" option is used to send all messages from file which was previously created with "Save Q" option. It will not properly parse your custom xml. Even if you format xml in a way that is parsable by custom delimiter, xml still contains only message payload and lacks message headers.
However rfhutil is part of lagerger IH03 SupportPack and there you will find:
additional software to send multiple custom messages like mqput
a manual how to do it (ih03.doc)
Simplest "out of head" solution would be to split xml in multiple smaller files and than send them all with one mqput2 command.
I have a script in my server that displays its performance to the user in a text file. When the same script is executed in parallel by multiple users the information in the text file gets mixed up. I append many details of the server in the text file which takes roughly less than a minute to come up with the output. If i do file locking will it hit the performance or is there any way i need to look upon.
Please help me on how to proceed .
Thanks
Balakrishnan
You could make use of a message queueing system:
POSIX message queues:
http://www.linuxhowtos.org/manpages/7/mq_overview.htm
Beanstalkd: http://kr.github.io/beanstalkd/
POSIX Message Queue for Ruby: http://rubygems.org/gems/posix_mq
Perl: http://search.cpan.org/~iljatabac/POSIX-RT-MQ-0.03/MQ.pm
Python IPC: http://semanchuk.com/philip/posix_ipc/
Other threads:
Are message queues obsolete in linux?
https://unix.stackexchange.com/questions/70837/linux-command-to-check-posix-message-queue
https://stackoverflow.com/questions/40296/what-is-the-best-free-tool-for-managing-msmq-queues-and-messages
The idea is to create a server process that would receive messages and store on a buffer. It would only print a line on the logfile everytime a message from a process already has a complete line.
FLoM http://sourceforge.net/projects/flom/ can manage the lock you need: it's easy to use, it's fast, the same resource can be locked/unlocked by different users and it implements a rich lock model.
This example use case could give you some ideas about the tool: http://sourceforge.net/p/flom/wiki/Use%20Case%206/
Cheers
Ch.F.
Is there a tool which can take 1000 Seperate HL7 Messages and combine them into a single document for 7edit? I need to run a test, and if I can do one document and choose send all, it will be better than me running it manually for each of these 1000 messages.
Yes, There exist a way to combine those messages in a single file. You can do that using any integration engine, I will take Mirth in this case.
Follow these steps in sequential order
Download Mirth Connect from here using the .exe installer (in case you don't have it).
Setup your account and do initial configuration on your local system.
Create a Channel called Appending Channel, put Source inbound and outbound connector as HL7v2.x.
Go to Source Tab, Put Connector type as File Reader. Give the location of the directory where your messages will reside(D:\x\read in my case). Make sure you have the directory shared
You can make Delete file after read as a Yes, which will prune the files after they are read from this location.If you do a NO, then specify where you want to move those files to.
Put Process Batch files as a No.
Go to Destinations tab, create a Destination called as Appender and make it a File Writer type.
Give the directory(D:\x\Output in my case) where your final file will be placed.Give the file name as final.txt.
Choose Append on the file exists tab.
In Template, Drag Raw Data from the list on the right hand side, and put it here or else what you can do is type ${message.rawData} in the template section.
Save Channel and Deploy it.
Place all your messages in the read folder (mentioned above), and wait for Mirth to poll the folder (default setting is 1000 ms).
Once that is done, go to final.txt to see all the messages appended in the same file.
The downside is that even though this process is 100 percent working, the message thus appended will not be seperated by any means. So it will look like below
|2688684|||||||||||||||||||||||||199912271408||||||002376853MSH|^~\&|EPIC|EPICADT|
^ End of first message
You don't need any tool for that. 7edit is able to read multi-message files. You just need to append each message into one single text file like this (two ADT messages):
MSH|^~\&|SystemA|CompanyA|SystemB|CompanyB|20121116122025||ADT^A01|101|T|2.5||||||UNICODE UTF-8
EVN|A01|20130823080958
PID|||1000||Lastname^Firstname
PV1||I
MSH|^~\&|SystemA|CompanyA|SystemB|CompanyB|20121116122026||ADT^A01|102|T|2.5||||||UNICODE UTF-8
EVN|A01|20130823080958
PID|||1000||Lastname^Firstname
PV1||I
Open this file with 7edit and you will see this (multiple messages):
Now you can send all messages at once by pressing on Send and then select All Messages:
It is that simple - no other tool necessary (just to make the append in one file maybe)
You could also try to use HL7Browser (www.nule.org), a tool that is similar to 7Edit, with less features but free.
You should be able to open many single HL7 messages files, HL7Browser will cache them in its viewer and should allow you to save them all to a single file.
Hope helps
Davide
if you have multiple HL7 files in one folder and want to combine them into 1 hl7 file, you can do following:
create a batch file in this folder named combine.cmd
write following into this batch file
del combined.hl7
for %%f in (*.hl7) do type "%%f" >> combined.hl
move combined.hl combined.hl7
run this batch file
result: all hl7 files in this folder are combined into a single file called "combined.hl7"
If I'm writing a simple text log file from multiple processes, can they overwrite/corrupt each other's entries?
(Basically, this question Is file append atomic in UNIX? but for Windows/NTFS.)
You can get atomic append on local files. Open the file with FILE_APPEND_DATA access (Documented in WDK). When you omit FILE_WRITE_DATA access then all writes will ignore the the current file pointer and be done at the end-of file. Or you may use FILE_WRITE_DATA access and for append writes specify it in overlapped structure (Offset = FILE_WRITE_TO_END_OF_FILE and OffsetHigh = -1 Documented in WDK).
The append behavior is properly synchronized between writes via different handles. I use that regularly for logging by multiple processes. I do write BOM at every open to offset 0 and all other writes are appended. The timestamps are not a problem, they can be sorted when needed.
Even if append is atomic (which I don't believe it is), it may not give you the results you want. For example, assuming a log includes a timestamp, it seems reasonable to expect more recent logs to be appended after older logs. With concurrency, this guarantee doesn't hold - if multiple processes are waiting to write to the same file, any one of them might get the write lock - not just the oldest one waiting. Thus, logs can be written out of sequence.
If this is not desirable behaviour, you can avoid it by publishing logs entries from all processes to a shared queue, such as a named pipe. You then have a single process that writes from this queue to the log file. This avoids the conccurrency issues, ensures that logs are written in order, and works when file appends are not atomic, since the file is only written to directly by one process.
From this MSDN page on creating and opening Files:
An application also uses CreateFile to specify whether it wants to share the file for reading, writing, both, or neither. This is known as the sharing mode. An open file that is not shared (dwShareMode set to zero) cannot be opened again, either by the application that opened it or by another application, until its handle has been closed. This is also referred to as exclusive access.
and:
If you specify an access or sharing mode that conflicts with the modes specified in the previous call, CreateFile fails.
So if you use CreateFile rather than say File.Open which doesn't have the same level of control over the file access, you should be able to open a file in such a way that it can't get corrupted by other processes.
You'll obviously have to add code to your processes to cope with the case where they can't get exclusive access to the log file.
No it isn't. If you need this there is Transactional NTFS in Windows Vista/7.
I got an application which is polling on a folder continuously. Once any file is ftp to the folder, the application has to move this file to some other folder for processing.
Here, we don't have any option to verify whether ftp is complete or not.
One command "lsof" is suggested in the technical forums. It got a file description column which gives the file status.
Since, this is a free bsd command and not present in old versions of linux, I want to clarify the usage of this command.
Can you guys tell us your experience in file verification and is there any other alternative solution available?
Also, is there any risk in using this utility?
Appreciate your help in advance.
Thanks,
Mathew Liju
We've done this before in a number of different ways.
Method one:
If you can control the process sending the files, have it send the file itself followed by a sentinel file. For example, send the real file "contracts.doc" followed by a one-byte "contracts.doc.sentinel".
Then have your listener process watch out for the sentinel files. When one of them is created, you should process the equivalent data file, then delete both.
Any data file that's more than a day old and doesn't have a corresponding sentinel file, get rid of it - it was a failed transmission.
Method two:
Keep an eye on the files themselves (specifically the last modification date/time). Only process files whose modification time is more than N minutes in the past. That increases the latency of processing the files but you can usually be certain that, if a file hasn't been written to in five minutes (for example), it's done.
Conclusion:
Both those methods have been used by us successfully in the past. I prefer the first but we had to use the second one once when we were not allowed to change the process sending the files.
The advantage of the first one is that you know the file is ready when the sentinel file appears. With both lsof (I'm assuming you're treating files that aren't open by any process as ready for processing) and the timestamps, it's possible that the FTP crashed in the middle and you may be processing half a file.
There are normally three approaches to this sort of problem.
providing a signal file so that when your file is transferred, an additional file is sent to mark that transfer is complete
add an entry to a log file within that directory to indicate a transfer is complete (this really only works if you have a single peer updating the directory, to avoid concurrency issues)
parsing the file to determine completeness. e.g. does the file start with a length field, or is it obviously incomplete ? e.g. parsing an incomplete XML file will result in a parse error due to the lack of an end element. Depending on your file's size and format, this can be trivial, or can be very time-consuming.
lsof would possibly be an option, although you've identified your Linux portability issue. If you use this, note the -F option, which formats the output suitable for processing by other programs, rather than being human-readable.
EDIT: Pax identified a fourth (!) method I'd forgotten - using the fact that the timestamp of the file hasn't updated in some time.
There is a fifth method. You can also check if the FTP Session is still active. This will work if every peer has it's own ftp user account. As long as the user is not logged off from FTP, assume the files are not complete.